JP6278688B2 - IMAGING DEVICE, IMAGING DEVICE CONTROL METHOD, AND PROGRAM - Google Patents

Description

  The present invention particularly relates to an imaging apparatus including a plurality of imaging optical systems, an imaging apparatus control method, and a program.

  2. Description of the Related Art Conventionally, an imaging apparatus such as a digital still camera is known that detects a human face area from a sequentially captured subject or tracks a detected face area. Such an imaging apparatus holds information on the detected face area of the person inside and performs specific processing based on the held face area information. For example, in Patent Document 1, identification information indicating that a person's face is being tracked is displayed based on face area information, or a display holding time of identification information is set based on a focal length, a subject distance, or the like. Techniques to do this are disclosed.

JP 2010-81587 A

  On the other hand, there is also known an imaging apparatus including a plurality of imaging optical system mechanisms that are provided with a separate imaging optical system in the opposite direction, for example, so that the photographer himself can be imaged. However, in an imaging apparatus having a plurality of imaging optical system mechanisms, if an attempt is made to detect or track a person's face area from a subject that is sequentially captured by each imaging optical system, a large load is generated on the CPU. .

  The present invention has been made in view of the above-described problems, and an object of the present invention is to reduce the burden of processing due to subject detection and tracking in an imaging apparatus including a plurality of imaging optical systems.

An imaging apparatus according to the present invention includes an imaging unit that images a subject and generates image data of a plurality of frames, and a detection that detects a subject region that satisfies a predetermined condition in the image data generated by the imaging unit. means, in the image data generated by the imaging means, and a tracking means that performs tracking of high similarity with the detected subject region by the detecting unit area, the imaging means, the first imaging optical and the system, said first and a second imaging optical system facing different directions imaging optical system, the pre-Symbol image data generated from the first imaging optical system, said by the detection means thus both the tracking by detecting said tracking means of the subject region, wherein the image data generated from the second imaging optical system, performs the detection of the object area by said detecting means Characterized in that it is not performed tracking by serial tracking means.

  ADVANTAGE OF THE INVENTION According to this invention, in the imaging device provided with the some imaging optical system, the load of the process by a to-be-photographed object's detection and tracking can be reduced.

It is a block diagram which shows the structural example of the digital camera which concerns on embodiment. It is a figure which shows the detailed structural example of the optical system mechanism in the digital camera which concerns on embodiment. It is a figure for demonstrating the timing which produces | generates the image for evaluation by a subcamera. It is a figure for demonstrating the timing which produces | generates the image for evaluation by a main camera. It is a figure for demonstrating the timing which performs a face detection and face tracking based on the image for evaluation produced | generated by the main camera and the sub camera, respectively. It is a figure for demonstrating the holding | maintenance state of the face position information detected by the face detection or the face tracking process from the evaluation image of the main camera. It is a figure for demonstrating the holding | maintenance state of the face position information detected by the face detection from the image for evaluation of a sub camera.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a method for controlling detection and tracking processing of a subject area in a shooting sequence when shooting using a digital camera as an example of an imaging apparatus will be described. The digital camera according to the present embodiment has general functions such as AE (automatic exposure control) and AF (autofocus), and the AF function is a well-known contrast detection method.

FIG. 1 is a block diagram illustrating a configuration example of a digital camera 100 according to the present embodiment.
In FIG. 1, a digital camera 100 according to this embodiment includes a first photographing lens 101, a second photographing lens 102, a first aperture / shutter 103, a second aperture / shutter 104, and automatic exposure (AE). A processing unit 105 is provided. Furthermore, a first focus lens 106, a second focus lens 107, an autofocus (AF) processing unit 108, a first image sensor 109, a second image sensor 110, a first A / D converter 111, and A second A / D converter 112 is provided.

  The first photographing lens 101 and the second photographing lens 102 are lenses having a zoom mechanism. The first aperture / shutter 103 and the second aperture / shutter 104 receive the reflected light of the subject incident on the first image sensor 109 and the second image sensor 110, respectively, in accordance with instructions from the AE processing unit 105. Control light intensity and charge accumulation time. The AE processing unit 105 controls the operations of the first aperture / shutter 103 and the second aperture / shutter 104, respectively, and also includes a first A / D conversion unit 111 and a second A / D conversion unit 112 described later. To control.

  The first focus lens 106 and the second focus lens 107 are focused on the light receiving surfaces of the first image sensor 109 and the second image sensor 110, respectively, in accordance with a control signal from the AF processing unit 108, and are optical images. Is imaged. The first image sensor 109 and the second image sensor 110 convert an optical image formed on the light receiving surface into an electric signal by a photoelectric conversion means such as a CCD element or a CMOS element, and perform first A / D conversion, respectively. Output to the unit 111 and the second A / D converter 112.

  The first A / D converter 111 and the second A / D converter 112 each convert the received electrical signal (analog signal) into a digital signal. In addition, the first A / D converter 111 and the second A / D converter 112 are each a non-linear CDS circuit that removes noise from the received electrical signal, or before converting the received electrical signal into a digital signal. A non-linear amplifier circuit for amplifying is included.

  The digital camera 100 further includes an image processing unit 113, a format conversion unit 114, and a DRAM 115.

  The image processing unit 113 performs resize processing and color conversion processing such as predetermined pixel interpolation and image reduction on the digital signals output from the first A / D conversion unit 111 and the second A / D conversion unit 112. To output image data. The format conversion unit 114 converts the format of the image data generated by the image processing unit 113 in order to store the image data in the DRAM 115. The DRAM 115 is an example of a high-speed built-in memory, and is used as a high-speed buffer that manages temporary storage of image data, or as a working memory in image data compression / decompression processing.

  Further, the digital camera 100 includes an image recording unit 116, a system control unit 117, a VRAM 118, a display unit 119, an operation unit 120, a main switch (main SW) 121, a first switch (SW1) 122, and a second switch (SW2). 123.

  The image recording unit 116 includes a recording medium such as a memory card for recording still image data or moving image data, and an interface thereof. The system control unit 117 includes a CPU, a ROM, and a RAM. The CPU controls the entire operation of the digital camera 100 by developing and executing a program stored in the ROM in a RAM work area. Further, the system control unit 117 controls which mode is used from among a plurality of imaging drive modes for the first image sensor 109 and the second image sensor 110. Furthermore, the system control unit 117 also performs face detection processing and face tracking processing. As face detection processing, for example, an image signal is decomposed into color components and luminance components, and a region that matches a facial part (eyes, nose, mouth, etc.) is stored in advance using a well-known pattern recognition technique. In addition, there is a method of detecting a region including them as a face. When multiple faces are detected, an evaluation value is assigned to each face so that the larger the face size and the closer the face position is to the center of the image, the larger the evaluation value. A high face is the main subject. As the face tracking process, for example, there is a method of storing an image of a face set as a main subject as a template, and detecting an area having the highest similarity with the stored template image from images of subsequent frames. . The VRAM 118 is an image display memory. The display unit 119 is an LCD or the like, for example, and displays a screen for displaying an image or assisting an operation, displays a camera state, and displays a shooting screen and a distance measurement area at the time of shooting.

  When the user operates the operation unit 120, an instruction is input to the digital camera 100. The operation unit 120 includes, for example, a menu switch for performing various settings such as exposure correction, aperture value setting, image playback setting, a zoom lever for instructing a zoom operation of the photographing lens, and an operation mode switching between the photographing mode and the reproduction mode. It has a switch etc.

  The main switch 121 is a switch for turning on the power of the system of the digital camera 100. The first switch 122 is a switch for performing a shooting preparation operation such as AE processing or AF processing, and the second switch 123 is a switch for giving a shooting instruction to the system control unit 117 after the operation of the first switch 122. It is.

<Configuration of optical system mechanism>
FIG. 2 is a diagram illustrating a detailed configuration example of the optical system mechanism in the digital camera 100 according to the present embodiment. The digital camera 100 according to the present embodiment includes a first imaging optical system 205 and a second imaging optical system 210 as an optical system mechanism. Hereinafter, the first imaging optical system 205 is called a main camera, and the second imaging optical system 210 is called a sub camera.

  As shown in FIG. 2, in the first imaging optical system 205 that is the main camera, external light is condensed on the first imaging element 204 by the first lens 201. In the example shown in FIG. 2, one lens is used, but a lens unit composed of a plurality of lenses may be mounted. Further, the first lens 201 can adjust the focal point and adjust the angle of view by moving the position back and forth along the optical axis. The amount of light passing through the first lens 201 is adjusted by the first diaphragm 203. Further, during still image capturing, the exposure time can be adjusted by controlling the opening and closing time of the first mechanical shutter 202.

  In addition, as shown in FIG. 2, the digital camera 100 further includes a second imaging optical system 210 similar to the above-described mechanism so as to capture external light from the reverse direction. That is, the sub camera is an imaging system that assumes that the photographer himself is photographed. The second imaging optical system 210, which is a sub camera, includes a second lens 206, a second mechanical shutter 207, a second diaphragm 208, and a second imaging element 209. These functions are the main camera. It is the same.

<Face detection and face tracking processing>
Here, the timing for generating the evaluation image by the sub camera will be described with reference to the timing chart of FIG. As shown in FIG. 3, in the sub camera, exposure 302 is performed at each vertical synchronization (VD) based on the vertical synchronization signal 301 of the second image sensor 110.

  In the sub camera, image data exposed by each VD is read from the second image sensor 110, and an evaluation image generation process 303 is performed based on the read image data. That is, in the sub camera, evaluation images related to face detection of a person subject are sequentially generated from images taken in time series. The exposure 302 is performed for each VD, and the evaluation image generation process 303 is performed for each VD from the next VD. Specifically, the evaluation image generation process 303 for the image data in the exposure period 310 is performed in the next VD period 311.

  Next, the timing of generation of an image for evaluation, face detection, and face tracking by the main camera will be described with reference to the timing chart of FIG. As shown in FIG. 4, in the main camera, exposure 402 is performed at each vertical synchronization (VD) based on the vertical synchronization signal 401 of the first image sensor 109.

  In the main camera, image data exposed at each VD is read from the first image sensor 109, and an evaluation image generation process 403 is performed based on the read image data. That is, in the main camera, evaluation images relating to face detection of a human subject are sequentially generated from images taken in time series. The exposure 402 is performed for each VD, and the evaluation image generation processing 403 is performed for each VD from the next VD. Specifically, the evaluation image generation process 403 for the image data of the exposure time 410 is performed in the next VD period 411.

  Next, the timing for performing face detection and face tracking based on the evaluation images generated by the main camera and the sub camera, respectively, will be described with reference to the timing chart of FIG. As shown in FIG. 5, in the digital camera 100, face detection is performed at each vertical synchronization (VD) based on the vertical synchronization signal 501 of the first image sensor 109. In the example shown in FIG. 5, the period of the vertical synchronization signal 501 is the same as the vertical synchronization signal 401 of the first image sensor 109, but may be the same as the vertical synchronization signal 301 of the second image sensor 110, for example. Good.

  Specifically, as shown in FIG. 5, in the face detection at time 510, as the face detection target image 512 indicates, the evaluation image output from the main camera is the target. The first face detection is performed using. Then, the detected face position information is stored in the DRAM 115.

  Further, as the face tracking target image 513 indicates, the second face detection is performed using the face tracking process on the evaluation images 511 and 514 of the main camera. When the face position obtained by the first face detection described above and the face position obtained by the second face detection overlap, the face position information obtained by the first face detection is stored in the DRAM 115. To do. When the face position obtained by the first face detection and the face position obtained by the second face detection do not overlap, the face obtained by the first face detection is a face of another person. The face position information obtained by the second face detection is stored in the DRAM 115. In the first face detection, when the detection result is not detected, the face position information detected by the second face detection is stored in the DRAM 115. In addition, when not only the first face detection but also the detection result by the second face detection using the face tracking technology is not detected, the face position information stored in the DRAM 115 is not updated, and the previous face is not updated. Keep the location information.

  Further, in the face detection at the time 515, as the face detection target image 517 indicates, since the evaluation image output from the sub camera is the target, the first face detection is performed using the evaluation image 516 of the sub camera. . Then, the detected face position information is stored in the DRAM 115. Here, the face tracking process is not performed at the timing for the evaluation image of the sub camera, and the face position information stored in the DRAM 115 is updated if the detection result is not detected in the first face detection. Without holding the previous face position information.

  In the present embodiment, the photographer himself is assumed as a subject photographed by the sub camera, and it is assumed that the frequency of variation of the face position of the subject within the angle of view of the sub camera is low. For this reason, even if the face position of the subject cannot be temporarily detected from within the angle of view of the sub camera, it is unlikely that the face position has moved from the position at which it was last detected. Therefore, the face tracking process is not performed on the evaluation image of the sub camera, and the face is considered to exist at the same position as the face position when it was detected immediately before. In addition, the face detection process for the evaluation image of the sub camera is set to a low frequency of once every 10 times, and the face tracking process for the evaluation image of the sub camera is not performed. Yes.

<Retain face position information>
As shown in FIG. 6, the system control unit 117 stores face position information detected by face detection or face tracking processing in the DRAM 115 for a certain period. As described above, when the detection results of the face detection and the face tracking processing are not detected, the face position information stored in the DRAM 115 is not updated as in the face position information 610, for example, and the previous face position is not updated. Continue to retain information.

  Further, when the detection results of the face detection and the face tracking process are both undetected twice, the held face position information is deleted from the DRAM 115. Specifically, both the face detection result 613 and the face tracking result 614 for the main camera evaluation image 612 at time 611 are not detected, and are not detected twice consecutively together with the immediately preceding detection result. In this case, the main face position information stored in the DRAM 115 is deleted.

  FIG. 7 is a diagram for explaining the timing for deleting the face position information detected from the evaluation image of the sub camera. The VD 701 indicates the timing of each vertical synchronization for performing face detection on the evaluation image 503 of the sub camera, and in this example, indicates the period of the vertical synchronization signal once in 10 times. .

  When a face is detected by the face detection, the system control unit 117 stores face position information in the DRAM 115. As described above, when the detection result by the face detection is not detected, the face position information from the sub camera stored in the DRAM 115 is not updated as in the face position information 710, for example, from the previous sub camera. Will continue to hold face position information. Further, when the detection result by the face detection is not detected three times continuously, the held face position information from the sub camera is deleted. Specifically, the face detection result 713 for the sub-camera evaluation image 712 at the time 711 is not detected, and is stored in the DRAM 115 when it is not detected three times in succession together with the previous detection result. Delete face position information.

  As described above, according to the present embodiment, since the face tracking process is not performed on the image data generated from the sub camera, the load on the CPU can be reduced. Furthermore, by setting the retention time of the face position information when the face detection result is not detected to be long, it is possible to prevent frequent switching of the presence / absence of the face position information.

(Other embodiments)
Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

109 First image sensor 110 Second image sensor 113 Image processing unit 117 System control unit

Claims (11)

Imaging means for imaging a subject and generating image data of a plurality of frames ;
The image data generated by the imaging means and detecting means for detecting a predetermined condition is satisfied subject region,
The image data generated by the imaging means, and a tracking means that performs tracking of high similarity with the detected object area by said detecting means region,
The imaging means includes a first imaging optical system, and a second imaging optical system facing said first direction different from the imaging optical system, generated from the previous SL first imaging optical system For the image data , both the detection of the subject area by the detection means and the tracking by the tracking means are performed. For the image data generated from the second imaging optical system, the detection of the subject area by the detection means is performed. An image pickup apparatus that performs detection but does not perform tracking by the tracking means . Storage means for storing the detection result of the detection means and the tracking result of the tracking means;
The imaging apparatus according to claim 1, further comprising a control unit configured to control the storage unit so as to hold the position information of the subject for a predetermined period. The control means sets a retention time of the subject position information in the image data generated from the second imaging optical system from a retention time of the subject position information in the image data generated from the first imaging optical system. The imaging device according to claim 2, wherein the imaging device is also made longer. The detection means detects the frequency of detection by the detection means in the image data generated from the second imaging optical system from the frequency of detection by the detection means in the image data generated from the first imaging optical system. The imaging device according to claim 1, wherein the imaging device is also lowered.   5. The imaging apparatus according to claim 1, wherein the first imaging optical system and the second imaging optical system collect external light from opposite directions.   The imaging apparatus according to claim 1, wherein the subject area is a face. Imaging means for imaging a subject and generating image data of a plurality of frames;
Detecting means for detecting a subject area that satisfies a predetermined condition in the image data generated by the imaging means;
In the image data generated by the imaging means, tracking means for tracking an area having a high similarity with the subject area detected by the detection means;
Control means for controlling the position information of the subject based on at least one of the detection result of the detection means and the tracking result of the tracking means so as to be held in the storage means for a certain period;
The imaging means includes a first imaging optical system and a second imaging optical system facing a different direction from the first imaging optical system,
The control means sets a retention time of the subject position information in the image data generated from the second imaging optical system from a retention time of the subject position information in the image data generated from the first imaging optical system. An imaging device characterized in that the length of the imaging device is also long. A first imaging optical system, said first and a second imaging optical system facing different directions imaging optical system, an imaging means for imaging an object and generates image data of a plurality of frames A method for controlling an imaging apparatus comprising:
The image data generated by the imaging unit, a detection step for detecting a predetermined condition is satisfied subject region,
In the image data generated by the imaging means , a tracking step of tracking a region having a high similarity with the subject region detected in the detection step ,
Before SL for image data generated from the first imaging optical system, thus both the tracking by detecting said tracking step of the subject region by the detection step, the second image generated from the imaging optical system A method for controlling an image pickup apparatus, wherein data is detected in the subject area in the detection step but not in the tracking step . An image pickup means comprising a first image pickup optical system and a second image pickup optical system oriented in a different direction from the first image pickup optical system, and picks up an image of a subject and generates a plurality of frames of image data. A control method for an imaging apparatus provided with
A detection step of detecting a subject area that satisfies a predetermined condition in the image data generated by the imaging means;
In the image data generated by the imaging means, a tracking step for tracking a region having a high similarity to the subject region detected in the detection step;
A control step for controlling the position information of the subject based on at least one of the detection result of the detection step and the tracking result of the tracking step so as to be held in the storage unit for a certain period of time;
In the control step, the retention time of the subject position information in the image data generated from the second imaging optical system is set as the retention time of the subject position information in the image data generated from the first imaging optical system. A method for controlling an imaging apparatus, characterized by being longer than A first imaging optical system, said first and a second imaging optical system facing different directions imaging optical system, an imaging means for imaging an object and generates image data of a plurality of frames A program for controlling an imaging apparatus comprising:
The image data generated by the imaging unit, a detection step for detecting a predetermined condition is satisfied subject region,
In the image data generated by the imaging means, the computer is caused to execute a tracking step for tracking a region having a high similarity with the subject region detected in the detection step ,
Before SL for image data generated from the first imaging optical system, thus both the tracking by detecting said tracking step of the subject region by the detection step, the second image generated from the imaging optical system As for data, a program that detects the subject area by the detection step but does not perform tracking by the tracking step . An image pickup means comprising a first image pickup optical system and a second image pickup optical system oriented in a different direction from the first image pickup optical system, and picks up an image of a subject and generates a plurality of frames of image data. A program for controlling an imaging device provided,
A detection step of detecting a subject area that satisfies a predetermined condition in the image data generated by the imaging means;
In the image data generated by the imaging means, a tracking step for tracking a region having a high similarity to the subject region detected in the detection step;
Causing the computer to execute a control step of controlling the position information of the subject based on at least one of the detection result of the detection step and the tracking result of the tracking step to be held in the storage unit for a certain period of time,
In the control step, the retention time of the subject position information in the image data generated from the second imaging optical system is set as the retention time of the subject position information in the image data generated from the first imaging optical system. A program characterized by making it longer.

Images