JP6489150B2 - Image processing apparatus and imaging apparatus - Google Patents

Description

The present invention relates to an image processing apparatus and an imaging apparatus .

  Conventionally, various techniques have been devised for so-called tracking in which a focus area used for focus adjustment is followed in accordance with the movement of a subject. As a technique related to this tracking, there is a technique for changing an AF distance measurement frame in accordance with the shape or size of a subject that moves toward or away from the camera (see Patent Document 1).

JP 2009-69748 A

  Patent Document 1 described above is an effective technique when focus adjustment is performed on a subject that moves in a direction that approaches or moves away from the camera. However, the movement of the subject does not only move toward or away from the camera. For this reason, a technique for switching a subject to be tracked according to a change in the subject within the angle of view, such as when the subject to be tracked is out of the shooting range (angle of view) or when another subject enters. Is not considered.

  The present invention has been invented in view of the above-described problems, and an object thereof is to easily and accurately switch a subject to be tracked.

The image processing apparatus of the present invention is an image processing apparatus that targets a specific subject as a tracking target, and is included in a subject included in the first image and a second image generated after the first image. A detection unit that detects a subject that is present, and the detection unit detects a first subject that is a tracking target in a first region that includes a central portion of the first image, and a central portion of the second image The first subject is not detected in the second region having a larger area than the first region , and includes the central portion of the second image, and the third region having a smaller area than the second region. When a different second object is detected as 1 subject, and a setting unit that sets the second object as the specific object is switched from the first object, the third region, the second image In the second region Included, the first region and the area of the first image, the shape and position corresponding.
The image processing apparatus according to the present invention is an image processing apparatus for tracking a specific subject, and is included in a subject included in the first image and a second image generated after the first image. A detection unit that detects a subject that is being tracked, and a first subject that is detected in a first region including a central portion of the first image by the detection unit, A third region including a central portion of the second image and not detected in a second region having a larger area than the first region, and including a central portion of the second image and having a smaller area than the second region; When the second object is detected in, and a setting unit that sets the second object as the specific object is switched from the first object, the third region, said in the second image a second Included in the region, said Wherein the first region and the area of one image, shape and position correspond.
The imaging apparatus of the present invention includes the above-described image processing apparatus.

  According to the present invention, it is possible to easily and accurately switch the subject to be tracked.

It is a functional block diagram which shows one form in the structure of an imaging device. It is a functional block diagram showing one form of an image processing circuit. It is a figure which shows an example of the area | region A and the area | region B at the time of mask evaluation set with respect to a through image. It is a flowchart which shows the flow of the process at the time of imaging. It is a flowchart which shows the flow of the process at the time of imaging. (A) is a figure which shows an example of a through image and a mask in case the to-be-photographed object is located in the center part of an angle of view, (b) is a figure which shows an example of the change of the evaluation value with respect to a mask. (A) is a figure which shows one form of a through image and a mask in case the to-be-photographed object is located in the peripheral part of a view angle, (b) is a figure which shows the change of the evaluation value with respect to a mask. (A) is a figure which shows one form of the through image and mask extracted when a to-be-photographed object moves to the center part of a view angle, (b) is a figure which shows the change of the evaluation value with respect to a mask. (A) is a figure which shows one form of a through image and a mask in case the subject located in the center part of an angle of view moves outside an angle of view, (b) is a figure which shows the change of the evaluation value with respect to a mask. . (A)-(c) shows one form of a through image and a mask in the case where a subject located at the center portion of the angle of view moves outside the angle of view and other subjects move toward the center portion of the angle of view. FIG. 4D is a diagram illustrating one form of a through image and a mask when a subject that has moved outside the angle of view moves again within the angle of view. It is a figure which shows the change of the evaluation value of each mask in Fig.10 (a)-(c). (A) is a diagram showing one form of a through image and a mask when a plurality of subjects are included in the angle of view, and (b) is a through image and a mask when two subjects among the plurality of subjects move. It is a figure which shows one form. It is a figure which shows the change of the evaluation value of each mask in FIG.

  Hereinafter, an embodiment of the imaging device will be described with reference to FIG. The imaging device 10 includes a device main body 11 and a lens barrel 12. Here, the lens barrel 12 may be integrated with the apparatus main body 11 or may be detachable.

  The apparatus main body 11 includes an imaging unit 21, an image processing circuit 22, a display unit 23, a buffer memory unit 24, a storage unit 25, a CPU 26, an operation unit 27, and a communication unit 28. The imaging unit 21, the image processing circuit 22, the display unit 23, the buffer memory unit 24, the storage unit 25, the CPU 26, the operation unit 27, and the communication unit 28 are electrically connected via a bus. The image processing apparatus of the present invention includes a CPU 26 and an image processing circuit 22.

  The imaging unit 21 includes an imaging element 29 and an A / D (Analog / Digital) conversion unit 30. The operation of the imaging unit 21 is controlled by the CPU 26 in accordance with the set imaging conditions (for example, exposure value, shutter speed, etc.).

  The imaging device 29 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 12 into an electric signal, and outputs the electric signal to the A / D conversion unit 30. The imaging element 29 is configured by a photoelectric conversion element such as a CMOS (Complementary Metal Oxide Semiconductor), for example. Here, the imaging element 29 may convert the optical image into an electric signal in the entire region of the photoelectric conversion surface, or may convert the optical image into an electric signal in a partial region of the photoelectric conversion surface. The electrical signal output from the image sensor 29 becomes image data.

  The A / D conversion unit 30 digitizes the image data output from the image sensor 29 and outputs the digitized image data to the buffer memory unit 24 or the like.

  The image processing circuit 22 performs image processing on the image data temporarily stored in the buffer memory unit 24 based on the image processing conditions stored in the storage unit 25. Here, examples of the image processing include white balance adjustment, interpolation processing, color tone correction processing, and gradation conversion processing. Image data subjected to image processing by the image processing circuit 22 is temporarily stored in the buffer memory unit 24 and also stored in the storage medium 31 via the communication unit 28.

  In addition, the image processing circuit 22 executes a process of setting a main subject region using image data continuously obtained by the imaging unit 21 in a state where no photographing instruction is performed. Here, the image data continuously obtained by the imaging unit 21 in a state where no photographing instruction is performed is referred to as through image data. The process for setting the main subject area will be described later.

  An example of the display unit 23 is a liquid crystal display. The display unit 23 includes a control unit, and display control of images on the display unit 23 is performed by the control unit. Here, examples of the image displayed on the display unit 23 include an image based on image data obtained by the imaging unit 21 and an image displayed when various settings are performed. The buffer memory unit 24 temporarily stores the image data generated by the imaging unit 21 and the image data subjected to image processing by the image processing circuit 22. The storage unit 25 stores imaging conditions, determination conditions referred to by the CPU 26 in various determinations, and the like.

  The CPU 26 comprehensively controls each unit of the apparatus main body 11 based on information acquired from the image processing circuit 22 and the storage unit 25 and the operation state of the operation unit 27. Control by the CPU 26 includes focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, flash emission amount change setting, subject tracking setting, and various shooting mode settings. , Various image processing settings, various display settings, brightness optimization settings linked to zoom magnification, and the like.

  The operation unit 27 includes a power switch, a shutter button, a multi selector (cross key), or other operation keys. When the display unit 23 is the display unit 23 including a touch panel, the touch panel also functions as the operation unit 27. The operation unit 27 receives a user operation input and outputs a signal corresponding to the operation input to the CPU 26.

  The communication unit 28 is electrically connected to a storage medium 31 that can be attached to and detached from the apparatus main body 11 such as a memory card including a nonvolatile memory. The communication unit 28 writes data (image data, area information, etc.) to the storage medium 31 attached to the apparatus main body 11 and erases the data stored in the storage medium 31. Further, the communication unit 28 reads data stored in the storage medium 31 as necessary.

  The lens barrel 12 includes an imaging optical system 35, a lens driving unit 36, an encoder 37, and a barrel control unit 38. The imaging optical system 35 has a plurality of lenses such as a focus lens and a zoom lens. Each lens of the imaging optical system 35 is controlled by the lens driving unit 36 so that the position thereof moves in the optical axis direction. The imaging optical system 35 guides the optical image captured on the light receiving surface (photoelectric conversion surface) of the imaging element 29 provided in the apparatus main body 11.

  The lens driving unit 36 includes a driving mechanism such as a driving motor, a gear mechanism, or a helicoid mechanism, and moves each lens of the imaging optical system 35 in the optical axis direction. When the zoom lens or focus lens of the imaging optical system 35 moves in the optical axis direction, the encoder 37 detects the position and moving direction accompanying the movement and outputs the detected position and moving direction to the lens barrel control unit 38.

  The lens barrel control unit 38 controls the lens driving unit 36 in accordance with a drive control signal input from the CPU 26 of the apparatus main body 11. Here, the drive control signal is a control signal for moving each lens of the imaging optical system 35 in the optical axis direction. The lens barrel control unit 38 controls the number of steps of the pulse voltage output to the lens driving unit 36 according to the drive control signal. Further, the lens barrel control unit 38 outputs the position (focus position) of the focus lens to the CPU 26 of the apparatus main body 11 based on a signal corresponding to the movement amount of each lens of the imaging optical system.

  Next, the configuration of the image processing circuit 22 will be described. As described above, the image processing circuit 22 executes processing for setting a main subject area using through image data, in addition to performing image processing on image data based on image processing conditions. As shown in FIG. 2, the image processing circuit 22 has functions of a detection unit 41, a determination unit 42, a tracking unit 43, an evaluation unit 44, and a setting unit 45.

  The detection unit 41 uses the through image data acquired by the imaging unit 21 to detect the area of the subject included in the image based on the through image data. Hereinafter, an image based on the through image data is referred to as a through image. Here, the process for detecting the subject area includes a mask extraction process which is one method for detecting the subject area based on the feature amount obtained from the image. By performing this mask extraction process, the detection unit 41 detects the area of the subject included in the through image. The detection unit 41 stores information relating to the mask (hereinafter referred to as mask information) in the storage unit 25 using the detected subject area as a mask. Here, the mask information includes extraction conditions for mask extraction processing (color classification, mask classification, etc.), mask position, mask size and shape, mask movement (movement direction, movement vector), and subject corresponding to the mask. Contains distance information.

  The determination unit 42 determines whether or not the same mask is continuously detected n times (n frames) or more using mask information obtained from each of the plurality of through images. By performing this determination, it is determined whether or not the same mask is stable.

  For example, when the optical zoom magnification in the imaging optical system 35 (not shown) is constant, a mask having the same color and the same size as the target mask is detected in each of the through image data of n or more consecutive frames. It is determined whether or not it has been done. In this case, the subject itself may be moving or may be stationary.

  In some cases, the optical zoom magnification in the imaging optical system 35 is changed, or the subject or the photographer changes in the optical axis direction of the imaging optical system 35 while the optical zoom magnification in the imaging optical system 35 is constant. In such a case, the size of the area of the detected subject changes in through images obtained continuously. Therefore, whether or not a mask having the same color as the target mask and changing in size at an enlargement rate or reduction rate that matches the optical zoom magnification is detected in each of through images of n or more consecutive frames is detected. What is necessary is just to judge.

  In the determination unit 42, it is determined whether or not the same mask is continuously detected n times (n frames) or more, but it is continuously detected at least twice (2 frames) or more. The mask can be determined to be stable. In addition, about the through image used as determination of whether the mask is stable, all the through images output from the imaging unit 21 may be targeted, and among the through images output from the imaging unit 21, for example, It is also possible to make the above determination using a target through image with a through image obtained at predetermined frame intervals, such as every three frames, as the target through image.

  The tracking unit 43 uses, for example, a template matching method using a mask set as the main subject region for the through image obtained after the main subject region is set, and includes the main image included in the acquired through image. The position of the subject area is specified. By performing this process on each of the through images, the main subject can be tracked.

  The evaluation unit 44 performs an evaluation on a mask (including a mask set as a main subject region) that is continuously detected n times (n frames) or more in the determination unit 42. Here, the evaluation unit 44 evaluates the mask using the mask information stored in the storage unit 25.

  The setting unit 45 sets a main subject area based on the evaluation result in the evaluation unit 44. Here, as a process of setting the main subject area, in addition to setting a new main subject area when the main subject area is not set, the main subject area is set when the main subject area is set. A process of switching (resetting) the area of the subject to be an area is included. The setting unit 45 writes the set information in the storage unit 25 when the main subject area is set. This information may be stored in association with the mask information or may be stored individually.

  Next, evaluation in the evaluation unit 44 will be described. Hereinafter, the case where the position information of each mask is used as mask information used in the evaluation of the evaluation unit 44 will be described.

  The evaluation unit 44 obtains an evaluation value for a mask that is continuously detected n times (n frames) or more. The evaluation value is higher as the position of the target mask is closer to the center of the through image, and the evaluation value is lower as it is closer to the peripheral edge of the through image. After obtaining the evaluation value of the target mask, the evaluation unit 44 evaluates the evaluation value of the target mask. Using the result of this evaluation, the setting unit 45 sets a main subject area.

  The above-described evaluation is executed by comparing the evaluation value obtained for each mask with the values (thresholds Ta and Tb described later) for the areas A and B set for the through image TI. The

As shown in FIG. 3, the region A has a horizontal length W ₁ and a vertical length H _{1 that} are α (the horizontal length W ₀ and the vertical length H ₀ of the through image TI. α = 0.5 to 0.6) times. This area A is an area set based on the concept that if the subject corresponding to the target mask is the main subject, it is located in the center area of the angle of view. Therefore, the above-described threshold value Ta is the minimum value of the evaluation value of the mask that can be evaluated when the region A includes the mask.

The area B, the horizontal length W ₂ and vertical length H ₂ of lateral through-image length W ₀ and vertical length H ₀ β (β = 0.8~ 0.9) times. Here, the region B is not necessarily set, but is set in consideration of lens performance of the imaging optical system 35 such as lens aberration in the imaging optical system 35. This area B is an area used when evaluating whether or not the main subject has deviated from the angle of view. That is, the threshold value Tb is the minimum value of the evaluation value of the mask that can be evaluated when the region B includes the mask.

  Note that the values of the threshold value Ta and the threshold value Tb may vary depending on the shape and size of the mask, or may be fixed. Further, the values of α and β described above are not limited to the above, and may be set as appropriate.

  In the present embodiment, a case where position information of each mask is used as mask information used in the evaluation by the evaluation unit 44 will be described. However, either mask movement information or distance information to a subject corresponding to the mask is used. May be. For example, when using mask movement information and distance information, a mask having a high evaluation among the subjects included in the region A described above may be set as the main subject. Furthermore, it is also possible to calculate an evaluation value by using at least two pieces of information of the position information of the mask stored as mask information, the movement information of the mask or the distance information to the subject corresponding to the mask, or all of these pieces of information. is there.

  Next, the flow of processing during shooting will be described with reference to the flowcharts of FIGS. This flowchart is executed when the imaging apparatus is in an operation mode in which shooting can be performed.

  Step S101 is processing for obtaining through image data. The CPU 26 controls the imaging unit 21 to start obtaining through image data. The acquired image information of the through image data is temporarily stored in the buffer memory unit 24. This through image data is acquired continuously at predetermined time intervals.

  Step S102 is image processing. The CPU 26 instructs the image processing circuit 22 to perform image processing. In response to this instruction, the image processing circuit 22 reads through image data stored in the buffer memory unit 24. Then, the image processing circuit 22 performs image processing such as white balance adjustment, interpolation processing, color tone correction processing, and gradation conversion processing on the read through image data. After these image processes are executed, the image processing circuit 22 temporarily stores the through image data in the buffer memory unit 24.

  Step S103 is processing to display a through image. In the process of step S <b> 103, the CPU 26 reads through image data temporarily stored in the buffer memory unit 24 and outputs it to the display unit 23. In response to this, the display unit 23 displays an image based on the read through image data. Thereby, a through image is displayed.

  Step S104 is a mask extraction process. The CPU 26 instructs the image processing circuit 22 to set the main subject area. In response to this instruction, the image processing circuit 22 reads through image data from the buffer memory unit 24. Then, the image processing circuit 22 performs mask extraction processing using the read through image data. By executing the mask extraction process, a subject area is detected, and the detected subject area is extracted as a mask.

  Step S105 is processing to determine whether or not a mask has been extracted. If the subject area is detected by the processing in step S104 and the subject area is extracted as a mask, the CPU 26 sets the determination result in step S105 to Yes. In this case, the process proceeds to step S106. On the other hand, when the subject area cannot be detected in the process of step S104, no mask is extracted, so the CPU 26 sets the determination result of step S105 to No. In this case, the process proceeds to step S112.

  Step S106 is processing to record mask information. The CPU 26 records information relating to the mask extracted in step S104 in the storage unit 25 or the like as mask information.

  Step S107 is processing for determining whether or not there is a mask extracted n times or more continuously. The determination in step S107 is for determining whether or not the mask extracted by the mask extraction process in step S104 is stable. Therefore, if the same mask is extracted n times or more consecutively, the CPU 26 determines that the extracted mask is stable. That is, it can be determined that the main subject area is correctly detected. In this case, the CPU 26 sets the determination result in step S107 to Yes, and proceeds to step S108. On the other hand, if the same mask has not been extracted n times or more consecutively, the CPU 26 determines that the mask extracted by the mask extraction process in step S104 is not stable. In this case, the CPU 26 sets the determination result in step S107 to No, and proceeds to step S112. The CPU 26 records the determination result of step S107 in the storage unit 25.

  Step S108 is processing for evaluating the extracted mask. The image processing circuit 22 uses a mask extracted continuously n times (n frames) or more as a mask as a candidate for the main subject region. Then, the image processing circuit 22 reads out mask information corresponding to a mask that is a candidate for the main subject area from the storage unit 25. Then, the image processing circuit 22 obtains an evaluation value for the target mask based on the read mask information. The image processing circuit 22 evaluates the target mask by comparing the obtained evaluation value with the threshold value Ta.

  Step S109 is processing for setting a main subject area. The image processing circuit 22 sets a main subject region using the evaluation result in step S108.

  Step S110 is processing for storing information relating to the main subject area. The image processing circuit 22 records information on the main subject area in the storage unit 25 and the like. The information on the main subject area includes mask information related to the mask used, information on the position of the main subject area, and the size and shape of the main subject area.

  Step S111 is processing for displaying main subject area information. The CPU 26 outputs information on the main subject area to the display unit 23. The display unit 23 displays a display based on the output information on the main subject area so as to be superimposed on the through image. Examples of the display based on the information on the main subject area include a frame indicating that it is the main subject area. This frame may be a frame according to the shape of the main subject area, or may be a frame having a predetermined shape such as a rectangle or a circle. By performing such a display, the user can easily grasp what the position and shape of the main subject area detected automatically are.

  Step S112 is processing to determine whether or not there is a shooting instruction. When an input operation using the operation unit 27 is performed, a signal based on the input operation in the operation unit 27 is input to the CPU 26. The CPU 26 determines whether or not the input signal is a signal indicating a shooting instruction. If the input signal is a signal indicating a shooting instruction, the CPU 26 sets the determination result in step S112 to Yes. In this case, the process proceeds to step S113. On the other hand, if the input signal is not a signal indicating a shooting instruction, the CPU 26 sets the determination result in step S112 to No. In this case, the process proceeds to step S120.

  Step S113 is an imaging process. For example, when the main subject area is set in step S109, the CPU 26 sets focus adjustment (AF) setting processing, exposure adjustment (AE) setting processing, white balance adjustment processing (AWB) based on the set main subject area information. 3A) and the conditions for various image processing in the image processing circuit 22 are determined. If the main subject area is not set in step S113, the CPU 26 uses a specific area such as a central area, for example, a focus adjustment (AF) setting process, an exposure adjustment (AE) setting process, and a white balance. While performing 3A processing of adjustment processing (AWB), conditions for various image processing in the image processing circuit 22 are determined. And CPU26 performs an imaging process based on the result of the process mentioned above.

  Step S114 is processing to record an image. CPU26 records the image data acquired by the imaging process by step S113 to the storage medium 31 via the communication part 28, and complete | finishes a series of processes.

  If it is determined in the determination process of step S112 described above that there is no shooting instruction, the process proceeds to step S120.

  Step S120 is processing for acquiring through image data. The process of step S120 is the same process as step S101.

  Step S121 is image processing. The process of step S121 is the same process as step S102.

  Step S122 is processing to display a through image. The process of step S122 is the same process as step S103.

  Step S123 is processing for determining whether or not a main subject area is set. The CPU 26 refers to the storage unit 25. If the information on the main subject area is stored in the storage unit 25, the CPU 26 sets the determination result in step S123 to Yes. In this case, the process of step S124 and step S125 and the process of step S126 to step S128 are performed simultaneously in parallel.

  On the other hand, if the information on the main subject area is not stored in the storage unit 25, the CPU 26 sets the determination result in step S123 to No. In this case, the process returns to step S104. That is, in this case, the mask extraction process for setting the main subject area is executed again.

  Step S124 is a tracking process. The CPU 26 instructs the image processing circuit 22 to perform tracking processing. In response to this, the image processing circuit 22 reads through image data from the buffer memory unit 24. Further, the image processing circuit 22 reads information on the main subject area from the storage unit 25. Then, the image processing circuit 22 identifies the main subject region by using a method such as template matching using the read image data and information.

  Step S125 is processing to update information on the main subject area. The image processing circuit 22 stores the information related to the main subject area identified in step S124 in the storage unit 25 as new main subject area information. Here, the information on the main subject area newly obtained in step S124 may be stored in place of the information on the main subject area stored in advance, or may be newly obtained in step S124. The information on the subject area may be added to the information on the main subject area previously stored.

  As described above, the processing from step S126 to step S128 is executed concurrently with the processing of step S124 and step S125.

  Step S126 is a mask extraction process. Note that the process of step S126 is the same as the process of step S104.

  Step S127 is processing for determining whether or not a new mask has been extracted. The CPU 26 reads out the mask information stored in the storage unit 25, and determines whether or not there is a newly extracted mask among the masks obtained by the mask extraction process in step S126. When there is a newly extracted mask, the CPU 26 determines Yes in step S128. In this case, the process proceeds to step S128. On the other hand, when there is no newly extracted mask, the CPU 26 sets the determination result in step S128 to No. In this case, the process proceeds to step S129.

  Step S128 is a process for recording new mask information. The CPU 26 records information related to the new mask acquired in step S126 in the storage unit 25.

  Step S129 is processing for determining whether or not there is a mask extracted continuously n times or more in addition to the mask set in the main subject area. When the same mask is extracted n times or more continuously, the CPU 26 sets the determination result in step S129 to Yes. In this case, the process proceeds to step S130. On the other hand, if the same mask is not extracted continuously n times or more, the CPU 26 sets the determination result in step S129 to No. In this case, the process proceeds to step S131.

  Step S130 is processing for evaluating a target mask including the mask set as the main subject region. The image processing circuit 22 reads target mask information from the storage unit 25. Then, the image processing circuit 22 evaluates each of the target masks using the read mask information. In the evaluation in step S130, an evaluation using the threshold value Tb is performed on the mask corresponding to the subject area set as the main subject area. In addition, evaluation using a threshold Ta is performed for masks corresponding to other subject areas.

  If the determination result in step S129 is No, the process proceeds to step S131.

  Step S131 is processing for evaluating the mask set as the main subject region. The image processing circuit 22 reads information on the main subject area stored in the storage unit 25. Then, the image processing circuit 22 evaluates the mask serving as the main subject region using the read information on the main subject region. In this case, the image processing circuit 22 performs evaluation using the threshold value Tb.

  Step S132 is processing for determining whether or not to reset the main subject area. The image processing circuit 22 determines whether to reset the main subject area using the mask evaluation result executed in either step S130 or step S131. For example, if the mask set in the main subject area and other masks are evaluated in step S130, the main subject is determined from the evaluation results of the mask set in the main subject area and the evaluation results of the other masks. It is determined whether or not to reset the mask set as the area. Note that resetting the mask in this case refers to switching the mask. When resetting the mask set as the main subject area, the image processing circuit 22 sets the determination result in step S132 to Yes. In this case, the process proceeds to step S133. On the other hand, when it is not necessary to reset the mask set as the main subject area, the image processing circuit 22 sets the determination result in step S132 to No. In this case, the process proceeds to step S135.

  On the other hand, if the mask set in the main subject area is evaluated in step S131, whether or not the mask set as the main subject area is reset based on the evaluation of the mask set in the main subject area. Determine. The resetting in this case refers to canceling the setting of the mask that becomes the main subject area. Also in this case, when resetting the mask set as the main subject region, the image processing circuit 22 sets the determination result in step S132 to Yes. In this case, the process proceeds to step S133. On the other hand, when it is not necessary to reset the mask set as the main subject area, the image processing circuit 22 sets the determination result in step S132 to No. In this case, the process proceeds to step S135.

  Step S133 is processing for resetting the main subject area. The image processing circuit 22 resets the main subject area. For example, if the mask set in the main subject area and other masks are evaluated in step S130, the image processing circuit 22 cancels the mask setting as the main subject area and applies to other masks. Set the main subject area. In step S131, when the mask set in the main subject area is being evaluated, the image processing circuit 22 cancels the mask setting as the main subject area.

  Step S134 is processing to update main subject information. The image processing circuit 22 updates the main subject area information recorded in the storage unit 25 based on the result of resetting the main subject area executed in step S133.

  Step S135 is processing for determining whether or not there is a mask set for the main subject. The CPU 26 refers to the storage unit 25. If there is main subject area information in the storage unit 25, the CPU 26 sets the determination result in step S135 to Yes. In this case, the process proceeds to step S136. On the other hand, when there is no main subject area information in the storage unit 25, the CPU 26 sets the determination result in step S135 to No. In this case, the process returns to step S112. An example of the case where there is no main subject area information in the storage unit 25 is a case where the setting for the main subject area is canceled in the resetting of the main subject area in step S133.

  Step S136 is processing for displaying main subject information. This is the same process as that in step S111. When the process of step S136 is executed, the process returns to step S112.

  In this way, when the main subject region is not set, the subject region detected based on the feature amount obtained from the acquired through image data is extracted as a mask. This process is performed on a plurality of through image data, and masks extracted from the plurality of through image data are compared. This comparison determines whether the detected mask is stable. When it is determined that the mask is stable, the mask is evaluated, and the subject area corresponding to the mask with good evaluation is set as the main subject area.

  In general, when photographing a subject, the imaging device 10 is held so that the subject is located at the center of the angle of view. Therefore, as illustrated in FIG. 6A, a region of the subject located in the central portion (for example, region A) is extracted as a mask 51 from the through image TI obtained by the imaging device 10. Here, when shooting a still subject, the angle of view is fixed unless the position or orientation of the imaging device 10 changes or the imaging magnification (optical zoom magnification) in the imaging optical system 35 of the imaging device 10 is changed. The Therefore, the position of the subject with respect to the angle of view is the same position. That is, the same mask 51 is always detected from a plurality of through images TI obtained in succession, and therefore it is determined that the mask 51 is stable. Further, since the mask 51 is located inside the region A, the evaluation value T for the mask 51 is a constant value and exceeds the threshold value Ta (see FIG. 6B). Therefore, in such a case, the subject area corresponding to the mask 51 is set as the main subject area.

  On the other hand, depending on the composition, a stationary subject may be photographed not at the center of the angle of view but at the periphery of the angle of view. As shown in FIG. 7A, from the through image TI obtained by the imaging device 10, a region of the subject at a position shifted from the central portion is extracted as a mask 52. In this case, since the extracted mask 52 is detected from each of the plurality of through images TI, it is determined as a stable mask. However, the mask 52 is at a position outside the region A. Therefore, the evaluation value T for the mask 52 is a constant value, but is not more than the threshold value Ta (see FIG. 7B). Therefore, the subject area corresponding to the mask 52 is not set as the main subject area.

  On the other hand, there are cases where the moving subject enters within the angle of view of the imaging device 10 or deviates from the angle of view of the imaging device 10.

  First, consider a case where a moving subject enters an angle of view. As shown in FIG. 8A, when shooting a moving subject, the region of the subject detected from the plurality of through images TI, in other words, the extracted mask 53 moves in the X1 direction in the drawing, and the angle of view. Reach the middle part of. Here, since the masks 53 obtained from the plurality of through images TI are extracted as being the same mask, it is determined that the masks 53 are stable.

  Here, as shown in FIG. 8B, the evaluation value for the mask 53 gradually increases. When the mask 53 reaches the central portion of the through image TI, in other words, enters the region A, the evaluation value T exceeds the threshold value Ta. Therefore, after the mask 53 enters the central portion (region A) of the through image TI, the subject region corresponding to the mask 53 is set as the main subject region.

  Next, the subject located at the center of the angle of view may move. In this case, as shown in FIG. 9A, when the subject is located at the center of the angle of view, the subject is detected from a plurality of through images TI, and the subject areas are respectively used as masks 54. Detected. Here, since the mask is located in the region A of the through image TI, the evaluation value T for the mask 54 is a constant value and exceeds the threshold value Ta. Therefore, the subject area corresponding to the mask 54 is set as the main subject area.

  Thereafter, when the subject moves, the position of the mask 54 extracted from the plurality of through images TI changes in the X2 direction. As described above, the evaluation value for the mask becomes lower as it approaches the periphery of the through image TI. Therefore, as shown in FIG. 9B, the evaluation value T for the mask 54 moving in the X2 direction gradually decreases. Therefore, the evaluation value T is equal to or less than the threshold value Tb indicating that the angle of view is deviated. When the evaluation value T is equal to or less than the threshold value Tb, the setting of the main subject area for the subject area corresponding to the mask 54 is cancelled.

  For example, when the photographing operation is not executed after the main subject area is set, the through image TI is continuously acquired. Also in this case, it is determined whether the mask extracted from the through image TI is stable and the mask is evaluated. In this process, if there is a mask newly extracted from the through image TI, it is determined whether or not each newly extracted mask is stable and the mask is evaluated. For a mask based on the subject area set as the main subject area, the tracking process is executed, and then the mask detected by the tracking process is evaluated. Therefore, based on these evaluations, resetting of the main subject area is executed as necessary.

  For example, a case will be described in which a subject located at the center portion of the angle of view moves while another subject moves toward the center portion of the angle of view. In this case, as shown in FIG. 10A, the mask 55 based on the subject area set as the main subject area moves in the X3 direction in the figure. At the same time, the mask 56 based on the area of the other subject moves in the X3 direction in the figure. Therefore, as shown in FIG. 10B, the mask 55 set in the main subject region gradually disappears from the through image TI, and the mask 56 based on the regions of other subjects gradually becomes the through image TI. Approach the center part. Then, as shown in FIG. 10C, only the mask 56 based on the area of the other subject is included in the through image TI.

  FIG. 11 is a graph showing changes in evaluation values of the evaluation value T1 for the mask 55 and the evaluation value T2 for the mask 56. In FIG. The evaluation value T1 for the mask 55 is indicated by “◯”, and the evaluation value T2 for the mask 56 is indicated by “Δ” in the figure.

  Evaluation values T1 and T2 of the respective masks indicated by “t1” in FIG. 11 are the evaluation values of the masks in the through image TI illustrated in FIG. That is, in the through image TI shown in FIG. 10A, only the evaluation value T1 for the mask 55 is obtained. Here, since the mask 55 is located in the area A in the through image TI, the evaluation value T1 for the mask 55 exceeds the threshold value Ta. Therefore, the subject area corresponding to the mask 55 is set as the main subject area.

  Evaluation values T1 and T2 of the respective masks indicated by “t2” in FIG. 11 are mask evaluation values in the through image TI shown in FIG. That is, in the through image TI shown in FIG. 10B, in addition to the mask 55, the mask 56 is also extracted. Therefore, the evaluation values for these masks are obtained respectively. In this case, the evaluation value T1 for the mask 55 is equal to or less than the threshold value Tb, and the evaluation value T2 for the mask 56 exceeds the threshold value Ta. In this case, the setting of the subject area corresponding to the mask 55 is canceled as the main subject area. At the same time, the subject area corresponding to the mask 56 is set as the main subject area. Therefore, the subject area set as the main subject area is switched.

  Mask evaluation values T1 and T2 indicated by “t3” in FIG. 11 are mask evaluation values in the through image TI shown in FIG. In the through image TI shown in FIG. 10C, only the evaluation value T2 for the mask 56 is obtained. Here, since the mask 56 is located in the region A in the through image TI, the evaluation value T2 for the mask 56 exceeds the threshold Ta. Therefore, in the through image TI obtained thereafter, if the evaluation value T2 of the mask 56 exceeds the threshold value Tb, the subject area corresponding to the mask 56 continues to be the main subject area. As described above, even when the subject detected from the through image TI is changed to a different subject, the main subject region can be easily and accurately reset.

  Here, as shown in FIG. 10 (d), the region of the subject that is not detected from the through image TI (view angle) (the mask 55 in FIG. 10) is detected again by the through image TI obtained after a predetermined time has elapsed. May be. In such a case, the subject area set as the main subject area may be changed (switched) regardless of the evaluation value, and the subject area set as the main subject area at that time Both of the areas with the re-detected object area may be set as the main object area. In these cases, the mask information based on the subject area set as the main subject area may be stored in the storage unit 25 without being erased.

  In some cases, a plurality of subject areas extracted from the through image TI are detected. In such a case, the area of the subject corresponding to the mask having the highest evaluation value among the masks whose evaluation value exceeds the threshold value Ta is set as the main subject area. A case where masks 58, 59, and 60 are extracted from the through image TI as shown in FIG. At this time, as shown in FIG. 12B, the subject corresponding to the mask 58 moves in a direction deviating from the angle of view (X4 direction in the figure), and the subject corresponding to the mask 59 moves in the X5 direction in the figure. Suppose that it stops at the center of the angle of view. Further, it is assumed that the subject corresponding to the mask 60 is stationary.

  FIG. 13 is a graph showing changes in evaluation values for the masks 58, 59 and 60. Here, “◯” in FIG. 13 indicates the evaluation value T 3 for the mask 58, “Δ” indicates the evaluation value T 4 for the mask 59, and “は” indicates the evaluation value T 5 for the mask 60.

  As shown in FIG. 13, the evaluation value T3 for the mask 58 decreases with time, and the evaluation value is not calculated when the evaluation value T3 deviates from the angle of view. Further, the evaluation value T4 for the mask 59 increases to the evaluation value T4 by moving to the center portion of the angle of view. On the other hand, since the mask 60 corresponds to a stationary subject, the evaluation value T5 is a constant value.

  In this case, in the mask 58 and the mask 59 that are evaluation values exceeding the threshold Ta, the evaluation value T3 for the mask 58 is higher than the evaluation value T4 for the mask 59. Set as the main subject area.

  As described above, the mask 58 moves in a direction in which the corresponding subject deviates from the angle of view, so the evaluation value T3 for the mask 58 is low. On the other hand, the mask 59 has a higher evaluation value T4 for the mask 59 because the corresponding subject moves toward the center of the angle of view. When the evaluation value T3 for the mask 58 is equal to or less than the threshold value Tb, the evaluation value T5 for the mask 59 exceeds the threshold value Ta, so that the subject region set as the main subject region is switched. That is, the subject region set as the main subject region is switched from the subject region corresponding to the mask 58 to the subject region corresponding to the mask 59.

  Thus, even when a plurality of subjects move within the angle of view, the main subject region can be reset easily and accurately.

  In this embodiment, the case where the optical zoom magnification in the imaging optical system 35 is made constant is described. However, the present invention is not limited to this, and the optical zoom magnification in the imaging optical system 35 is changed. Also, the present invention can be applied. That is, when the optical zoom magnification is changed, the angle of view changes. For example, when the optical zoom magnification is increased, the actual shooting range included in the angle of view is narrowed. Therefore, the subject area set as the intended main subject area may deviate from the angle of view. In this case, even if the user does not intend, the area of another subject included in the angle of view is automatically set as the main subject area.

  On the other hand, when the optical zoom magnification is lowered, the actual shooting range included in the angle of view is widened, so that not only the subject intended by the user but also other subjects are included in the angle of view. In this case, even if the intended subject area is set as the main subject area, the evaluation of other subject areas may be high. In such a case, the main subject area is reset from the subject area intended by the user to another subject area. In this way, the main subject area switching in the through image TI can be easily and accurately performed. Note that, when the main subject area is changed to a main subject area by resetting the main subject area, the angle of view, orientation, and the like of the imaging apparatus may be adjusted.

  In the present embodiment, the main subject region is set by performing an evaluation on a mask determined to be stable. However, it is not necessary to be limited to this. For example, in the case of an imaging apparatus including a display unit including a touch panel as the display unit, the touch panel functions as the operation unit. In this case, the subject is detected from the acquired through image TI, and the result is superimposed on the through image TI and displayed. This display may be performed using a frame or the like for the detected subject area, for example. Then, when there is a subject intended by the user in the displayed through image TI, a touch operation on a portion corresponding to the region of the subject is performed on the touch panel. When the touch operation on the touch panel is performed, the area of the subject at the operated position may be set as the main subject area.

  Further, in the case of a display unit that does not include a touch panel, the frame for the subject region of the selection candidate is displayed with a display different from the other frames among the superimposed frames. Then, the display of the frame for the subject area to be selected is switched to another subject area by an operation of the cross key or the like. It is also possible to set the main subject region by performing an operation intended to be selected when the frame for the subject region of the selection candidate is moved to the intended subject region.

  In this embodiment, the imaging apparatus is taken as an example, but the present invention is not limited to this. For example, a portable information terminal such as a mobile phone or a smartphone having a camera function, or a camera including a camera and a computer. The present invention can be applied even to a system. In addition, an image processing apparatus that executes the processes of the flowcharts of FIGS. 4 and 5 by inputting the acquired image data may be used. In this case, the processing of step S101 and step S120 may be omitted.

  In addition, an image processing program capable of causing a computer to execute the processes of the flowcharts in FIGS. 4 and 5 may be used. In this case, the image processing program is preferably stored in a computer-readable storage medium such as a memory card, an optical disk, or a magnetic disk.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 21 ... Imaging part, 22 ... Image processing circuit, 23 ... Display part, 26 ... CPU, 41 ... Detection part, 42 ... Determination part, 43 ... Tracking part, 44 ... Evaluation part, 45 ... Setting part

Claims (10)

An image processing apparatus for tracking a specific subject,
A detection unit for detecting a subject included in the first image and a subject included in the second image generated after the first image;
The detection unit detects a first subject that is a tracking target in a first region that includes a central portion of the first image, and includes a central portion of the second image and has an area larger than that of the first region. The first subject is not detected in the second region having a large size, and a second subject different from the first subject is detected in the third region including the central portion of the second image and having a smaller area than the second region. Once, and a setting unit that sets the second object as the specific object is switched from the first object,
The third area is included in the second area in the second image, and the area, shape, and position of the first area correspond to the first area of the first image . An image processing apparatus for tracking a specific subject,
A detection unit for detecting a subject included in the first image and a subject included in the second image generated after the first image;
The first subject to be tracked , detected in the first region including the central part of the first image by the detection unit , includes the central part of the second image by the detection unit , and the first not detected in a large second region area than the area, and comprises a central portion of the second image, the second object is detected in the small third region area than the second region, the second the second object is switched from 1 object and a setting unit configured to set as the specific subject,
The third area is included in the second area in the second image, and the area, shape, and position of the first area correspond to the first area of the first image . The detection unit detects a subject included in the first image and a subject included in the second image based on a feature amount obtained from the first image and a feature amount obtained from the second image. The image processing apparatus according to claim 1 or 2 . An imaging device comprising the image processing device according to any one of claims 1 to 3. An image sensor that images a subject and outputs image data is provided.
  The detection unit includes a subject included in the first image generated by the first image data output from the image sensor, and a second image data output from the image sensor after the first image data. The imaging apparatus according to claim 4, wherein a subject included in the generated second image is detected. The imaging element has a photoelectric conversion element that converts light from an optical system including a focus lens into electric charge,
The imaging apparatus according to claim 5, further comprising a control unit that drives the focus lens based on information about the second subject set as the specific subject by the setting unit . The imaging apparatus according to claim 6, wherein the control unit causes the display unit to display an image indicating the specific subject set by the setting unit. The imaging apparatus according to claim 7, wherein the image is displayed so as to surround at least a part of the specific subject set by the setting unit. The imaging device according to any one of claims 6 to 8 , wherein the control unit performs exposure adjustment based on information on the second subject set as the specific subject by the setting unit. Wherein the control unit, the imaging apparatus according to any one of claims 9 claims 6 to perform white balance adjustment on the basis of the second object of the information set as a specific subject by the setting unit.

Images