JP2018116723A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a processing load, while maintaining accuracy or stability of extraction, when performing extraction of a main subject area continuously.SOLUTION: An image processing system includes an acquisition part for acquiring successively a plurality of image data generated temporally continuously, an area detection part for calculating a feature quantity of an image shown by image data acquired by the acquisition part, and detecting a subject area based on the feature quantity, and a tracking part for detecting a tracking object from a plurality of images, by using a detected subject area as the tracking object, when detecting the subject area by the area detection part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus.

  Conventionally, various techniques relating to subject extraction have been considered. For example, in the invention of Patent Document 1, an imaging device that automatically adjusts an automatic focus on a subject even when the subject moves in the camera direction by deforming an AF distance measurement frame according to the amount of change in the subject shape. It is disclosed.

JP 2009-069748 A

  Since the imaging device disclosed in Patent Document 1 deforms the AF distance measurement frame in accordance with the amount of change in the subject shape, a favorable effect can be obtained when the subject moves in the camera direction. However, there is a problem that the processing load increases according to the accuracy.

  The present invention has been made in view of the above points, and an object of the present invention is to reduce the processing load while maintaining the accuracy and stability of extraction when the main subject region is continuously extracted.

  The image processing apparatus of the present invention calculates an image feature amount indicated by the image data acquired by the acquisition unit that sequentially acquires a plurality of image data generated successively in time, and the acquisition unit, A region detection unit that detects a subject region based on a feature amount, and a tracking unit that detects the subject region from the plurality of images with the detected subject region as a tracking target when the region detection unit detects the subject region With.

  According to the present invention, when the main subject area is continuously extracted, the processing load can be reduced while maintaining the accuracy and stability of the extraction.

2 is a block diagram illustrating configurations of a lens barrel 10, an imaging device 20, and a storage medium 40. FIG. It is a flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is another flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is another flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution. It is another flowchart which shows operation | movement of CPU26 at the time of automatic detection mode execution.

<First Embodiment>
Hereinafter, a first embodiment will be described in detail with reference to the drawings.

  In the first embodiment, an apparatus including a lens barrel 10, an imaging device 20, and a recording medium 40 as illustrated in FIG. 1 will be described as an example.

  The imaging device 20 captures an optical image incident from the lens barrel 10. The obtained image is stored in the storage medium 40 as a still image or a moving image.

  The lens barrel 10 includes a focus adjustment lens (hereinafter referred to as an “AF (Auto Focus) lens”) 11, a lens driving unit 12, an AF encoder 13, and a lens barrel control unit 14. The lens barrel 10 may be detachably connected to the imaging device 20 or may be integrated with the imaging device 20.

  The imaging device 20 includes an imaging unit 21, an image processing device 22, a display unit 23, a buffer memory unit 24, a storage unit 25, a CPU 26, an operation unit 27, and a communication unit 28. The imaging unit 21 includes an imaging element 29 and an A / D (Analog / Digital) conversion unit 30. The imaging unit 21 is controlled by the CPU 26 in accordance with the set imaging conditions (for example, aperture value, exposure value, etc.).

  In the lens barrel 10, the AF lens 11 is driven by the lens driving unit 12 and guides an optical image to the light receiving surface (photoelectric conversion surface) of the imaging element 29 of the imaging device 20. The AF encoder 13 detects the movement of the AF lens 11 and outputs a signal corresponding to the movement amount of the AF lens 11 to the lens barrel control unit 14. Here, the signal corresponding to the movement amount of the AF lens 11 may be, for example, a sine wave signal whose phase changes according to the movement amount of the AF lens 11.

  The lens barrel control unit 14 controls the lens driving unit 12 in accordance with a drive control signal input from the CPU 26 of the imaging device 20. Here, the drive control signal is a control signal for driving the AF lens 11 in the optical axis direction. The lens barrel control unit 14 changes, for example, the number of steps of the pulse voltage output to the lens driving unit 12 according to the drive control signal. Further, the lens barrel control unit 14 outputs the position (focus position) of the AF lens 11 in the lens barrel 10 to the CPU 26 of the imaging device 20 based on a signal corresponding to the movement amount of the AF lens 11. Here, the lens barrel control unit 14 integrates, for example, signals according to the movement amount of the AF lens 11 according to the movement direction of the AF lens 11, thereby moving the AF lens 11 in the lens barrel 10 ( Position) may be calculated. The lens driving unit 12 drives the AF lens 11 according to the control of the lens barrel control unit 14 and moves the AF lens 11 in the optical axis direction within the lens barrel 10.

  In the imaging device 20, the imaging element 29 includes a photoelectric conversion surface, converts an optical image formed on the photoelectric conversion surface by the lens barrel 10 (optical system) into an electric signal, and supplies the electric signal to the A / D conversion unit 30. Output. The imaging element 29 is configured by a photoelectric conversion element such as a CMOS (Complementary Metal Oxide Semiconductor), for example. Further, the image sensor 29 may convert an optical image into an electric signal for a partial region of the photoelectric conversion surface (image cutout). Further, the image sensor 29 outputs an image obtained when a photographing instruction from the user is received via the operation unit 27 to the storage medium 40 via the A / D conversion unit 30 and the communication unit 28. On the other hand, the image pickup device 29 converts the continuously obtained image into a through image and converts it into a buffer memory unit 24 and the display unit 23 in the state before accepting a shooting instruction from the user via the operation unit 27. The data is output via the unit 30.

  The A / D conversion unit 30 digitizes the electrical signal converted by the image sensor 29 and outputs an image, which is a digital signal, to the buffer memory unit 24 and the like.

  The image processing device 22 performs image processing on the image temporarily stored in the buffer memory unit 24 based on the image processing conditions stored in the storage unit 25. The image after image processing is stored in the storage medium 40 via the communication unit 28. Further, the image processing device 22 performs mask extraction processing on the image temporarily stored in the buffer memory unit 24 (details will be described later). The extracted mask information is output to the CPU 26 and stored in the storage unit 25, the storage medium 40, and the like.

  The display unit 23 is, for example, a liquid crystal display, and displays an image generated by the imaging unit 21, an operation screen, and the like. The buffer memory unit 24 temporarily stores the image generated by the imaging unit 21. The storage unit 25 stores imaging conditions, determination conditions referred to by the CPU 26 in various determinations, and the like.

  The CPU 26 appropriately acquires necessary information from the image processing unit 22, the storage unit 25, and the like, and comprehensively controls each unit in the imaging device 20 based on the acquired information. Control by the CPU 26 includes focus adjustment (AF) setting, exposure adjustment (AE) setting, white balance adjustment (AWB) setting, flash emission amount change setting, subject tracking setting, and various shooting mode settings. , Various image processing settings, various display settings, brightness optimization settings linked to zoom magnification, and the like. In addition, the CPU 26 monitors the operation state of the operation unit 27 and outputs image data to the display unit 23.

  The operation unit 27 includes, for example, a power switch, a shutter button, a multi-selector (cross key), or other operation keys. The operation unit 27 receives a user operation input when operated by the user, and outputs a signal corresponding to the operation input to the CPU 26. Output to.

  The communication unit 28 is connected to a removable storage medium 40 such as a card memory, and writes, reads, or deletes information (image data, area information, etc.) to the storage medium 40.

  The storage medium 40 is a storage unit that is detachably connected to the imaging device 20 and stores information (image data, area information, and the like). Note that the storage medium 40 may be integrated with the imaging device 20.

  The imaging device 20 includes an automatic detection mode for automatically detecting the main subject area in addition to the normal mode for detecting the main subject area based on the focus adjustment information at the time of shooting. In the automatic detection mode, a main subject area is automatically and continuously detected based on a through image for composition confirmation, etc., and information on the detected main subject area is displayed on the display unit 23, and the buffer memory unit 24 and storage This mode is stored in the unit 25 or the like. This automatic detection mode may be set by a user operation via the operation unit 27, or may be automatically set by the CPU.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowchart of FIG.

  In step S <b> 101, the CPU 26 controls the imaging unit 21 to start acquiring a through image. The acquired image information of the through image is temporarily stored in the buffer memory unit 24. This through image is continuously generated at a predetermined time interval. The acquisition of the through image by the CPU 26 is sequentially performed sequentially in time.

  In step S102, the CPU 26 controls the image processing device 22 to perform normal image processing. Normal image processing includes white balance adjustment, interpolation processing, color tone correction processing, gradation conversion processing, and the like. Since the specific method of each process is the same as that of a well-known technique, description is abbreviate | omitted. The image processing device 22 acquires the image data of the target image from the buffer memory unit 24, performs image processing, and then outputs the image data to the buffer memory unit 24 again.

  In step S103, the CPU 26 controls the image processing device 22 to perform mask extraction processing. The mask extraction process is a technique for calculating a feature amount in an image and detecting a main subject region based on the feature amount. For example, the mask extraction is performed by obtaining an evaluation value from the feature amount in the image and obtaining a continuous region having the same evaluation value. Any method of mask extraction may be used. In addition, since a specific method for extracting a mask is the same as that of a known technique, description thereof is omitted. The CPU 26 records the result of the mask extraction process in the buffer memory unit 24, the storage unit 25, etc. as mask information. The mask information includes information such as the extraction conditions (color classification, mask classification, etc.) of the mask extraction process in step S103, the position of the mask, and the size and shape of the mask.

  Further, it may be configured to determine whether or not the main subject region has been extracted by mask extraction after the process of step S103. That is, after the process of step S103, it is determined whether the mask extraction is successful by the mask extraction process. For example, in the mask extraction process in step S103, the feature amount cannot be calculated or the evaluation value is too low. In this case, it is determined that the mask extraction has failed and the main subject area cannot be extracted. In such a case, the CPU 26 may repeat the normal image processing in step S102 and the mask extraction processing in step S103 until the main subject region can be extracted without performing the subsequent processing. In this case, when the main subject region is extracted by the mask extraction process, a tracking process described later may be started.

  Further, the CPU 26 may control the display unit 26 to display information on the main subject area extracted by mask extraction on the display unit 23. For example, the CPU 26 can display the main subject area so as to be visible by superimposing it on the through image and displaying a frame or the like in the main subject area. The frame may be in accordance with the shape of the main subject area, or may be a predetermined shape such as a rectangle or a circle. Also, the line thickness, shading, color, etc. may be anything. In any case, by performing such display, the user can easily grasp the result of the mask extraction process performed by the CPU 26.

  In step S104, the CPU 26 sets a template based on the result of the mask extraction process in step S103. This template is a template used for tracking processing described later. The CPU 26 determines the main subject area based on the mask extracted by the mask extraction in step S103, and sets a tracking target template for each of the RGB colors, for example.

  In step S105, based on the template set in step S104, the CPU 26 starts the tracking process by performing the template matching process for the through image that has been acquired in step S101. The tracking process includes storing and updating the position and area of the tracking target in time series, and analyzing, updating, using, and displaying the state. A specific method such as a template matching process in the tracking process is the same as that of a known technique, and thus the description thereof is omitted. The CPU 26 performs the tracking process with the main subject region extracted by the mask extraction process performed in step S103 as a tracking target, thereby repeating the initial mask extraction process with a process having a lighter load than the mask extraction process. Appropriate detection results can be obtained.

  Note that the CPU 26 may control the display unit 26 to display information on the main subject area detected by the tracking process on the display unit 23 as in the case of the mask extraction process described in step S103.

  Further, the template setting described in step S104 and the tracking process described in step S105 are examples. In the tracking process, a technique other than the template matching process may be used as long as the processing load is lighter than the mask extraction process described in step S103. For example, a tracking process using a block matching process, a tracking process using a motion vector, a tracking process using a limited mask process may be used. Further, the tracking process may be performed by combining a plurality of methods.

  The tracking process using the motion vector can be executed by obtaining a motion vector indicating the motion of the main subject region based on the result of the mask extraction process described in step S103 and monitoring the motion vector distribution. Note that this motion vector may be diverted from a motion vector used when generating an MPEG moving image.

  Further, the tracking process using the limited mask process can be executed by a mask extraction process that is looser than the extraction conditions of the mask extraction process described in step S103. For example, the color classification of the mask to be extracted is limited (reduced) than the mask extraction process described in step S103, or the threshold value for creating a binarized image in the mask extraction process is the mask described in step S103. The tracking process may be executed by changing the threshold value in the extraction process or performing a limited mask process.

  In step S106, the CPU 26 determines whether or not a shooting instruction has been issued. The CPU 26 waits until a shooting instruction is given, and proceeds to step S107 when determining that the shooting instruction has been given. The photographing instruction is performed by a user operation via the shutter button of the operation unit 27. This user operation may be a so-called half shutter or all shutter.

  In step S <b> 107, the CPU 26 performs shooting by controlling each unit. At this time, the CPU 26 performs 3A processing of focus adjustment (AF) setting processing, exposure adjustment (AE) setting processing, and white balance adjustment processing (AWB) based on the result of the tracking processing started in step S105. The conditions for various image processing in the image processing apparatus 22 are determined.

  In step S <b> 108, the CPU 26 records the image generated by the imaging on the storage medium 40 via the communication unit 28 and ends the series of processes.

  As described above, according to the first embodiment, a plurality of pieces of image data generated successively in time are sequentially acquired, the feature amount of the image indicated by the image data is calculated, and based on the feature amount. A main subject area is detected. When the main subject area is detected, the detected main subject area is set as a tracking target, and the tracking target is detected from a plurality of images. Therefore, with the above-described configuration, it is possible to perform tracking processing with a lighter processing load using the main subject area detected based on the feature amount as a tracking target, so that the extraction is performed when the main subject area is continuously extracted. The processing load can be reduced while maintaining the accuracy and stability. Therefore, it can be expected to speed up the extraction of the main subject area.

  Further, according to the first embodiment, the main subject region is detected by performing mask extraction based on the feature amount. Therefore, the main subject area detected with high accuracy can be set as the tracking target.

  Further, according to the first embodiment, when performing the tracking process, a block matching process, a template matching process, a tracking process using a motion vector, a tracking process using a limited mask process, and the like are executed. Therefore, the processing load can be reduced.

Second Embodiment
Hereinafter, the second embodiment will be described in detail with reference to the drawings. The second embodiment is a modification of the first embodiment. Therefore, only the parts different from the first embodiment will be described, and the description of the same parts as the first embodiment will be omitted.

  In the second embodiment, a description will be given by taking as an example an apparatus including the lens barrel 10, the imaging device 20, and the recording medium 40 shown in FIG. 1 of the first embodiment.

  In the first embodiment, as described above, the basic example in which the tracking process is performed based on the result of the mask extraction process has been described. In the second embodiment, a configuration will be described in which mask extraction processing and tracking processing are performed in combination to further improve the extraction stability of the main subject region.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowchart of FIG.

  In step S <b> 201, the CPU 26 controls the imaging unit 21 and starts acquiring a through image, similarly to step S <b> 101 of the first embodiment.

  In step S202, the CPU 26 controls the image processing device 22 to perform normal image processing, similarly to step S102 of the first embodiment.

  In step S203, the CPU 26 controls the image processing device 22 to perform mask extraction processing, as in step S103 of the first embodiment.

  In step S204, the CPU 26 sets a template based on the result of the mask extraction process in step S203, similarly to step S104 in the first embodiment.

  In step S205, the CPU 26 starts tracking processing based on the template set in step S204, similarly to step S105 of the first embodiment.

  In step S206, the CPU 26 determines whether or not the tracking process started in step S205 has continued for N frames. If the CPU 26 determines that the tracking process started in step S205 is not continued for N frames, the process proceeds to step S207. On the other hand, if the CPU 26 determines that the tracking process started in step S205 has continued for N frames, the process returns to step S203. N is a predetermined value (for example, N = 5). The case where the tracking process started in step S205 is continued for N frames means that the tracking process is continued to some extent, and there is a possibility that some change (shape change, position movement, etc.) may occur in the main subject area. is there. In such a case, the process returns to step S203, and the mask extraction process is performed again, so that the continuous tracking process and the tracking process based on the mask extraction process can be switched according to the elapsed time of tracking. Therefore, the main subject area can be reliably captured.

In step S207, the CPU 26 determines whether or not a shooting instruction has been issued, in substantially the same manner as in step S106 of the first embodiment. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S208. On the other hand, if it is determined that no shooting instruction is given, the CPU 26 returns to step S206, and performs the processing from step S206 onward for the image of the next frame.

  In step S <b> 208, the CPU 26 controls each unit to perform shooting as in step S <b> 107 of the first embodiment.

  In step S209, as in step S108 of the first embodiment, the CPU 26 records an image generated by imaging on the storage medium 40 via the communication unit 28, and ends the series of processes.

  As described above, according to the second embodiment, the first mode for continuously detecting the tracking target at the time of the previous tracking and the main subject area newly detected by the mask extraction process are detected as the tracking target. The main subject area is tracked by switching to the second mode. Therefore, even if some change occurs in the main subject area, the tracking target can be updated as appropriate, and the main subject area can be reliably extracted.

  Further, according to the second embodiment, every time the tracking process is executed a predetermined number of times in the first mode, the tracking process is executed at least once in the second mode. Therefore, it is possible to use a continuous tracking process capable of speeding up and an accurate mask extraction process in a balanced manner.

<Third Embodiment>
Hereinafter, a third embodiment will be described in detail with reference to the drawings. The third embodiment is a modification of the second embodiment. Therefore, only different parts from the first embodiment and the second embodiment will be described, and the description of the same parts as the first embodiment and the second embodiment will be omitted.

  In the third embodiment, a description will be given by taking as an example an apparatus including the lens barrel 10, the imaging device 20, and the recording medium 40 shown in FIG. 1 of the first embodiment.

  In the second embodiment, as described above, the basic example in which the mask extraction process and the tracking process are performed in combination has been described. In the third embodiment, a configuration will be described in which switching is performed more appropriately when the mask extraction process and the tracking process are used in combination.

  Hereinafter, the operation of the CPU 26 when the automatic detection mode is executed will be described with reference to the flowchart of FIG.

  In step S301, the CPU 26 controls the imaging unit 21 and starts acquiring a through image, similarly to step S101 of the first embodiment.

  In step S302, the CPU 26 controls the image processing device 22 to perform normal image processing, as in step S102 of the first embodiment.

  In step S303, the CPU 26 performs mask extraction processing by controlling the image processing device 22, as in step S103 of the first embodiment. However, unlike the first embodiment and the second embodiment, the CPU 26 continuously performs mask extraction processing. Note that the mask extraction process may be performed on all frames, or may be performed on a part of the frames subjected to thinning.

In step S304, the CPU 26 sets a template. However, CPU
26, in step S304 for the first time, as in step S104 of the first embodiment, a template is set based on the result of the mask extraction process in step S303. In step S304 for the second and subsequent times, step S308, which will be described later, is set. A template is set based on the result of the correction mask extraction process.

  In step S305, the CPU 26 starts tracking processing based on the template set in step S304, as in step S105 of the first embodiment. That is, at this time, the mask extraction process started in step S303 and the tracking process are performed in parallel. However, the mask extraction process described in step S303 may be performed at a lower frame rate than the tracking process started in step S305.

  In step S306, the CPU 26 compares the latest mask extraction process result with the latest tracking process result, and determines whether the difference is large. If the CPU 26 determines that the difference is large, the process proceeds to step S307. If the CPU 26 determines that the difference is not large, the process proceeds to step S309 described later.

  The difference can be obtained based on, for example, the main subject region (region A) based on the mask extracted in the latest mask extraction processing and the main subject region (region B) detected in the latest tracking processing. More specifically, differences in the position, size, brightness, color, shape, and the like of region A and region B can be considered. The difference may be obtained in any way. For example, evaluation values representing the above-described position, size, brightness, color, shape (aspect ratio, filling rate, etc.), etc. are obtained for each of the regions A and B. If the difference between the obtained and obtained evaluation values is larger than a predetermined threshold, it may be determined that the difference is large.

  Note that the main subject region (region A) based on the above-described mask extraction processing and the main subject region (region B) based on the tracking processing are examples. Also, any method may be used for obtaining the difference.

  By the way, the case where the difference is large is a case where some change (shape change, position movement, etc.) occurs in the main subject area, for example, when the tracking process is continued to some extent. In such a case, it is considered that the main subject region (region A) based on the mask extraction process and the main subject region (region B) based on the tracking process do not match or the similarity is reduced. In step S307, the cause is analyzed.

  On the other hand, when the difference is not large (that is, the difference is small), the main subject region (region A) based on the mask extraction processing and the main subject region (region B) based on the tracking processing match, or Since the similarity is considered to be sufficiently high, the current state is maintained and the process proceeds to step S309 described later.

  In step S307, the CPU 26 analyzes the cause of the large difference based on the result of the mask extraction process in step S303 and the result of the tracking process started in step S305.

  When the difference becomes large, it can be considered that some change has occurred in the main subject area. Causes include, for example, a change in the position, size, brightness, color, shape, and the like of the subject itself, and a change in the detected main subject region due to the operation of the user of the imaging device 20. and so on. The CPU 26 analyzes the cause of the large difference in view of the content of the determination described in step S306.

For example, if the similarity in brightness, color, and shape is high to some extent, but the similarity in position and size is low, it is considered that the position of the subject itself has changed. For example, if the similarity in position and size is high to some extent, but the similarity in brightness, color, and shape is low, the subject itself is wrong, that is, the main subject region (region) based on mask extraction processing There is a high possibility that A) is different from the main subject region (region B) based on the tracking process.

  The cause analysis may be performed by any method. Further, the CPU 26 stores the result of the cause analysis in the buffer memory unit 24, the storage unit 25, and the like, and processes other than the correction mask extraction process in step S308 described later (for example, the imaging execution process in step S310 described later). The information may also be used at the time of

  In step S308, the CPU 26 performs a correction mask extraction process according to the mask extraction process described in step S303 based on the result of the cause analysis performed in step S307. If CPU26 performs a correction mask extraction process, it will return to step S304 and will set a template again.

  The correction mask extraction process is a new mask extraction process for resetting a template used for the tracking process based on the result of the cause analysis performed in step S307. The CPU 26 performs mask extraction processing by optimizing the extraction conditions based on the contents of the determination described in step S306 and the result of the cause analysis performed in step S307.

  At this time, the correction mask extraction process may be a mask extraction process that is looser than the extraction conditions of the mask extraction process described in step S303. For example, based on the result of the cause analysis performed in step S307, the color classification of the mask to be extracted is limited (reduced) than the mask extraction process described in step S303, or the binarized image is extracted in the mask extraction process. A limited mask process may be performed by changing the threshold at the time of creation from the threshold in the mask extraction process described in step S303.

  Instead of performing the correction mask extraction process, either a main subject area (area A) based on the mask extraction process or a main subject area (area B) based on the tracking process may be selected. For example, if the difference is large, it is considered that the tracking process is inaccurate and needs to be reset, so the main subject area (area A) based on the latest mask extraction process is selected, and the process returns to step S304. A template may be set. Further, for example, when only the difference regarding the position and size is large, it is considered that the subject itself is not wrong, and the main subject region (region B) based on the tracking process may be selected. In any case, by using a complementary combination of mask extraction processing and tracking processing, the processing load is reduced while maintaining the accuracy and stability of extraction when continuously extracting the main subject area. be able to.

  In step S309, the CPU 26 determines whether or not a shooting instruction has been issued, substantially as in step S106 of the first embodiment. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S310. On the other hand, if it is determined that no shooting instruction is given, the CPU 26 returns to step S306, and performs the processing from step S306 on the image of the next frame. That is, the determinations in step S306 and step S309 are repeated until it is determined in step S306 that the difference is large, or until it is determined in step S309 that a shooting instruction has been issued.

  In step S <b> 310, the CPU 26 performs shooting by controlling each unit as in step S <b> 107 of the first embodiment.

In step S <b> 311, the CPU 26 records an image generated by imaging on the storage medium 40 via the communication unit 28 and ends a series of processes, as in step S <b> 108 of the first embodiment.

  As described above, according to the third embodiment, based on the result of tracking, the first mode for continuously detecting the tracking target at the time of the previous tracking and the main subject newly detected by the mask extraction process The main subject area tracking process is performed by switching to the second mode in which the area is detected as a tracking target. Therefore, when tracking is suitably performed, tracking with a small processing load is continuously performed, and when tracking is not suitably performed due to some change in the main subject area, an appropriate tracking target And the main subject area can be reliably extracted.

  Further, according to the third embodiment, when the similarity in the tracking process is below a predetermined threshold, the mode is switched to the second mode. Therefore, when tracking is not performed properly, it is possible to promptly shift to tracking processing based on accurate mask extraction processing.

  Further, according to the third embodiment, the evaluation value of the tracking target detected by the tracking process is calculated, and based on the amount of change between the calculated evaluation value and the evaluation value of the predetermined tracking target detected in the past, Determine the tracking mode. Therefore, the stability of the tracking process can be monitored according to the evaluation value, and when the tracking is not suitably performed, the tracking process based on the accurate mask extraction process can be promptly shifted.

  Further, according to the third embodiment, as the evaluation value described above, the value indicating the position of the tracking target, the value indicating the size of the tracking target, the value indicating the brightness of the tracking target, the value indicating the color of the tracking target, When a value indicating the shape of the tracking target is used and the change amount of the evaluation value exceeds a predetermined threshold, the mode is switched to the second mode described above. Therefore, by using the evaluation value according to the change of the subject, it is possible to use in a well-balanced manner the continuous tracking processing that enables high speed and the tracking processing based on the accurate mask extraction processing.

<Fourth embodiment>
Hereinafter, the fourth embodiment will be described in detail with reference to the drawings. The fourth embodiment is a modification of the second embodiment. Therefore, only different parts from the first embodiment to the third embodiment will be described, and description of the same parts as the first embodiment to the third embodiment will be omitted.

  In the fourth embodiment, an apparatus including the lens barrel 10, the imaging device 20, and the recording medium 40 illustrated in FIG. 1 of the first embodiment will be described as an example.

  In the second embodiment, as described above, a basic example in which mask extraction processing and tracking processing are used in combination will be described. In the third embodiment, when mask extraction processing and tracking processing are used in combination, The configuration for switching more appropriately has been described. In the fourth embodiment, another configuration will be described in which switching is performed more appropriately when the mask extraction process and the tracking process are used together.

  Hereinafter, the operation of the CPU 26 during execution of the automatic detection mode will be described with reference to the flowchart of FIG.

  In step S401, the CPU 26 controls the imaging unit 21 and starts acquiring a through image, similarly to step S101 of the first embodiment.

  In step S402, the CPU 26 controls the image processing device 22 to perform normal image processing, as in step S102 of the first embodiment.

In step S403, the CPU 26 controls the image processing apparatus 22 to perform mask extraction processing, similarly to step S103 of the first embodiment. However, unlike the third embodiment, the CPU 26 does not perform continuous mask extraction processing.

  In step S404, the CPU 26 sets a template. However, the CPU 26 sets a template in the first step S404 based on the result of the mask extraction process in step S403 in the same manner as in step S104 in the first embodiment, and in the second and subsequent steps S404, step S403. A template is set based on the result of the mask extraction process or the result of the correction mask extraction process in step S409 described later.

  In step S405, the CPU 26 starts tracking processing based on the template set in step S404, as in step S105 of the first embodiment.

  In step S406, the CPU 26 determines whether or not the tracking process started in step S405 has failed. If the CPU 26 determines that the tracking process has failed, it returns to step S403 and performs the mask extraction process again. On the other hand, if it is determined that the tracking process has succeeded (not failed), the process proceeds to step S407.

  The case where it is determined that the tracking process has failed is a case where the main subject area cannot be detected by the tracking process, or a case where the detected main subject area is clearly abnormal. Any method known in the art may be used to determine whether or not the tracking process has failed.

  If it is determined that the tracking process has failed, the CPU 26 performs the mask extraction process again, resets the template, and starts the tracking process again. On the other hand, when determining that the tracking process has succeeded (not failed), the CPU 26 performs further determination.

  In step S407, the CPU 26 determines whether or not a predetermined time has elapsed. If the CPU 26 determines that the predetermined time has elapsed, it proceeds to step S408, and if it determines that the predetermined time has not elapsed, it proceeds to step S411 described later. The predetermined time is a predetermined time and corresponds to the N frame described in step S206 of the second embodiment. When the predetermined time has elapsed since the tracking process was started in step S405, the tracking process is continued to some extent, and there is a possibility that some change (shape change, position movement, etc.) has occurred in the main subject area. This is the case. In such a case, the process proceeds to step S408, and the change in the mask state can be confirmed according to the elapsed time of tracking by determining the change in the mask state (details will be described later). On the other hand, if it is determined that the predetermined time has not elapsed, the CPU 26 confirms the presence or absence of a shooting instruction (details will be described later).

  In step S408, the CPU 26 determines whether or not the mask state has changed. If the CPU 26 determines that the mask state has changed, it proceeds to step S409, and if it determines that the mask state has not changed, it returns to step S407.

  The determination of whether or not the mask state has changed can be obtained based on the change in the main subject area detected in the tracking process. More specifically, for example, a change in the mask state is obtained based on a change in position, size change, brightness change, color change, shape change, etc. of the main subject area detected in the tracking process. Can do. Each change described above may be obtained in any way. For example, an evaluation value representing the position, size, brightness, color, shape (aspect ratio, filling rate, etc.) described above is obtained for each. When the change in the evaluation value is larger than a predetermined threshold, it may be determined that the mask state has changed.

When the CPU 26 determines that the mask state has changed, as described above, step S409 is performed.
Proceed to, and corrective processing is performed (details will be described later). On the other hand, when determining that the mask state has not changed, the CPU 26 returns to step S407 and determines again whether or not a predetermined time has elapsed. Note that when returning to step S407, the CPU 26 performs measurement for a predetermined time from zero, for example, by resetting a measurement timer.

  In step S409, the CPU 26 performs a correction mask extraction process.

  The correction mask extraction process is substantially the same as step S308 in the third embodiment, and a new mask for resetting a template used for the tracking process based on the result of the mask state change determination performed in step S408. It is an extraction process. The CPU 26 performs the correction mask extraction process by optimizing the extraction condition based on the result of the mask state change determination performed in step S408.

  At this time, the correction mask extraction process may be a mask extraction process (limited mask process) that is looser than the extraction conditions of the mask extraction process described in step S403, as in step S308 of the third embodiment.

  In step S410, the CPU 26 determines whether the correction mask extraction process is successful. If the CPU 26 determines that the correction mask extraction processing performed in step S409 is successful, the CPU 26 returns to step S404 and sets the template again. On the other hand, if it is determined that the correction mask extraction process is not successful (failed), the process returns to step S403 and the mask extraction process is performed again.

  That is, if it is determined in step S408 that the mask state has changed, a correction mask extraction process is performed for the purpose of resetting the template. If the correction mask extraction process is successful, the template is reset and the correction mask is reset. If the extraction process fails, the mask extraction process is performed again to reset the template.

  In step S <b> 411, the CPU 26 determines whether or not a shooting instruction has been issued, substantially as in step S <b> 106 of the first embodiment. If the CPU 26 determines that a shooting instruction has been given, the process proceeds to step S412. On the other hand, if it is determined that the shooting instruction is not performed, the CPU 26 returns to step S406 and performs the processing from step S406 on the image of the next frame.

  In step S <b> 412, the CPU 26 controls each unit to perform shooting as in step S <b> 107 of the first embodiment.

  In step S413, as in step S108 of the first embodiment, the CPU 26 records an image generated by imaging on the storage medium 40 via the communication unit 28, and ends the series of processes.

  As described above, according to the fourth embodiment, the first mode for continuously detecting the tracking target at the previous tracking based on the tracking result, the tracking elapsed time, the mask extraction processing result, and the like. The main subject area tracking process is performed by switching between the second mode in which the main subject area newly detected by the mask extraction process is detected as a tracking target. Therefore, the main subject region can be reliably extracted by appropriately changing the processing contents in accordance with the status of the tracking or mask extraction processing.

The second embodiment exemplifies a configuration that appropriately switches between two types of processing, tracking processing and mask extraction processing. In the third and fourth embodiments, tracking processing, mask extraction processing,
A configuration in which the three types of correction mask extraction processing are appropriately switched is illustrated. In the second to fourth embodiments, the number of frames for which the tracking process has been continued, the difference between the result of the tracking process and the result of the mask extraction process, the success or failure of the tracking process, the time for which the tracking process has been continued, The configuration in which each process described above is appropriately switched based on the state of the mask based on the success or failure of the mask extraction process is illustrated. However, any combination of processes described in the embodiments and combinations of triggers for switching processes may be used.

As triggers, the following ones from 1) to 6) can be considered including the above-described examples.
1) When the amount of change in the position of the main subject area detected in the tracking process exceeds a predetermined threshold with reference to the initial tracking process, or when the above-described change amount is the previous (or several times before) When a predetermined threshold is exceeded based on the tracking process. Concerning the mask extraction processing, conversely, when the amount of change in the position of the main subject area detected in the mask extraction processing falls below a predetermined threshold with reference to the initial mask extraction processing, or the amount of change described above When the threshold value is below a predetermined threshold on the basis of the previous (or several times before) mask extraction process. With respect to the correction mask extraction process, the amount of change in the position of the main subject area detected in the correction mask extraction process in relation to the tracking process or the mask extraction process described above is based on the initial correction mask extraction process. When the threshold value exceeds or falls below the predetermined threshold value, or when the above-described change amount exceeds or falls below the predetermined threshold value based on the correction mask extraction process of the previous time (or several times before).
2) A change amount of the size of the main subject area detected in the tracking process instead of the change amount of the position of 1).
3) Instead of the change amount of the position of 1), the change amount of the brightness of the main subject area detected in the tracking process.
4) The amount of change in the color of the main subject area detected in the tracking process instead of the amount of change in the position of 1).
5) A change amount of the shape (aspect ratio, filling rate, etc.) of the main subject area detected in the tracking process instead of the change amount of the position of 1).
6) When the detection fails in any of the tracking process, the mask extraction process, and the correction mask extraction process, or when the detection accuracy is very low.

In addition, as the process switching, the following 1) and 2) are conceivable including the above-described examples.
1) The above-described plurality of types of processing are appropriately switched according to the degree of each change amount described above. For example, when switching between three types of processing, that is, tracking processing, mask extraction processing, and correction mask extraction processing, the detection accuracy is higher in the order of mask extraction processing, correction mask extraction processing, and tracking processing, and the detection load is tracking processing and correction mask processing. Light in the order of mask processing.

  Therefore, for example, during the tracking process, the amount of change described above is monitored, and when a predetermined threshold is exceeded, the process shifts to the correction mask extraction process with emphasis on accuracy, and after the process shifts to the correction mask extraction process, When the amount of change further exceeds a predetermined threshold value, the process may be shifted to mask extraction processing with an emphasis on further accuracy. Also, during the tracking process, the above-described change amount is monitored, and if the change amount is moderately large, the process proceeds to the correction mask extraction process, whereas if the change amount is very large, the correction mask extraction process is performed. The process may be shifted from the tracking process to the mask extraction process without going through the process.

Conversely, while the mask extraction process is being performed, the amount of change described above is monitored, and if it falls below a predetermined threshold value, the shift to the correction mask extraction process is made with emphasis on reducing the load, and the process proceeds to the correction mask extraction process. Later, when the amount of change further falls below a predetermined threshold value, the load may be shifted to the tracking process with an emphasis on further load reduction. In addition, during the mask extraction process, the amount of change described above is monitored,
When the amount of change is moderate, the process proceeds to correction mask extraction processing, while when the amount of change is very small, the process proceeds from mask extraction processing to tracking processing without going through correction mask extraction processing. good.
2) The plurality of types of processing described above are switched as appropriate according to the success or failure of detection by any of the tracking processing, mask extraction processing, or correction mask extraction processing described above. For example, if detection by the tracking process fails during execution of the tracking process, the process shifts to the correction mask extraction process with emphasis on accuracy, and after the shift to the correction mask extraction process, the detection by the correction mask extraction process fails. In this case, the mask extraction process may be shifted with emphasis on further accuracy. Further, when the detection by the tracking process fails during the tracking process, the tracking process may be shifted to the mask extracting process without going through the correction mask extracting process.

On the other hand, when the mask extraction process is successful, if the detection succeeds to some extent by the mask extraction process, after shifting to the correction mask extraction process with emphasis on reducing the load, If the detection by the correction mask extraction process succeeds to some extent, the process may be shifted to the tracking process with an emphasis on further load reduction.

  In addition, a plurality of triggers and process switching methods described above may be provided, and the process may be switched by using a trigger predetermined for each case according to the type of image analysis, scene analysis, and shooting mode. . In this case, it is preferable to determine an appropriate threshold value and determination condition for each case.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  In each of the above-described embodiments, a single main subject region or a plurality of main subject regions may be detected in the detection by the tracking process, the mask extraction process, and the correction mask extraction process. When a plurality of main subject areas are detected, it is preferable to devise display contents when displaying information on the main subject area on the display unit 23. For example, it is preferable to make the distinction possible by changing the color of the display frame, changing the thickness of the line, changing the density of the line, changing the type of line (solid line / dotted line), and the like.

  In addition, when performing the following processes for only a part of the main subject areas among the plurality of main subject areas, a part of the processing is also performed for the main subject areas that are not targeted, and only information recording is performed. May be performed. Further, when a plurality of main subject areas are detected, an error display may be performed using the display unit 23 or the like.

  Further, in each of the above-described embodiments, the automatic detection mode interruption processing and resumption processing may be appropriately executed. For example, in the above-described automatic detection mode, when detection in the automatic detection mode is difficult, such as when the main subject region cannot be detected even after a certain period of time, the automatic detection mode is automatically switched to the normal mode. It is good also as a structure changed by. Furthermore, after changing from the automatic detection mode to another mode, the automatic detection mode may be resumed according to the amount of motion of the imaging device 20 or the output of the motion sensor.

  In each of the above embodiments, an example in which a series of processing is performed based on a through image for composition confirmation has been described, but the present invention is not limited to this example. For example, the present invention can be similarly applied to a case where a live view image for composition confirmation generated in a single-lens reflex camera or the like is targeted. Further, the present invention can be similarly applied to a case where a moving image recorded on the recording medium 40 or the like is targeted.

  In each of the above embodiments, an example in which a series of processing is performed for all frames has been described. However, the present invention is not limited to this example. For example, a plurality of images generated intermittently in time may be targeted. Specifically, a plurality of images subjected to frame thinning as appropriate may be targeted. By performing such processing, the processing load can be reduced.

  Further, the image processing described in each of the above-described embodiments may be realized by software by a “computer system” including a computer and an image processing program. In this case, the computer system may be configured to execute part or all of the processing of the flowcharts described in the embodiments. For example, part or all of the processing from step S101 to step S105 in FIG. 2 may be executed by a computer. Also, part or all of the processing from step S201 to step S206 in FIG. 3 may be executed by a computer. Also, some or all of the processing from step S301 to step S308 in FIG. 4 may be executed by a computer. Also, some or all of the processing from step S401 to step S410 in FIG. 5 may be executed by a computer. By adopting such a configuration, it is possible to perform the same processing as in the above-described embodiments.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a www system is used. The computer-readable recording medium is a writable nonvolatile memory such as a flexible disk, a magneto-optical disk, a ROM, or a flash memory, a portable medium such as a CD-ROM, and a storage such as a hard disk built in the computer system. Refers to the device.

  Further, the computer-readable recording medium is a volatile memory (for example, DRAM (Dynamic Random Access) in a computer system serving as a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Memory)) that holds a program for a certain period of time is also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

DESCRIPTION OF SYMBOLS 20 ... Imaging device, 21 ... Imaging part, 22 ... Image processing apparatus, 23 ... Display part, 26 ... CPU

Claims (1)

An acquisition unit that sequentially acquires a plurality of pieces of image data generated continuously in time;
A region detection unit that calculates a feature amount of an image indicated by the image data acquired by the acquisition unit and detects a subject region based on the feature amount;
An image processing apparatus comprising: a tracking unit configured to detect the tracking target from a plurality of the images, with the detected subject region detected as the tracking target when the region detection unit detects the subject region.

Images