JP2022048077A - Image processing apparatus and control method for the same - Google Patents

Abstract

To enable a selection of a focus target area on which AF can be suitably executed.SOLUTION: An image processing apparatus is operable to determine a focus target area of an image capturing apparatus. The image processing apparatus includes: obtainment means configured to obtain a first area to be a focus target in a first image captured by the image capturing apparatus at a first point in time; detection means configured to detect a second area to be a focus target candidate from a second image captured by the image capturing apparatus at a second point in time succeeding the first point in time; and determination means configured to determine, on the basis of the first area and the second image, a focus target area in the second image from among one or more partial areas in the second area.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for selecting an image region.

When shooting with a camera, there is an autofocus (AF) function that automatically focuses. As a method of selecting an area to be focused at the time of shooting (hereinafter referred to as an area to be focused), a method of manually selecting by the user using a touch panel or the like, or automatic based on detection results such as face detection and object detection. There is a way to select. Regardless of the method selected for the in-focus target area, the position or shape on the image may change due to the movement of the object or the camera itself in the selected in-focus target area. At this time, by tracking or continuously detecting the selected in-focus target area, it is possible to continue AF in the area desired by the user.

Patent Document 1 discloses a method of detecting a pupil region and using it for an in-focus target region to perform AF. According to this method, since the focusing target area having a constant distance from the camera is used, it is possible to focus accurately.

Japanese Unexamined Patent Publication No. 2019-121860

However, in the method described in Patent Document 1, it is necessary to detect a specific part such as a pupil. Therefore, it cannot be applied when the specific part cannot be observed due to being shielded by another object or the like. Further, in the method described in Patent Document 1, it is difficult to focus accurately when the difference in depth (distance from the camera) in the focusing target region is large. Therefore, it is difficult to apply it when it is desired to set a part having a certain size such as a torso or an arm as an in-focus target area.

The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique for selecting a focus target region in which AF can be suitably performed.

In order to solve the above-mentioned problems, the image processing apparatus according to the present invention has the following configurations. That is, the image processing device that determines the focus target area of the image pickup device is an acquisition means for acquiring the first focus target area in the first image captured by the image pickup device at the first time point. A detection means for detecting a second region as a candidate for focusing target from a second image captured by the image pickup apparatus at a second time point following the first time point, the first region, and the above. A determination means for determining an in-focus target region in the second image from one or more partial regions of the second region based on the second image is provided.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide a technique that enables selection of an in-focus target region in which AF can be suitably performed.

It is a figure which shows the positional relationship between a camera and a subject. It is a figure which shows an example of the structure of the AF system in 1st Embodiment. It is a flowchart explaining the process executed by the AF system in 1st Embodiment. It is a figure explaining the comparison (S108) of a reference area and each partial area. It is a figure which shows the detection example of the head and the torso of a human body. It is a figure which shows the hardware configuration of an image processing apparatus. It is a flowchart explaining the process executed by the AF system in 2nd Embodiment. It is a figure explaining the comparison of the reference region and the head partial region. It is a figure explaining the comparison of the reference area and the body part area.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are given the same reference numbers, and duplicate explanations are omitted.

(First Embodiment)
As a first embodiment of the image processing apparatus according to the present invention, an autofocus (AF) system including a photographing apparatus and an area selection apparatus will be described below as an example. In particular, in the first embodiment, the AF system detects the human body based on the image acquired from the photographing apparatus, and extracts the in-focus target area to be focused from the detected human body area.

<Device configuration>
FIG. 1 is a diagram showing a positional relationship between a photographing device (camera) and a subject. FIG. 1 (a) schematically shows a subject G-1 which is a human body in a standing position, and FIG. 1 (b) schematically shows a subject G-2 which is a human body in a supine position. .. The camera exists in the lower left direction of the figure, and a plurality of dotted lines indicating the distance (depth, depth) from the camera are shown.

As shown in FIG. 1, the distance from the camera to each part of the body changes depending on the posture of the subject. For example, in the case of standing as shown in the subject G-1, the distance from the camera does not change significantly at any part. On the other hand, when lying substantially parallel to the line-of-sight direction of the camera as shown in the subject G-2, the depth changes greatly depending on the part of the body. When the change width of the depth in the subject is wider than the depth of field of the image pickup apparatus, it is generally impossible to focus on the entire subject. As a result, AF may not be suitable for continuous execution. Therefore, in the first embodiment, an example in which suitable AF can be continuously realized even when the change width of the depth in the subject is wider than the depth of field of the image pickup apparatus will be described.

<Device configuration>
FIG. 2 is a diagram showing an example of the configuration of the AF system according to the first embodiment. As shown in FIG. 2, the AF system includes a photographing device 10 and an area selection device 20.

The photographing device 10 is a camera device that images a scene of the surrounding environment. The photographing device 10 includes an image acquisition unit 11 and a distance measuring unit 12. Examples of the photographing device 10 include a digital single-lens reflex camera, a smartphone, a wearable camera, a network camera, and a Web camera. However, the present invention is not limited to these examples, and any device that can image the surrounding scene may be used.

The image acquisition unit 11 uses an image sensor or the like to image a scene around the photographing device 10 and outputs the image to the area selection device 20. The image acquired by the image acquisition unit 11 may be RAW data before the demosaiking process, or may be an image in which all pixels have RGB values by demosiking or the like. It may also be an image for live view.

The distance measuring unit 12 has a distance measuring function for measuring the depth information which is the distance between the photographing device 10 and the subject, and outputs the measured depth information to the area selection device 20. The depth information can be associated with each pixel or each area of the image acquired by the image acquisition unit 11. Further, the depth information is arbitrary information that correlates with the length of the distance in space. For example, it may be the length itself in space, or it may be the amount of defocus based on a phase difference sensor that detects the phase difference of incident light. Further, it may be the amount of contrast change of the image when the focal plane of the lens is moved.

The area selection device 20 (image processing device) detects the area of the human body based on the image and the depth information input from the photographing device 10. Then, the focusing target area to be focused is selected from the detected areas. The area selection device 20 includes a detection unit 21, a partial area extraction unit 22, a reference area acquisition unit 23, a comparison unit 24, and a selection unit 25. Although the area selection device 20 is shown in FIG. 2 as being separate from the photographing device 10, it may be configured as an integrated device. Further, when it is configured as a separate body, it may be connected by a wired or wireless communication function. Further, each functional unit of the area selection device 20 can also be realized by the central processing unit (CPU) executing a software program.

FIG. 6 is a diagram showing a hardware configuration of the information processing device. The CPU 1001 uses the RAM 1003 as a work memory to read out and execute an OS and other programs stored in the ROM 1002 and the storage device 1004. Then, each configuration connected to the system bus 1009 is controlled to perform calculations and logical determinations for various processes. The process executed by the CPU 1001 includes information processing of the embodiment. The storage device 1004 is a hard disk drive, an external storage device, or the like, and stores programs and various data related to information processing of the embodiment. The input unit 1005 is an image pickup device such as a camera, an input device such as a button for inputting a user instruction, a keyboard, and a touch panel. The storage device 1004 is connected to the system bus 1009 via an interface such as SATA, and the input unit 1005 is connected to the system bus 1009 via a serial bus such as USB, but the details thereof will be omitted. The communication I / F 1006 communicates with an external device by wireless communication. The display unit 1007 is a display.

The detection unit 21 detects the human body region in the image and outputs it to the partial region extraction unit 22. The human body region may correspond to the whole body or may correspond to a specific part such as a face or a torso. The method for detecting the human body region is not limited to a specific method. For example, an application of an object detection technique as described in "Joseph Redmon, Ali Farhadi," YOLOv3: An Incremental Improvement ", arXiv e-prints (2018)" is available. Further, it may be detected based on the contour shape of the head, limbs, or the like. Further, it may be detected based on the motion information extracted from the time-series images. In addition, it may be configured to detect from information of a heat source based on far infrared rays or the like.

The human body region to be detected may correspond to a predetermined part or may correspond to a part selected by the user. For example, a function for setting a category of a part such as a face or a torso may be provided, and a detection process corresponding to the category set by the user may be performed. Further, for example, the corresponding detection process may be automatically set based on the information of the area selected by the user on the touch panel.

When the detection unit 21 detects a plurality of human body regions, one or a plurality of corresponding human body regions may be selected and output based on the reference region described later. For example, the selection may be based on the distance to the reference area and the degree of image similarity.

The partial region extraction unit 22 extracts one or more partial regions satisfying a predetermined condition based on the human body region or depth information, and outputs them to the comparison unit 24. Subregion extraction is based on the similarity of distance and depth information in image space.

For example, when the depth information of each pixel in the human body region is input, a set in which the distance between the pixels in the image space is short and the depth information is similar may be extracted as a subregion. At this time, the closed region included in the partial region may be integrated into the partial region or may be extracted as a different partial region. Further, the extraction may be performed by setting a threshold value for the distance between pixels and the area of the partial area.

As another method, for example, when the distance measuring unit 12 measures the depth information for a specific distance measuring area, the range measuring area in the human body area in which the variance of the depth information is equal to or less than the threshold value is extracted as a partial area. You may.

The reference area acquisition unit 23 acquires a reference area, which is a specific portion to be AF, and outputs the reference area to the comparison unit 24. The method of acquiring the reference area is not limited to a specific method. For example, the photographer may tap the live view screen of the camera to use the tapped area as a reference area. Alternatively, for example, using a face detection method, a region of the face that is closer to the center of the screen and appears larger may be used as the reference region. When the detection method is used, the detection method may be the same as or different from the detection unit 21. Here, it is assumed that the reference region (first region) is held as the focusing target region in the first image taken at the first time point which is the start of AF. At the start of shooting, a predetermined area of the initial image (for example, the center of the screen) is set in advance as a reference area.

The comparison unit 24 compares the partial area input from the partial area extraction unit 22 with the reference area input from the reference area acquisition unit 23, and outputs the comparison result to the selection unit 25.

The element to be compared by the comparison unit 24 (hereinafter referred to as a comparison element) includes depth information, and may be a plurality of different types of information. For example, it may include relative positions in the image space between the reference region and the subregion, or it may include pixel values on the image corresponding to the region. Further, when the relative position on the image space is used, the size of the relative position may be normalized based on the depth information. For example, the relative position value may be increased for a region where the distance from the camera is small, and the relative position value may be increased for an region where the distance is large.

However, the comparison method of the comparison unit 24 is not limited to a specific method. For example, the comparison elements in each area may be averaged and the difference between the average values may be output as the comparison result. Further, when the shapes of the partial region and the reference region are the same, the difference between the comparison elements may be output as the comparison result for each corresponding pixel. In addition, the distribution of depth information in the region may be compared, and an index such as the amount of Kullback-Leibler information may be output as a comparison result.

Further, the comparison unit 24 compares the estimated depth information with the depth information of the partial region by estimating the amount of change in the depth information from the time when the reference region is acquired based on the time change of the depth information of the object of interest. You may. In this case, more stable AF may be realized even for a subject whose distance to the camera changes dynamically with the passage of time.

The selection unit 25 selects an in-focus target area for the current input image based on the comparison result between the reference area and each partial area input from the comparison unit 24, and outputs the image to the photographing apparatus 10. As an example of the method in which the selection unit 25 selects the in-focus target area, a partial area similar to the reference area (that is, the difference from the depth of the reference area is relatively small) is selected in the comparison result with the reference area. There is a way. According to this method, AF can be continued in a region similar to the reference region.

Further, instead of using only the comparison result, the priority of selecting each partial region as the focusing target region may be evaluated, and the one having a high priority may be easily selected. For example, when there are a plurality of partial regions whose difference from the depth of the reference region is less than a predetermined value, the larger the area, the higher the priority may be. Alternatively, for example, the closer to the center of the image, the higher the priority. Further, the selection unit 25 may determine the in-focus target area in the second image based on a plurality of criteria. For example, the plurality of criteria include the similarity between the first region and the second region, and each criterion for the area of one or more partial regions.

Further, the selection criterion of the focusing target area may be changed according to the elapsed time from the time when the reference area is acquired. For example, if the elapsed time is short, priority may be given to the comparison result with the reference area having a close degree of similarity, and as the elapsed time becomes longer, the priority criteria other than the comparison result such as the area of the partial area may be emphasized and selected. .. Further, when the comparison result includes a plurality of types of elements, the similarity of each element may be selected in consideration of different weights.

<Operation of the device>
FIG. 3 is a flowchart illustrating a process executed by the AF system according to the first embodiment. Each of S101 to S111 represents a specific process, and is executed in order in principle. However, the AF system does not necessarily have to perform all the processes described in this flowchart, and the execution order of the processes may change. Further, a plurality of processes may be executed in parallel.

In step S101, the image acquisition unit 11 acquires an image (first image) at the AF start time point (time t-1). For example, an RGB image of a live view is acquired. Further, in step S102, the ranging unit 12 measures the depth information at the start of AF. For example, when the ranging unit 12 includes a phase difference sensor, the defocus amount is measured. The measured depth information is subsequently acquired (depth information acquisition) by the area selection device 20.

In step S103, the reference area acquisition unit 23 acquires the reference area (first area) at the AF start time point (first time point). That is, the area used as the in-focus target area at the start of AF is acquired (reference acquisition). For example, the area selected by the user on the touch panel or the automatically detected area of the face or human body is acquired. When the reference area is acquired based on the detection process, the detection unit 21 or the like may be used. Further, when there are a plurality of candidates for the reference region, the reference region may be selected by using the selection unit 25 or the like. Further, when shooting continuously, the previous focusing area may be acquired.

In step S104, the image acquisition unit 11 acquires an image (second image) at a time point (time t, second time point) at which the focusing target area is selected. The time point at which the in-focus target area is selected is the time following the AF start time point. The first time point and the second time point may not be continuous times, and for example, the focus position may be changed at regular time intervals. In step S105, the ranging unit 12 measures the depth information at the time of selecting the focusing target area. The measured depth information is subsequently acquired (depth information acquisition) by the area selection device 20.

In step S106, the detection unit 21 detects a human body region (second region) as a candidate for focusing target from the image acquired in S104. For example, it detects a predetermined object to be focused (a human face or whole body, an animal such as a dog or a cat, or an area such as a car or a building). A region having similar image features to the reference region may be detected as the second region. For example, deep learning or semantic segmentation may be used to detect the second region. Further, a region having a predetermined depth may be detected as a second region based on the depth information corresponding to the image acquired in S104. For example, when there is only one subject, the area in front (that is, the area showing the same depth) may be detected. In this case, S107 may be skipped. Then, in step S107, the partial region extraction unit 22 extracts one or more partial regions satisfying a predetermined condition from the human body region detected in S106. For example, the partial region is extracted based on the depth information, and specifically, the partial region in which the depth indicated by the depth information is included in the range of the depth of field is extracted.

The processing of S107 will be described with reference to FIG. In S107, the extraction standard of the partial region is changed based on the depth of field. For example, when the subject G-1 (upright human body) is detected, the distance from the camera in the human body region is almost constant (for example, the calculated depth difference is 50 cm or less). Therefore, in general, the entire human body region can be extracted as one partial region. However, when the depth of field of the photographing device is narrow (for example, several cm), partial regions such as the eyes, nose, limbs, etc., which are close to the photographing device may be extracted.

As an example of a method of extracting a partial region close to the camera, there is a method of using a clustering method such as K-means. Specifically, pixel clusters with close depth information are extracted, and each cluster is extracted as a partial area. At this time, the distance between the pixels on the image may be taken into consideration or may be ignored.

In step S108, the comparison unit 24 compares the reference region acquired in S103 with each partial region extracted in S107. For example, the distribution of depth information between the reference region and each subregion, and the image features in the reference region and the image features in the extracted subregion are compared.

FIG. 4 is a diagram illustrating a comparison between the reference region and each partial region in S108. FIG. 4A shows a region G-4 which is a reference region acquired from the first image. Further, FIG. 4B shows a region G-5a and a region G-5b which are partial regions extracted from the second image. In S108, for example, the average depth may be obtained from the depth information for each of the reference region and each partial region, and the absolute value of the difference between the average depths from the depth information between the reference region and the partial region may be output as a comparison result. .. In the example shown in FIG. 4, the region G-5a is closer to the region G-4 than the region G-5b. Therefore, as a comparison result, the difference in depth information is output to be smaller. When comparing using image features, the similarity between the image features in the reference region and the image features in the extracted partial region is compared with a preset threshold value. If the degree of similarity is equal to or higher than the threshold value, they are similar and are likely to be the same site. On the other hand, if it is below the threshold value, there is a high possibility that it is a different site because it is not similar.

In step S109, the selection unit 25 selects a partial region based on the comparison result and determines it as the focusing target region. Here, the partial region having the average depth closest to the average depth in the reference region is determined as the focusing target region. Instead of the average depth in the region, the depth at a typical position may be used. Further, the partial region having the smallest difference in depth (smaller than a predetermined value) may be determined as the focusing target region. Determining a region with a close depth of field in this way also leads to a reduction in focusing time. Further, as an example of the method of selecting the in-focus target area, for example, a partial area most similar to the reference area may be selected. Further, the subregion having the largest area may be selected from the subregions whose difference from the reference region is equal to or less than the threshold value in the comparison result. A plurality of selection criteria may be combined.

In step S110, the area selection device 20 determines whether or not to continue the AF process. If the AF process is to be continued, the process proceeds to S111, and if not, the process is terminated. In step S111, the area selection device 20 replaces the second image with the first image. After that, the process returns to S103 and the process is repeated.

By the above processing, it becomes possible to suitably select the region in the current (time t) image corresponding to the reference region selected as the AF target immediately before (time t-1).

As described above, according to the first embodiment, the partial region suitable for AF is selected as the focusing target region in the detected human body region by using the information of the reference region. In particular, select a partial region with a smaller (more similar) difference from the reference region. This allows the AF system to suitably continue AF even when the difference in depth within the detected region is large.

Although the case of detecting the human body region has been described in the above description, it may be applied to a detection target other than the human body region. For example, an animal other than a human may be detected, or a specific object such as a vehicle may be detected. In addition to being usable for shooting with a digital camera, it can also be used for a system that changes the focus position after shooting by post-processing.

(Second Embodiment)
In the second embodiment, a mode for dealing with a case where the site categories of the reference region and the in-focus target region are different will be described. Hereinafter, an example will be described in which the head and the body are detected based on an image acquired from a camera or the like, and when the head is not detected, the focusing target area is selected from the partial area of the body.

In addition, here, the term "torso" refers to the trunk part of a person who does not include the head. However, the torso may include the neck, limbs, etc. in addition to the trunk. Moreover, it may be a part of a part such as a chest and an abdomen instead of the whole trunk part.

The configuration of the AF system in the second embodiment is almost the same as that in the first embodiment (FIG. 2). However, since the operation of each functional unit is different from that of the first embodiment, the parts different from the first embodiment will be described below.

FIG. 7 is a flowchart illustrating a process executed by the AF system according to the second embodiment. S201 to S215 each represent a specific process.

In steps S201 and S202, the first image and its depth information are acquired. Since the processing performed in S201 and S202 is the same as in S101 and S102 in the first embodiment, the description thereof will be omitted.

In step S203, the detection unit 21 detects the head region and the torso region of the human body from the first image and outputs them to the partial region extraction unit 22 and the reference region acquisition unit 23. The method of detecting the torso region is not limited to a specific method. For example, the torso may be detected directly by using a semantic region division method, or the torso may be detected by detecting a part contained in the torso such as a shoulder or a waist by using an object detection method. May be good.

FIG. 5 is a diagram showing an example of detection of a human body. FIG. 5A shows an example when the detection unit 21 detects the head portion and the body portion. Regions G-8 and G-6a are the detected heads and torso. On the other hand, FIG. 5B shows an example in which the head is shielded by the structure (region G-9) and the detection unit 21 detects only the body portion G-6b.

The size and position of the portion detected as the torso, such as the regions G-6a and G-6b, do not necessarily correspond to the entire trunk. Further, it does not necessarily have to be represented by a rectangle, and may be any shape such as an ellipse or a polygon. It may also be expressed as a distribution.

In step S204, the reference area acquisition unit 23 acquires the reference area to be AF from the first image and outputs it to the comparison unit 24. The position of the reference region is designated by the user or by using the detection result by the detection unit 21 in the same manner as in S103 of the first embodiment. In the following description, the reference region will be described assuming that the head is preferentially selected, but the present invention is not limited to the head priority. When giving priority to the torso, it is advisable to replace the head and the torso in the following explanation.

If the designated position is a head region as shown in the region G-8 of FIG. 5A, the head region is used as a reference region. The head region acquired by the reference region acquisition unit 23 may be a region such as a face included in the head instead of the entire head as in the region G-8. When the designated position is the body region as shown in the region G-6a of FIG. 5A, the region G-8, which is the head region closest to the designated position, is the reference region. Is selected as. If the designated position is the torso area as shown in the area G-6b of FIG. 5B and the corresponding head area does not exist, the area G-6b at the specified position is selected as the reference area. do.

Hereinafter, the case where the body region is designated as the reference region is not shown, but the processing is the same as the case where the head region is designated as the reference region. When shooting continuously, the in-focus target area selected at the previous time is used as the reference area.

In steps S205 and S206, the second image and its depth information are acquired. Since the processing of S205 and S206 is the same as that of S104 and S105 of the first embodiment, the description thereof will be omitted.

In step S207, the detection unit 21 detects the head region and the body region from the second image. In step S208, the partial region extraction unit 22 extracts each partial region based on the depth information for each of the head region and the trunk region detected from the second image. Since the specific extraction method of the partial region has already been described in detail in S107 of the first embodiment, it is omitted here.

In step S209, the comparison unit 24 compares the reference region acquired from the first image with the partial region belonging to the head region detected in the second image using the depth information in each image. Perform processing. The specific method for comparing the reference region and the partial region is the same as the method described in S108 of the first embodiment.

In step S210, the selection unit 25 verifies the comparison result obtained in S209. As a verification content, it is determined whether or not a partial region of the head exists within a predetermined range of the second image. Here, the predetermined range is a tracking range in continuous shooting, but the range is not limited to a specific range. For example, since the tracking target is the human body, a predetermined range is set from the common sense movement speed of the human body and the frame rate of continuous shooting. As a verification result, if the partial region of the head exists within the predetermined range, the process proceeds to S211. If not, the process proceeds to S212. The case where the partial region of the head does not exist within the predetermined range is considered to be the case where the head region is shielded or the case where the head detection fails, as shown in FIG. 9B, for example.

In step S211th, the selection unit 25 selects the focusing target area. For example, in the same procedure as in S109 of the first embodiment, in the second image shown in FIG. 8 (b) with respect to the reference region G-10 in the first image shown in FIG. 8 (a). The focusing target area is selected from the head partial areas G-11a and G-11b. After that, the process proceeds to S214.

In step S212, the comparison unit 24 compares the reference region acquired from the first image with the partial region belonging to the body region detected in the second image using the depth information in each image. Perform processing. The specific method for comparing the reference region and the partial region is the same as the method described in S108 of the first embodiment. When the comparison process is completed, the process proceeds to S213.

In step S213, the selection unit 25 determines the focusing target region from the comparison result obtained in S212. For example, in the same procedure as in S109 of the first embodiment, in the second image shown in FIG. 9 (b) with respect to the reference region G-10 in the first image shown in FIG. 9 (a). The in-focus target area is selected from the body partial areas G-7a and G-7b of the above. After that, the process proceeds to S214.

In step S214, the area selection device 20 determines whether or not to continue the AF process. If the AF process is to be continued, the process proceeds to S215, and if not, the process is terminated. In step S215, the area selection device 20 replaces the second image with the first image. After that, the process returns to S204 and the process is repeated.

As described above, according to the second embodiment, when the portion that was the reference region in the preceding image cannot be detected in the image to be processed due to shielding or the like, a partial region suitable for AF is combined in the detected human body region. Select as the focus area. As a result, the AF system can suitably continuously execute AF even when shielding or the like is present.

In the present embodiment, the case where the detection unit 21 detects the body region and the reference area acquisition unit 23 acquires the reference region based on the head region has been described, but it is applied to other than the combination of the head and the body. It is possible. For example, it may be a combination of a face and the whole body, or it may be a combination of a single person and a group of dense people. In addition, it may be a combination of the license plate of the car and the whole body.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiment, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to publicize the scope of the invention.

10 Imaging device; 11 Image acquisition unit; 12 Distance measurement unit; 20 Area selection device; 21 Detection unit; 22 Partial area extraction unit; 23 Reference area acquisition unit; 24 Comparison unit; 25 Selection unit

Claims (18)

An image processing device that determines the in-focus target area of the image pickup device.
An acquisition means for acquiring a first region to be focused in a first image captured by the image pickup device at a first time point, and an acquisition means.
A detection means for detecting a second region as a candidate for focusing from a second image captured by the image pickup apparatus at a second time point following the first time point.
A determination means for determining an in-focus target region in the second image from one or more partial regions of the second region based on the first region and the second image.
An image processing device characterized by comprising. Further, it has a depth information acquisition means for acquiring depth information for each region of the image captured by the image pickup device.
The determining means is characterized in that the focusing target region in the second image is determined based on the depth information corresponding to the first region and the depth information corresponding to the second region. The image processing apparatus according to claim 1. The feature is that the in-focus target region is determined based on the difference in depth calculated by the determination means and the depth information corresponding to the first region and the depth information corresponding to the second image. The image processing apparatus according to claim 1 or 2. The determination means sets a partial region of the one or more partial regions in which the difference between the depth of the first region and the depth of each of the one or more partial regions is smaller than a predetermined threshold value as the focusing target region. The image processing apparatus according to claim 3, wherein the image processing apparatus is determined to be. When there are a plurality of partial regions whose difference from the depth of the first region is less than a predetermined value, the determination means has a relatively large area or a relatively small depth among the plurality of partial regions. The image processing apparatus according to claim 4, wherein the partial area is determined to be the focusing target area. The image processing apparatus according to any one of claims 2 to 5, wherein the depth information is based on the distance between the image pickup device and the subject or the phase difference of the incident light in the image pickup device of the image pickup device. Claims 1 to 6 are characterized in that the detection means detects a region having image features similar to the image features extracted from the first region in the second image as the second region. The image processing apparatus according to any one of the above items. The acquisition means acquires a specific part of a person or an object as the first region, and obtains it.
The image processing apparatus according to claim 7, wherein the detection means detects the same portion as the specific portion as the second region. The acquisition means acquires a specific part of a person or an object as the first region, and obtains it.
The image processing apparatus according to claim 8, wherein the detection means detects a portion different from the specific portion as the second region when the same portion as the specific portion cannot be detected. Further having an extraction means for extracting one or more partial regions satisfying a predetermined condition from the second region.
The invention according to any one of claims 1 to 9, wherein the determination means determines an in-focus target region in the second image from the one or more partial regions extracted by the extraction means. Image processing equipment. The extraction means is characterized in that, of the second region, a region whose depth indicated by the depth information corresponding to the second image is included in a predetermined range is extracted as one or more partial regions. The image processing apparatus according to claim 10. 10. The extraction means is characterized in that the predetermined condition of the one or more partial regions is changed based on the depth of field when the image pickup apparatus captures the second image. Or the image processing apparatus according to 11. The determination means determines an in-focus target area in the second image based on a plurality of criteria.
Claims 1 to 12, wherein the plurality of criteria include the similarity between the first region and the second region, and the respective criteria for the area of the one or more partial regions. The image processing apparatus according to any one of the above items. 13. The image processing apparatus according to claim 13, wherein the determination means changes the plurality of criteria of the focusing target region based on the elapsed time from the first time point. The image processing apparatus according to any one of claims 1 to 14, wherein the second region is a region showing a head and / or a torso of a person. Claims 1 to 15 are characterized in that the determination means preferentially determines a partial region having an area larger than a predetermined value among the one or more partial regions as a focus target region in the second image. The image processing apparatus according to any one of the above items. It is a control method of an image processing device that determines an in-focus target area of an image pickup device.
The acquisition step of acquiring the first region to be focused in the first image captured by the image pickup device at the first time point, and the acquisition step.
A detection step of detecting a second region as a candidate for focusing from a second image captured by the image pickup apparatus at a second time point following the first time point.
A determination step of determining an in-focus target region in the second image from one or more partial regions of the second region based on the first region and the second image.
A control method characterized by including. A program for making a computer function as each means of the image processing apparatus according to any one of claims 1 to 16.

Images