JP6669376B2 - Image processing apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, a control method thereof, and a program, and more particularly, to a technique for calculating a visual saliency in detecting an attention area.

  2. Description of the Related Art In a conventional image processing apparatus, a technique for detecting a subject existing in an image by comparing feature amounts of two regions has been disclosed (see Patent Documents 1 and 2).

  In Patent Literature 1, a subject is detected by comparing a motion vector between a candidate area and a surrounding area. In Patent Literature 2, a remarkable region that is visually conspicuous is detected based on a difference in feature amount between a training data extraction region having a small radius and a verification data extraction region having a large radius. In other words, it is determined that the greater the difference between the feature amounts of the training data extraction region and the verification data extraction region that is the surrounding region, the higher the visually significant degree (hereinafter, referred to as “visually significant degree”). I have.

JP 2010-218361 A JP 2012-123631 A

  However, when detecting the subject by comparing the feature values of the two regions, if a surrounding region and an inner region smaller than the surrounding region are set at the center position, at a position where the inner region is in contact with the image edge, Part of the surrounding area protrudes outside the image. Therefore, there is a problem that the feature amount of the surrounding area cannot be calculated, and the subject cannot be detected.

  In Patent Documents 1 and 2 described above, examples are described in which a peripheral region is set around an inner region, but there is no specific reference to the positional relationship between the two regions.

  The present invention has been made in view of the above problems, and has an image processing apparatus capable of calculating a visual saliency to an end of an image and expanding a detection range of a subject, a control method thereof, and a program. The purpose is to provide.

  In order to achieve the above object, an image processing apparatus according to the present invention includes: a feature amount calculating unit configured to calculate a first feature amount in a first region of an image and a second feature amount in a second region of the image; A visual saliency calculating means for calculating a visual saliency in the first region or the second region based on a comparison result between the first characteristic amount and the second characteristic amount; Sets the first region and the second region at a plurality of positions in the image, calculates the first characteristic amount and the second characteristic amount at each position, and calculates the second region A relative position between the first area and the second area is changed according to a position in the image.

  According to the present invention, the relative positional relationship between the inner region and the surrounding region is changed according to the position in the image of the region for which the feature value is calculated. Thus, the visual saliency can be calculated up to the end of the image, and the detection range of the subject can be expanded.

FIG. 1 is a block diagram illustrating a schematic configuration of a digital camera that is an example of an image processing device according to a first embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration example of an attention area detection unit in FIG. 1. FIG. 5A is a diagram illustrating an example of an inner region and a surrounding region for detecting a region of interest, and FIG. 6B is a diagram illustrating a visual saliency that is a detected degree of interest. 9 is a flowchart illustrating an operation process of an attention area detection unit. FIG. 5 is a diagram for explaining a method of detecting a region of interest in the first embodiment. FIG. 6 is a diagram illustrating a YUV 4: 2: 2 image in a case where a horizontal sampling rate differs between luminance and color, one pixel at a time. FIG. 9 is a block diagram illustrating a schematic configuration of a digital camera that is an example of an image processing device according to a second embodiment of the present invention. FIG. 11 is a region layout diagram when the inner region is always located at the center of the surrounding region. It is a figure for explaining the detection method of the attention area in a 2nd embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram illustrating a schematic configuration of a digital camera which is an example of an image processing apparatus according to a first embodiment of the present invention.

  In FIG. 1, a digital camera 100 includes a lens 101 including a zoom lens and a focus lens, a shutter 102 having an aperture function, and an imaging unit 103 including a CCD sensor, a CMOS sensor, and the like for converting an optical image into an electric signal. The digital camera 100 includes an A / D converter 104 that converts an analog signal output from the imaging unit 103 into a digital signal, and a CCD sensor or a CMOS sensor that converts an optical image into an electric signal for AF control. The AF sensor 105 is provided.

  The digital camera 100 also performs white balance processing on image data output from the A / D converter for AF 106 and the A / D converter 104 for converting an analog signal output from the AF sensor 105 into a digital signal. An image processing unit 107 that performs gamma correction processing and the like is provided.

  The digital camera 100 includes an image memory 108, a memory control unit 109 that controls the image memory 108, and a D / A converter 110 that converts a digital signal input from the memory control unit 109 into an analog signal. In addition, the digital camera 100 includes a display unit 111 such as an LCD, and a codec unit 112 that compresses / encodes / decodes image data.

  Further, the digital camera 100 includes a recording medium 113 such as a memory card or a hard disk for storing image data, a recording medium I / F 114 serving as an interface with the recording medium 113, and an attention area detection unit 115 for detecting an attention area in an image. Prepare.

  Further, the digital camera 100 includes a system control unit 50 for controlling the entire device, an operation unit 120 for inputting operation instructions from various users, a power switch 121, and a power supply unit 122.

  The digital camera 100 includes a nonvolatile memory 123 such as an EEPROM, and a system timer 124 that measures time used for various controls and time of a built-in clock. Further, the digital camera 100 includes a system memory 125 for developing constants and variables for operation of the system control unit 50, programs read from the nonvolatile memory 123, and the like.

  Next, a basic operation of the digital camera 100 shown in FIG. 1 during shooting will be described.

  The imaging unit 103 photoelectrically converts the light incident through the lens 101 and the shutter 102 and outputs the light to the A / D converter 104 as an input image signal. The A / D converter 104 converts an analog image signal output from the imaging unit 103 into a digital image signal and outputs the digital image signal to the image processing unit 107. The AF sensor 105 receives the light incident through the lens 101 and the shutter 102 with a plurality of pairs of line sensors, and outputs the light to the A / D converter 106 for AF. The A / D converter for AF 106 converts an analog signal output from the AF sensor 105 into a digital signal and outputs the digital signal to the system control unit 50. The system control unit 50 detects a relative displacement amount of the light beam from the subject in the dividing direction based on the image signals output from the pair of line sensors, and performs so-called phase difference AF control.

  The image processing unit 107 performs various image processing such as white balance processing and gamma correction processing on the image data from the A / D converter 104 or the image data read from the memory control unit 109. The image data output from the image processing unit 107 is written to the image memory 108 via the memory control unit 109. Further, in the image processing unit 107, predetermined calculation processing is performed using the image data captured by the imaging unit 103, and the system control unit 50 performs exposure control and distance measurement control based on the obtained calculation results. Thus, AE (automatic exposure) processing, AF (autofocus) processing, and the like are performed.

  The image memory 108 stores image data output from the imaging unit 103 and image data to be displayed on the display unit 111. The D / A converter 110 converts the data for image display stored in the image memory 108 into an analog signal and supplies the analog signal to the display unit 111. The display unit 111 performs display on a display such as an LCD according to the analog signal from the D / A converter 110. The codec 112 compresses and encodes the image data stored in the image memory 108 based on a standard such as JPEG or MPEG.

  The attention area detection unit 115 detects an attention area in the image, and outputs attention area information to the system control unit 50. The system control unit 50 determines an area in which the predetermined process is prioritized based on the attention area information. The predetermined process includes, for example, a process of performing AF control so that, when a plurality of subjects are present in an image, the subject belonging to the attention area is focused. Further, when a plurality of subjects are present in the image, a process for performing AE control so that the subjects belonging to the attention area have appropriate brightness is included. The details of the attention area detection unit 115 will be described later.

  The system control unit 50 realizes various processes to be described later by executing a program stored in the nonvolatile memory 123 in addition to the basic operation.

  Next, the attention area detection unit 115 that detects the attention area in the image will be described with reference to FIGS.

  FIG. 2 is a block diagram illustrating a configuration example of the attention area detection unit 115 in FIG.

  The attention area detection unit 115 includes an inside area setting unit 201, a surrounding area setting unit 202, an inside area feature amount calculation unit 203, a surrounding area feature amount calculation unit 204, and a visual saliency calculation unit 205. Detailed operations of these units will be described later.

  FIG. 3A is a diagram illustrating an example of an inner region and a surrounding region for detecting a region of interest in an image, and FIG. 3B is a diagram illustrating a visual saliency that is a detected degree of interest. It is. FIG. 3B shows that the whiter, the higher the visual saliency.

  The attention area detection unit 115 moves the inside area a301 and the surrounding area b301 over the entire image based on the setting information from the system control unit 50, and compares the feature quantity of the inside area a301 with the feature quantity of the surrounding area b301 at a plurality of positions. I do. If the difference between the feature amount of the inner region a301 and the feature amount of the surrounding region b301 is large from the comparison result, the region is determined to be a conspicuous region, and a large visual saliency is calculated. On the other hand, when the difference between the feature value of the inner region a301 and the feature value of the surrounding region b301 is small, the region is determined to be inconspicuous, and a small visual saliency is calculated.

  As shown in FIG. 3A, when a ball is present in the background and the inner region is located around the ball, the difference between the feature amount of the inner region a301 and the feature amount of the surrounding region b301 is large, so that the difference is visually noticeable. The degree shows a large value. On the other hand, when the inner region is located other than around the ball, the difference between the feature amount of the inner region a301 and the feature amount of the surrounding region b301 is small, and the visual saliency shows a small value.

  As described above, the attention area detection unit 115 calculates the visual saliency at the inside area position while moving the inside area a301 and the surrounding area b301 in the entire image.

  As shown in FIG. 3A, when the inner area a301 is in a relative positional relationship that always exists at the center of the surrounding area b301, when the inner area a301 is in a position in contact with the edge of the image, a part of the surrounding area b301 becomes an image. It protrudes outside. Therefore, some pixel information of the surrounding area b301 is missing, and the visual saliency cannot be calculated. Therefore, the attention area detection unit 115 detects the attention area.

  FIG. 4 is a flowchart showing the flow of the attention area detection operation executed by each unit shown in FIG. FIG. 5 is a diagram for explaining the attention area detection operation shown in FIG.

  In step S401, the system control unit 50 determines the inner region initial position and the inner region moving amount, and the surrounding region initial position and the surrounding region moving amount. FIG. 5 shows an example in which a501 is determined as the inner region initial position, a511 is determined as the inner region movement amount, b501 is determined as the surrounding region initial position, and b511 is determined as the surrounding region movement amount. Here, in order to change the relative positional relationship between the inner region and the surrounding region according to the position in the image, the system control unit 50 determines the inner region moving amount to be larger than the surrounding region moving amount. . Details of the movement amount will be described later.

  In the present embodiment, the initial positions of the inner area and the surrounding area are defined as the initial start coordinates and the initial end coordinates, but the present invention is not limited to this. For example, the initial positions of the inner area and the surrounding area may be defined as the initial start coordinates and the area size.

  The inside area initial position and the inside area movement amount determined in step S401 are sent to the inside area setting unit 201 in the attention area detection unit 115, and the surrounding area initial position and the surrounding area movement amount are sent to the surrounding area setting unit 202. Sent out.

  Next, in step S402, the inside area setting unit 201 sets the inside area based on the inside area initial position and the inside area movement amount. Specifically, upper left coordinates (inner_sx, inner_sy) and lower right coordinates (inner_ex, inner_ey) of the rectangle indicating the inner area are calculated based on the following equations (1) to (4).

inner_sx = inner_inisx + inner_stepx * step_cntx (1)
inner_ex = inner_iniex + inner_stepx * step_cntx (2)
inner_sy = inner_inisy + inner_stepy * step_cnty (3)
inner_ey = inner_iniey + inner_stepy * step_cnty (4)
In the above equations (1) and (2), the variables are as follows. Note that the horizontal initial start coordinates, the horizontal initial end coordinates, and the horizontal movement amount are input from the system control unit 50. Also, the number of horizontal movements is initialized to 0 at the right end of the image immediately before the inner region protrudes outside the image.

inner_sx: horizontal start coordinates of the inner area inner_ex: horizontal end coordinates of the inner area inner_inisx: horizontal initial start coordinates of the inner area inner_iniex: horizontal initial end coordinates of the inner area inner_stepx: inner area horizontal movement amount.

step_cntx: Number of horizontal movements In the above equations (3) and (4),
inner_sy: vertical start coordinates of the inner area,
inner_ey: vertical end coordinates of the inner area,
inner_inisy: vertical initial start coordinates of the inner area,
inner_iniey: vertical initial end coordinates of the inner area,
inner_stepy: The vertical movement amount of the inner area. Note that the vertical initial start coordinates, the vertical initial end coordinates, and the vertical movement amount are input from the system control unit 50.

  Also, step_cnty: the number of vertical movements. Note that the number of vertical movements is initialized to 0 at the lower end of the image immediately before the inner region protrudes outside the image.

  Next, an operation of setting the inner region at a plurality of positions will be described with reference to FIG.

  First, a case where the inner region is set at a plurality of positions in the horizontal direction will be described.

  Since the inner area a501 in FIG. 5 belongs to the initial position, it is calculated as step_cntx = 0 and step_cnty = 0 according to the equations (1), (2), (3), and (4).

  Next, since the inner area a502 in FIG. 5 is an area that has been moved once in the horizontal direction from the inner area a501, step_cntx = 1, according to the equations (1), (2), (3), and (4). It is calculated as step_cnty = 0.

  Similarly, since the inner area a503 in FIG. 5 is an area that has moved horizontally twice from the inner area a501, step_cntx = 2 according to the equations (1), (2), (3), and (4). It is calculated as step_cnty = 0.

  Next, a case where the inner region is set at a plurality of positions in the vertical direction will be described.

  Since the inner area a504 in FIG. 5 is an area that has moved vertically once from the inner area a501, step_cntx = 0 and step_cnty = 1 according to the equations (1), (2), (3), and (4). Is calculated as

  Similarly, since the inner area a505 in FIG. 5 is an area that has moved vertically twice from the inner area a501, step_cntx = 0 according to equations (1), (2), (3), and (4). It is calculated as step_cnty = 2. Other inner regions are set by the same calculation method as described above.

  As described above, the inner area setting unit 201 sets the inner area at a plurality of positions in the image.

  The inside area set in step S402 is sent from the inside area setting unit 201 to the inside area feature amount calculation unit 203.

  Returning to FIG. 4, in step S403, the surrounding area setting unit 202 sets the surrounding area based on the surrounding area initial position and the surrounding area movement amount. Specifically, the upper left coordinates (surround_sx, surround_sy) and the lower right coordinates (surround_ex, surround_ey) of the rectangle indicating the surrounding area are calculated based on the following calculation formula.

surround_sx = surround_inisx + surround_stepx * step_cntx (5)
surround_ex = surround_iniex + surround_stepx * step_cntx (6)
surround_sy = surround_inisy + surround_stepy * step_cnty (7)
surround_ey = surround_iniey + surround_stepy * step_cnty (8)
In Equations (5) and (6), the variables are as follows. Note that the horizontal initial start coordinates, the horizontal initial end coordinates, and the horizontal movement amount are input from the system control unit 50. Also, the number of horizontal movements is initialized to 0 at the right end of the image immediately before the surrounding area extends outside the image.

surround_sx: the horizontal start coordinate of the surrounding area surround_ex: the horizontal end coordinate of the surrounding area surround_inisx: the horizontal initial start coordinate of the surrounding area surround_iniex: the horizontal initial end coordinate of the surrounding area surround_stepx: the horizontal movement amount of the surrounding area step_cntx: the number of horizontal movements And in equation (8), the variables are as follows: Note that the vertical initial start coordinates, the vertical initial end coordinates, and the vertical movement amount are input from the system control unit 50. In addition, the number of vertical movements is initialized to 0 at the lower end of the image immediately before the surrounding area extends outside the image.

surround_sy: the vertical start coordinate of the surrounding area surround_ey: the vertical end coordinate of the surrounding area surround_inisy: the vertical initial start coordinate of the surrounding area surround_iniey: the vertical initial end coordinate of the surrounding area surround_stepy: the vertical movement amount of the surrounding area, next step number, vertical movement The operation of setting an area at a plurality of positions will be described with reference to FIG.

  First, a case where the surrounding area is set at a plurality of positions in the horizontal direction will be described.

  Since the surrounding area b501 in FIG. 5 belongs to the initial position, it is calculated as step_cntx = 0 and step_cnty = 0 according to the equations (5), (6), (7), and (8).

  Next, since the surrounding region b502 in FIG. 5 is a region that has been moved once in the horizontal direction from the surrounding region b501, step_cntx = 1 according to Expressions (5), (6), (7), and (8). It is calculated as step_cnty = 0.

  Similarly, since the surrounding region b503 in FIG. 5 is a region that has moved horizontally twice from the surrounding region b501, step_cntx = 2 according to Expressions (5), (6), (7), and (8). It is calculated as step_cnty = 0.

  Next, a case where the surrounding area is set at a plurality of positions in the vertical direction will be described.

  The surrounding area b504 in FIG. 5 is an area that has moved once in the vertical direction from the surrounding area b501, and therefore, according to Equations (5), (6), (7) and (8), step_cntx = 0 and step_cnty = 1 Is calculated as

  Similarly, since the surrounding region b505 in FIG. 5 is a region that has moved vertically twice from the surrounding region b501, step_cntx = 0 according to Expressions (5), (6), (7), and (8). It is calculated as step_cnty = 2. Other surrounding areas are set by the same calculation method as described above.

  As described above, the surrounding area setting unit 202 sets the surrounding area at a plurality of positions in the image.

  The surrounding area set in step S403 is sent from the surrounding area setting unit 202 to the surrounding area feature amount calculating unit 204.

  Returning to FIG. 4, in step S404, the inside region feature amount calculation unit 203 calculates the feature amount of an image in the inside regions at a plurality of positions set by the inside region setting unit 201 in step S402. The calculated inside region feature amount is sent from the inside region feature amount calculation unit 203 to the visual saliency calculation unit 205.

  Next, in step S405, the surrounding area feature amount calculation unit 204 calculates the feature amount of the image in the surrounding areas at a plurality of positions set by the surrounding area setting unit 202 in step S403. The calculated surrounding region feature amount is sent from the surrounding region feature amount calculating unit 204 to the visual saliency calculating unit 205.

  Here, the feature amounts calculated by the above-described inner region feature amount calculation unit 203 and surrounding region feature amount calculation unit 204 are brightness, color, edge strength, brightness histogram, color histogram, and edge strength histogram in the region. It includes at least one of them.

  Next, in step S406, the visual saliency calculating unit 205 calculates the visual saliency based on the difference between the inner region feature and the surrounding region feature located around the inner region. For example, the visual saliency is calculated based on the difference between the feature amount of the inner region a501 and the feature amount of the surrounding region b501 shown in FIG. As a specific method of calculating the difference between the feature amounts, there is a method of calculating the visual saliency by evaluating the absolute value of the difference between the feature amounts. For example, when the feature amount is brightness, the absolute value of the difference between the average luminance value of the inner region and the average luminance value of the surrounding region is set as the visual saliency. When the feature amount is a brightness histogram, the absolute value of the difference between the brightness histogram of the inner region and the brightness histogram of the surrounding region for each bin is calculated, and the integrated value is used as the visual saliency.

  In step S407 in FIG. 4, the system control unit 50 determines whether or not all the search areas of the entire image have been searched, and if not, repeats the processing from step S402. On the other hand, if all the search areas of the entire image have been searched, the present process is terminated.

  Next, the initial position and the movement amount determined by the system control unit 50 will be described in detail with reference to FIG.

  The system control unit 50 determines that the initial positions of the inner area and the surrounding area are in contact with the upper left corner in the image as in a501 and b501 shown in FIG. Then, the system control unit 50 determines the horizontal inner region movement amount a511 and the surrounding region movement amount b511 in the upper right corner of the image such that the inner region is located at the upper right of the surrounding region. At this time, the horizontal inner region movement amount a511 is larger than the surrounding region movement amount b511.

  Similarly, in the vertical direction, the system control unit 50 determines the vertical inner region movement amount a511 and the peripheral region movement amount b511 such that the inner region is located at the lower left of the surrounding region at the lower left corner of the image. At this time, the vertical inner region movement amount a511 is larger than the surrounding region movement amount b511.

  As described above, the initial positions of the inner region and the surrounding region are determined, and a predetermined amount of movement is cumulatively added from the determined initial position to determine the positions of the inner region and the surrounding region. It will be located at the center of the surrounding area.

  As described above, by changing the relative position between the surrounding region and the inner region according to the position in the image, it is possible to calculate the visual saliency up to the end of the image.

  When the luminance sampling rate and the color sampling rate of the image data are different, the adjustment may be performed so that the inner area and the peripheral area match the boundaries of the pixel information sampled at a lower sampling rate.

  A case where the horizontal sampling rate differs between luminance and color (YUV 4: 2: 2 image) will be specifically described with reference to FIG.

  FIG. 6 is a diagram in which the YUV 4: 2: 2 image is represented one pixel at a time, and the areas calculated by the inner area setting unit 201 or the surrounding area setting unit 202 are indicated by 601 and 602. Note that, even when the sampling rate in the vertical direction is different, the area is adjusted by the same processing.

  When the horizontal start coordinates and the horizontal end coordinates of the inside area calculated by the formulas (1) and (2) are the area 601 in FIG. 6, the inside area setting unit 201 sets the boundary between the color pixel information having a low sampling rate. The horizontal start coordinates and the horizontal end coordinates are adjusted as shown in an area 602 in FIG.

  The surrounding area setting unit 202 also adjusts the horizontal start coordinates and the horizontal end coordinates by the same processing as the inside area setting unit 201.

  When the inner region protrudes outside the surrounding region, the inner region setting unit 201 may adjust the position of the inner region so that the inner region exists in the surrounding region. Similarly, when the inner region protrudes outside the surrounding region, the surrounding region setting unit 202 may adjust the position of the surrounding region so that the inner region exists in the surrounding region.

  According to the first embodiment, the relative positional relationship between the inner region and the surrounding region is changed according to the position in the image of the region for which the feature value is calculated. Thus, the visual saliency can be calculated up to the end of the image, and the detection range of the subject can be expanded.

[Second embodiment]
In the first embodiment, when the relative position between the inner region and the surrounding region is changed according to the position in the image as in the region arrangement shown in FIG. 5, the visual saliency can be calculated up to the end of the image.

  However, depending on the position in the image, there is a possibility that the relative positional relationship between the inner region and the surrounding region for calculating the difference between the feature amounts may be biased. For example, on the left side of the image, the difference in the feature amount between the inner region and the surrounding region biased to the right is calculated, and on the right side of the image, the difference in the feature amount between the inner region and the surrounding region biased to the left side is calculated. Is calculated. Therefore, the visual saliency can be more appropriately calculated when the inner region is always located at the center of the surrounding region. However, in this case, the visual saliency cannot be calculated up to the end of the image.

  In the second embodiment of the present invention, in order to solve the above-described trade-off, it is estimated whether or not a subject exists at an end of an image, and an area arrangement for calculating visual saliency is determined. . Note that the same parts as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted. Hereinafter, only differences from the first embodiment will be described.

  FIG. 7 is a block diagram illustrating a schematic configuration of a digital camera which is an example of an image processing device according to a second embodiment of the present invention.

  The digital camera 700 shown in FIG. 7 is different from the digital camera 100 shown in FIG. 1 only in that an image end subject existence estimating unit 701 is added, and the other components are the same as those in FIG.

  The image end subject existence estimating unit 701 estimates whether or not the subject exists at the image end, and outputs the position to the system control unit 50 when it is estimated that the subject exists. Then, the system control unit 50 calculates the initial position and the movement amount of the inner area and the surrounding area based on the estimation result by the image end subject existence estimating unit 701 and outputs the calculated initial position and the moving amount to the attention area detecting unit 115. Note that the configuration and operation of the attention area detection unit 115 are the same as in the first embodiment, and a description thereof will be omitted.

  Next, the image end subject existence estimating unit 701 will be described in detail with reference to FIGS.

  FIG. 8 is an area layout diagram when the inner area is always located at the center of the surrounding area. That is, in the illustrated example, the relative position between the inner region and the surrounding region does not change depending on the position in the image. A region 800 indicates a region where the inner region is arranged when the inner region is always located at the center of the peripheral region. That is, the attention area detection unit 115 can calculate the visual saliency only inside the area 800.

  The image end subject existence estimating unit 701 estimates whether or not a subject exists in an image end area outside the area 800. For example, by detecting a motion vector between frames, it is estimated whether or not the subject of the previous frame exists in the image end area of the current frame. Alternatively, it may be estimated whether or not a subject exists in the image end region by evaluating the inter-frame difference absolute value of the image end region. This makes it possible to estimate whether or not a new subject has entered the frame.

  When the image end subject existence estimating unit 701 estimates that the subject does not exist at the image end outside the region 800, the system control unit 50 determines that the inner region is always located at the center of the surrounding region as shown in FIG. The initial position and the movement amount are calculated and output to the attention area detection unit 115.

  When the image end subject existence estimating unit 701 estimates that the subject exists at the image end outside the region 800, the system control unit 50 can set the inner region up to the subject estimation position at the image end as shown in FIG. The initial position and the movement amount are calculated and output to the attention area detection unit 115. An area 900 illustrated in FIG. 9 indicates the position of the image end subject estimated by the image end subject existence estimation unit 701. In this case, as in the first embodiment, the relative position between the inner region and the surrounding region changes based on the position in the image.

  When there is pixel information of a region wider than the search region for which the target region is to be detected, the system control unit 50 sets the initial position so that the inner region is always located at the center of the surrounding region as shown in FIG. May be calculated and output to the attention area detection unit 115. For example, at the time of electronic zooming, when searching for an attention area only in the electronic zoom area, pixel information is present around the electronic zoom area, so that the inner area may be always arranged at the center of the surrounding area. It is not limited to the electronic zoom that the inner area is always arranged at the center of the surrounding area. The same applies to electronic image stabilization that requires margin pixel information for camera shake correction or to surround view in which an angle of view wider than the recording angle of view is displayed on the display unit.

  Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications and changes can be made within the scope of the gist.

  The present invention is also realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program and reads the program. This is the process to be performed.

50 System control unit 100 Digital camera 103 Imaging unit 107 Image processing unit 115 Attention area detection unit 201 Inner area setting unit 202 Surrounding area setting unit 203 Inner area feature amount calculation unit 204 Surrounding area feature amount calculation unit 205 Visual saliency calculation unit

Claims (12)

Feature value calculating means for calculating a first feature value in a first region of the image and a second feature value in a second region of the image;
A visual saliency calculating means for calculating a visual saliency in the first region or the second region based on a comparison result between the first characteristic amount and the second characteristic amount,
The feature amount calculation means sets the first region and the second region at a plurality of positions in the image, calculates the first feature amount and the second feature amount at each position, An image processing apparatus, wherein a relative position between the first region and the second region is changed according to a position of a second region in the image. The apparatus further includes position determination means for determining the positions of the first area and the second area,
2. The image processing apparatus according to claim 1, wherein the position determining unit determines the positions of the first area and the second area based on an initial position and a moving amount.   The image processing apparatus according to claim 2, wherein the position determining unit determines the positions of the first area and the second area by cumulatively adding a movement amount to the initial position.   The position determining means determines the positions of the first area and the second area based on the amount of movement of the second area different from the amount of movement of the first area. The image processing apparatus according to claim 2, wherein a relative position of the two areas is changed. The image processing apparatus according to claim 4, wherein the movement amount of the first region, larger than the movement amount of the second region.   The image processing apparatus according to claim 2, wherein the position determining unit corrects a position of the first area so as to be located inside the second area. A control unit that performs a predetermined process by giving priority to an area determined to have high visual saliency; the first area is located inside the second area;
6. The visual saliency calculating unit determines that the greater the difference between the first feature amount and the second feature amount is, the higher the visual saliency is. The image processing apparatus according to claim 1.   When calculating the visual saliency based on image signals having different sampling rates, the position determining means aligns the first area and the second area with a boundary of an image signal sampled at a low sampling rate. The image processing apparatus according to claim 2, wherein the position of the area is adjusted as described above. Estimating means for estimating whether or not a subject is present at an image end of the image,
Said position determining means, in accordance with the estimation result of the estimating means, to any one of claims 2 to 6, characterized in that to change the initial position and the movement amount of the first region and the second region The image processing apparatus according to any one of the preceding claims.   10. The position determination unit according to claim 9, wherein when the estimation unit estimates that the subject does not exist at the image end, the relative position between the first region and the second region is not changed. Image processing device. A feature value calculating step of calculating a first feature value in a first region of the image and a second feature value in a second region of the image;
A visual saliency calculating step of calculating a visual saliency in the first region or the second region based on a comparison result between the first characteristic amount and the second characteristic amount,
The feature amount calculating step sets the first region and the second region at a plurality of positions in the image, calculates the first feature amount and the second feature amount at each position, A method for controlling an image processing apparatus, comprising: changing a relative position between the first region and the second region according to a position of a second region in the image.   A computer-readable program for causing an image processing apparatus to execute the control method according to claim 11.