JP2009146033A - Partial image extraction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately extract a partial image including a part to which a user pays attention in an image. <P>SOLUTION: In a partial image extraction method, a partial image extraction section 104a sets a frame (A) including a designation point 2a designated by a user in an image, generates a plain image for each color component based on the image in the frame (A), performs binary coding of the generated plain image for each color component with a threshold set for each plain image, generates a mask image for each color component, calculates an evaluation value for determining whether the mask image is suitable for extracting a partial image with the mask image for each color component as a target, selects an optimum mask image most suitable for extracting a partial image among the mask images for each color component based on the calculated evaluation value, sets a frame (B) to extract a partial image in the selected optimum mask image, and extracts an image in an area corresponding to the frame (B) set in the optimum mask image as a partial image from the target images. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a partial image extraction method for extracting a partial image from an image.

  There are known methods for detecting a face area of a shooting screen as follows. In this face area detection method, the size of a person's face is estimated from the shooting magnification at the time of shooting, and the size of the skin color area in the screen is calculated. Then, by comparing the estimated face size of the person and the size of the skin color area on the screen, the face area of the person in the shooting screen is detected (for example, Patent Document 1).

Japanese Patent Laid-Open No. 07-306483

  However, according to the conventional method, the size of the person's face is estimated from the shooting magnification at the time of shooting, but the size of the person's face in the image also varies depending on the distance from the camera to the person. Therefore, the size estimated from the shooting magnification at the time of shooting is not always accurate, and if the estimated value is incorrect, there is a possibility that the human face area cannot be detected accurately.

The present invention relates to a partial image extraction method for extracting a partial image including a portion of interest from a target image represented by a color system having a plurality of color components. , Set a frame containing the specified point specified by the user, generate a color component image for each color component based on the image in the frame, and set each generated color component image for each color component image To generate a mask image for each color component and determine whether the mask image is suitable for extracting a partial image for each mask image for each color component. An evaluation value is calculated, and based on the calculated evaluation value, a mask image most suitable for extracting a partial image is selected as an optimal mask image from the mask images for each color component, and the selected optimal mask image And partial image Set the extraction area for extraction, from the target image, the image in the region corresponding to the extracted area set within an optimal mask image, and extracts a partial image.
In the present invention, the position and size of the extraction region set in the optimal mask image may be determined based on the barycentric position of the white pixels in the optimal mask image and the number of white pixels.
A density average value or a median density value may be calculated as a threshold value for each color component image, and a mask image for each color component may be generated using the threshold value calculated for each color component image.
The evaluation value may be calculated based on the number of white pixels in the mask image for each color component and the moment of inertia around the center of gravity of the white pixels.
Weighting is performed according to the distance from the centroid position of the white pixel in the optimal mask image to each white pixel to correct the centroid position of the white pixel so that the corrected centroid position of the white pixel becomes the extraction area center position. The position of the extraction region set in the optimal mask image may be determined.

  ADVANTAGE OF THE INVENTION According to this invention, the partial image containing the part which the user is paying attention in an image can be extracted correctly.

  FIG. 1 is a block diagram showing the configuration of an embodiment when a camera is used as the partial image extraction apparatus in the present embodiment. The camera 100 includes an input switch 101, a lens 102, an image sensor 103, a control device 104, a memory 105, a memory card slot 106, and a monitor 107.

  The input switch 101 includes various input members operated by the user. For example, the input switch 101 includes a release button and various operation buttons for operating the camera 100. The lens 102 is composed of a plurality of optical lens groups, but is represented by a single lens in FIG.

  The image sensor 103 is, for example, a CCD or a CMOS, images a subject image input through the lens 102, and outputs image data obtained by the imaging to the control device 104. The memory 105 is, for example, a non-volatile flash memory, in which data of a program executed by the control device 104, various parameters read at the time of program execution, and the like are recorded.

  The memory card slot 106 is a slot for inserting a memory card as a storage medium, and writes and records the image file output from the control device 104 on the memory card. Further, based on an instruction from the control device 104, an image file stored in the memory card is read.

  The monitor 107 is a liquid crystal monitor (rear monitor) mounted on the back of the camera 100, and the monitor 107 displays an image stored in a memory card, a setting menu for setting the camera 100, and the like. .

  The control device 104 includes a CPU, a memory, and other peripheral circuits. The control device 104 outputs and displays image data input from the image sensor 103 on the monitor 107, and further, in the memory card inserted in the memory card slot 106. To record. Note that the memory constituting the control device 104 includes ROM and RAM.

  The control device 104 also functionally includes a partial image extraction unit 104a, an AF (autofocus) processing unit 104b, and an imaging processing unit 104c. In the camera 100 according to the present embodiment, the control device 104 displays a through image on the monitor 107 by acquiring image data from the image sensor 103 at predetermined time intervals and outputting the image data to the monitor 107. When the release button included in the input switch 101 is half-pressed by the user, AF (autofocus) processing is performed. When the release button is further fully pressed, shooting is performed to acquire image data, and the memory card is obtained. To record.

  In the present embodiment, when a through image is input from the image sensor 103, the partial image extraction unit 104a extracts an image (partial image) of a portion that the user is paying attention from within a frame with the through image. To do. The AF processing unit 104b executes AF processing based on the partial image extracted by the partial image extraction unit 104a at the timing when the release button is half-pressed by the user. As a result, even when it is difficult to focus with the conventional method, such as when the subject is moving violently, it is possible to focus on the portion that the user is paying attention to. The AF process is a known process and will not be described. Hereinafter, the partial image extraction process from the image by the partial image extraction unit 104a will be described in detail.

  First, the user calls the menu screen by operating the input switch 101 and selects “extract partial image” on the menu screen to instruct the start of processing. The partial image extraction unit 104a displays a through image on the monitor 107 when the user instructs the start of processing. The user operates the input switch 101 to set a designated point in the image for specifying the position of the subject of interest in the image displayed on the monitor 107. For example, when the user is paying attention to the face of a person on the image shown in FIG. 2, a designated point (user designated point) 2a is set on the face.

  The partial image extraction unit 104a extracts a partial image of a portion that is noticed by the user from the image based on the position of the designated point 2a set in the image. In the example shown in FIG. 2, the designated point 2 a is off the face of the person, but the partial image extraction unit 104 a performs processing within the region including the face of the person that the user is paying attention to by the following processing. Can be extracted as a partial image.

  The partial image extraction unit 104a sets a frame A in the image so as to surround a predetermined range centered on the designated point 2a set by the user. Then, the partial image extraction unit 104a extracts an image in the frame A, and based on the extracted image in the frame A, an image for each color component (color component image), that is, an image of a Y component (Y plane image). , A Cr component image (Cr plane image) and a Cb component image (Cb plane image) are respectively generated.

  That is, when the image data to be processed is image data represented in the YCbCr color system, the partial image extraction unit 104a uses a luminance image composed of a Y component (luminance component) as a Y plane image as a Cb image. A color difference image composed of components (color difference components) is generated as a Cb plane image, and a color difference image composed of Cr components (color difference components) is generated as a Cr plane image.

  On the other hand, when the image data to be processed is represented by another color system, the partial image extraction unit 104a converts the image data in the frame A into image data of the YCbCr color system in advance. Convert it. For example, when the image data to be processed is expressed in the RGB color system, the partial image extraction unit 104a converts the image data expressed in the RGB color system into the following expressions (1) to (3). ) Is used to convert a luminance image composed of luminance components and a color difference image composed of color difference components in the YCbCr color space.

That is, the partial image extraction unit 104a generates a luminance image composed of a Y component as a Y plane image using the following equation (1) for the image data in the frame A, and the following equations (2) and (3): Is used to generate a Cb component color difference image and a Cr component color difference image as a Cb plane image and a Cr plane image, respectively.
Y = 0.299R + 0.587G + 0.114B (1)
Cb = −0.169R−0.332G + 0.500B (2)
Cr = 0.500R−0.419G−0.081B (3)

  The partial image extraction unit 104a creates a density histogram with the horizontal axis representing the density value (0 to 255) and the vertical axis representing the number of pixels for the generated Y plane image, Cb plane image, and Cr plane image. That is, for each of the Y plane image, the Cb plane image, and the Cr plane image, the density value of all the pixels in the image is examined, the number of pixels is counted for each density value, and the number of pixels for each density value appears. Create a density histogram showing the frequency.

  As a result, for example, a density histogram as shown in FIG. 3 is obtained. The density histogram shown in FIG. 3 is created for any one of a Y plane image, a Cb plane image, and a Cr plane image, for example, a Y plane image, and for other Cb plane images and Cr plane images. Is created in the same way.

  The partial image extraction unit 104a calculates the median value of the density histogram of each plane image created in this way, that is, the density median value (Median) 3a. Then, the calculated density median value 3a is set as a threshold value, and each of the generated Y plane image, Cb plane image, and Cr plane image is binarized to generate a binary mask image. As a result, as shown in FIG. 4, a binary mask image for each plane image can be generated.

  That is, the partial image extraction unit 104a generates a Y mask image shown in FIG. 4A by binarizing the Y plane image using the density median value 3a of the Y plane image as a threshold, and generates the Cb plane. By binarizing the image using the median density value 3a of the Cb plane image as a threshold value, a Cb mask image shown in FIG. 4B is generated. Also, the partial image extraction unit 104a generates a Cr mask image shown in FIG. 4C by binarizing the Cr plane image using the density median value 3a of the Cr plane image as a threshold value. Further, the partial image extraction unit 104a generates the Y complement mask image shown in FIG. 4D by inverting the Y mask image.

  In the present embodiment, an example in which a mask image is generated by binarizing each plane image using the density median value 3a as a threshold value has been described. However, other values are used as threshold values. Also good. For example, the average value of the density histogram of each plane image, that is, the density average value (Mean) 3b may be calculated and used as a threshold value.

  The partial image extraction unit 104a calculates an evaluation value for each mask image in order to select a mask image used for extracting the partial image from the generated mask images. Specifically, the partial image extraction unit 104a first extracts white pixels of each mask image. Then, the number of white pixels in the mask image is calculated for each mask image, and the barycentric position of the white pixels in the mask image is specified.

  Then, the partial image extraction unit 104a calculates, for each mask image, the moment of inertia around the centroid of the white pixel based on the centroid position of the white pixel specified in the mask image. That is, the partial image extraction unit 104a performs the moment of inertia around the center of gravity of the white pixel of the Y mask image, the moment of inertia around the center of gravity of the white pixel of the Cb mask image, the moment of inertia around the center of gravity of the white pixel of the Cr mask image, The moment of inertia around the center of gravity of the white pixel of the complement mask image is calculated. A method for calculating the moment of inertia around the center of gravity of the white pixel in the mask image obtained by binarization is well known and will not be described in detail. For example, the square of the pixel distance from the center of gravity of the white pixel × It can be calculated by the sum of 0 or 1 density values.

The partial image extraction unit 104a calculates the evaluation value of each mask image by the following equation (4) based on the number of white pixels calculated for each mask image by the above-described processing and the moment of inertia around the center of gravity of the white pixels. calculate. In the following equation (4), the coefficient is a preset value.
Evaluation value = (number of white pixels / moment of inertia) × coefficient (4)

  For example, for each mask image shown in FIG. 4, the evaluation value when the coefficient is calculated as 10,000 is shown at the bottom of each mask image in FIG. The partial image extraction unit 104a selects a mask image having the largest calculated evaluation value as an optimum mask image for extracting a partial image from the mask images. That is, in the example illustrated in FIG. 4, the partial image extraction unit 104 a selects the Cr mask image having the largest evaluation value as the optimal mask image.

  The partial image extracting unit 104a extracts a frame B (extraction in the selected optimum mask image with the center as the barycentric position of the white pixel in the mask image and the size as the square root of the number of white pixels calculated in the mask image. Area). If the square root of the number of white pixels is a decimal number, the partial image extraction unit 104a rounds it up to an integer and sets this to the size of the frame B. For example, as shown in FIG. 5, the partial image extraction unit 104a sets a frame B in the Cr mask image selected as the optimal mask image. The partial image extraction unit 104a extracts, as a partial image, an image in an area corresponding to the frame B set in the optimal mask image from the through image shown in FIG.

  As a result, as shown in FIG. 6, an image in the frame B including the subject focused by the user is extracted as a partial image from the frame A set based on the designated point 2a designated by the user. be able to. As described above, the designated point 2a is not included in the face of the person actually focused on by the user, but the extracted partial image includes the face of the person focused on by the user. It can be an image.

  The AF processing unit 104b performs known AF processing on the partial image extracted by the partial image extraction unit 104a at the timing when the release button is half-pressed by the user, and performs focusing. Thereafter, when the release button is fully pressed by the user, the imaging processing unit 104c performs imaging by recording image data input from the imaging element 103 on a memory card.

  FIG. 7 is a flowchart showing processing of the camera 100 in the present embodiment. As described above, the process shown in FIG. 7 is executed by the control device 104 as a program to be started when the user operates the input switch 101 and instructs “extract partial image” on the menu screen.

  In step S10, the partial image extraction unit 104a displays the above-described through image on the monitor 107, and proceeds to step S20. In step S20, the partial image extraction unit 104a determines whether or not the designated point 2a is instructed by the user on the through image. If the determination is affirmative, the process proceeds to step S30.

  In step S30, the partial image extraction unit 104a sets a frame A in the image so as to surround a predetermined range centered on the designated point 2a designated by the user. Thereafter, the process proceeds to step S40, where the partial image extraction unit 104a extracts the image in the frame A, and based on the extracted image in the frame A, the Y plane image, the Cr plane image, and the Cb plane image described above are extracted. Generate each. Thereafter, the process proceeds to step S50.

  In step S50, the partial image extraction unit 104a creates the density histogram shown in FIG. 3 for each plane image, and calculates the density median (Median) 3a of the density histogram of each plane image. Then, the partial image extraction unit 104a generates a mask image corresponding to each plane image as shown in FIG. 4 by binarizing each plane image using the calculated density median value 3a as a threshold value. To do. Further, the partial image extraction unit 104a generates a Y complement mask image by inverting the generated Y mask image. Thereafter, the process proceeds to step S60.

  In step S60, as described above, the partial image extraction unit 104a uses the equation (4) to calculate each mask image based on the number of white pixels calculated for each mask image and the moment of inertia around the centroid of the white pixels. The evaluation value of is calculated. Thereafter, the process proceeds to step S70, where the partial image extraction unit 104a selects a mask image having the largest calculated evaluation value as an optimum mask image for extracting the partial image from each mask image, and then proceeds to step S80. .

  In step S80, as described above, the partial image extraction unit 104a uses the center of the selected mask image as the center of gravity of the white pixels in the mask image, and the number of white pixels calculated in the mask image. A frame B that is the square root of is set. Thereafter, the process proceeds to step S90, where the partial image extraction unit 104a extracts an image in an area corresponding to the frame B set in the mask image from the through image, and extracts this as a partial image. Then, it progresses to step S100.

  In step S100, the AF processing unit 104b determines whether or not the release button is half-pressed by the user. If the determination is affirmative, the process proceeds to step S110, where the AF processing unit 104b performs a known AF process on the partial image extracted by the partial image extraction unit 104a, and performs focusing. Then, it progresses to step S120.

  In step S120, the imaging processing unit 104c determines whether or not the release button has been fully pressed by the user. If the determination is affirmative, the process proceeds to step S130, and the imaging processing unit 104c performs imaging by recording the image data input from the image sensor 103 on a memory card. Thereafter, the process ends.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The partial image extraction unit 104a sets a frame A including the designated point 2a designated by the user in the image, generates a plain image for each color component based on the image in the frame A, The generated plane image of each color component was binarized using a threshold value set for each plane image, and a mask image for each color component was generated. Also, a Y complement mask image was generated by inverting the Y mask image. Then, the partial image extraction unit 104a calculates, for each mask image for each color component, an evaluation value for determining whether the mask image is suitable for extracting the partial image, and the calculated evaluation value Based on the above, the most suitable mask image for extracting a partial image from the mask images for each color component was selected as the optimum mask image. Furthermore, the partial image extraction unit 104a sets a frame B for extracting the partial image in the selected optimal mask image, and an area corresponding to the area in the frame B set in the optimal mask image from the target image. The image inside was extracted as a partial image. This makes it possible to accurately extract a partial image that includes a portion that the user is paying attention to in the image.

(2) The partial image extraction unit 104a determines the position and size of the frame B set in the optimal mask image based on the barycentric position of the white pixels in the optimal mask image and the number of white pixels. did. In this way, in the mask image obtained by binarization, the user pays attention to the fact that the range corresponding to the subject image is represented as a white area and the other background area is represented as a black area. The frame B can be set in the optimal mask image so as to include an area including a subject that is likely to be detected.

(3) The partial image extraction unit 104a calculates the density average value or the median density value as a threshold value for each plane image of each color component, and uses the threshold value calculated for each color component image to mask each color component. An image was generated. Thereby, each plane image can be binarized using an optimum threshold value.

(4) The partial image extraction unit 104a calculates the evaluation value based on the color (4) based on the number of white pixels in the mask image for each color component and the moment of inertia around the center of gravity of the white pixels. . As a result, if there are many white pixels in the mask image and the moment of inertia around the centroid of the white pixels is small, it means that many white areas are concentrated around the centroid of the white pixels. This indicates that there is a high possibility that a subject exists around the center of gravity of the white pixel, so that the evaluation value of such a mask image is increased to extract a partial image. An optimal mask image can be selected.

-Modification-
The camera according to the above-described embodiment can be modified as follows.
(1) In the embodiment described above, it is assumed that the image data to be processed is represented in the YCbCr color system. However, similar processing is possible even when the image data to be processed is image data in the YIQ color system.

(2) In the above-described embodiment, the example in which the AF processing is performed on the partial image extracted from the image by the partial image extraction unit 104a has been described. However, the present invention is not limited to this, and the exposure process may be performed on a partial image extracted from the image by the partial image extraction unit 104a. Alternatively, the partial image extracted from the image by the partial image extraction unit 104a may be extracted as a template image and used for subject tracking between frames input in time series.

(3) In the above-described embodiment, the partial image extraction unit 104a uses the center of the selected optimal mask image as the center of gravity of the white pixel in the mask image and the size calculated in the mask image. An example has been described in which a frame B is set as the square root of the number of pixels, and an image in an area corresponding to the frame B set in the optimal mask image is extracted as a partial image from the through image. However, when the frame B is set so that the center of gravity of the white pixel in the mask image is centered in this way, when the white pixel exists at a position unrelated to the subject such as the edge of the image. May be set so that the frame B deviates from the subject that the user is paying attention to. For this reason, in order to solve such a problem, the partial image extraction unit 104a may correct the barycentric position of the white pixels in the mask image as follows.

  For example, as a result of performing the above-described processing on the image shown in FIG. 8A, as shown in FIG. 8B, the Y mask image is selected as the optimum mask image, and the frame B is included in the Y mask image. Is set. In the example shown in FIG. 8B, the position of the center of gravity of the white pixel deviates from the subject position due to the influence of the upper left white pixel in the mask image, so that the frame B is set deviated from the subject. . In this modification, the partial image extraction unit 104a corrects the barycentric position of the white pixel so that the frame B is set to an appropriate position as shown in FIG. 8C.

  FIG. 9 is a diagram showing the distribution of white pixels in the Y mask image shown in FIG. In FIG. 9, “1” represents a white pixel and “0” represents a black pixel. The number on the right side of FIG. 9 represents the total number of white pixels included in the corresponding column, and the number on the lower side represents the total number of white pixels included in the corresponding row. The center position of the white pixel calculated when the white pixels are distributed as shown in FIG. 9 is the pixel 9a. Here, the pixel 9a is referred to as a temporary center of gravity.

  The partial image extraction unit 104a calculates the distance from the temporary centroid 9a for each pixel arranged in the Y direction (vertical direction) and the X direction (horizontal direction) from the temporary centroid 9a. For example, the pixel one pixel above the temporary centroid 9a has a distance in the Y direction of 1, and the pixel two pixels to the right of the temporary centroid 9a has a distance in the X direction of 2. The partial image extraction unit 104a specifies the maximum distance value in the Y direction and the maximum distance value in the X direction among the distances calculated in this way. In the example shown in FIG. 9, the maximum value of the distance in the Y direction is 11 of the distance to the top pixel, and the maximum value of the distance in the X direction is 9 of the distance to the rightmost pixel. .

  The partial image extraction unit 104a subtracts the distance to each pixel for which the distance in the Y direction is calculated from the specified maximum value +1 (11 + 1 = 12) of the distance in the Y direction. As a result, the calculation result (Y direction calculation result) shown on the left side of FIG. 9 is obtained for each pixel in the Y direction. Further, the partial image extraction unit 104a subtracts the distance to each pixel for which the distance in the X direction is calculated from the specified maximum value +1 (9 + 1 = 10) of the distance in the X direction. As a result, the calculation result (X direction calculation result) shown on the upper side of FIG. 9 is obtained for each pixel in the X direction.

The partial image extraction unit 104a corrects the barycentric position of the white pixel in the Y mask image by performing weighting using the X direction calculation result and the Y direction calculation result thus obtained. Specifically, the partial image extraction unit 104a can calculate the coordinate values (x _w , y _w ) of the corrected center of gravity 9b by performing weighting according to the equation shown in FIG. Incidentally, FIG. 10 (b) is a diagram showing a calculation example of a molecule of x _w, FIG. 10 (c) is a diagram showing an example of calculation of the denominator of x _w,

  By calculating the corrected center of gravity 9b and setting the frame B around the corrected center of gravity 9b, white pixels that are far from the temporary center of gravity 9a have a small weight and contribute to the center of gravity calculation. Since it becomes difficult, the influence by surrounding noise can be suppressed and the frame B can be set at an appropriate position.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired.

It is a block diagram which shows the structure of one Embodiment of a camera. It is a figure which shows the specific example of the designated point 2a set on the image. It is a figure which shows the specific example of the density histogram produced based on the plain image. It is a figure which shows the specific example of the mask image produced | generated by binarizing a plain image. It is a figure which shows the specific example at the time of setting the flame | frame B in Cr mask image. It is a figure which shows the specific example of the partial image extracted from the inside of an image. FIG. 6 is a flowchart showing processing of the camera 100. It is a figure which shows the example of a setting of the flame | frame B in a modification (3). It is a figure which shows the example of distribution of the white pixel in a modification (3). It is a figure which shows the specific example of the correction formula of the gravity center position of the white pixel in a modification (3).

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Camera, 101 Input switch, 102 Lens, 103 Image pick-up element, 104 Control apparatus, 104a Partial image extraction part, 104b AF process part, 104c Shooting process part, 105 Memory, 106 Memory card slot, 107 Monitor

Claims (5)

A partial image extraction method for extracting a partial image including a portion of interest by a user from a target image represented by a color system having a plurality of color components,
A frame including a designated point designated by the user is set in the target image,
Based on the image in the frame, generate a color component image for each color component,
Each generated color component image is binarized using a threshold set for each color component image to generate a mask image for each color component,
For each of the mask images for each color component, calculate an evaluation value for determining whether the mask image is suitable for extracting the partial image,
Based on the calculated evaluation value, a mask image most suitable for extracting the partial image from the mask images for each color component is selected as an optimal mask image,
In the selected optimal mask image, set an extraction region for extracting the partial image,
A partial image extraction method, wherein an image in an area corresponding to the extraction area set in the optimum mask image is extracted as the partial image from the target image. The partial image extraction method according to claim 1,
A partial image extraction method for determining a position and a size of the extraction region set in the optimum mask image based on a barycentric position of white pixels in the optimum mask image and the number of white pixels. . The partial image extraction method according to claim 1 or 2,
A partial image characterized by calculating a density average value or a median density value as a threshold value for each color component image, and generating the mask image for each color component using the threshold value calculated for each color component image Extraction method. In the partial image extraction method as described in any one of Claims 1-3,
A partial image extraction method, wherein the evaluation value is calculated based on the number of white pixels in the mask image for each color component and the moment of inertia around the center of gravity of the white pixels. In the partial image extraction method according to claim 2,
Weighting according to the distance from the center of gravity of the white pixel in the optimal mask image to each white pixel is performed to correct the center of gravity of the white pixel, and the center of gravity of the white pixel after correction is the center position of the extraction region The partial image extraction method is characterized in that the position of the extraction region to be set in the optimum mask image is determined.

Images