JP5146223B2 - Program, camera, image processing apparatus, and image contour extraction method - Google Patents

Description

The present invention, program, camera, relates contour extraction method of the image processing apparatus and an image.

Conventionally, with respect to digital captured images acquired with an electronic camera or the like, an edge amount included in a region of interest of the image is calculated, and the edge amount is compared with a threshold value to extract a contour and perform various processes. The technique is known (see Patent Document 1 as an example).
JP 2003-274258 A

  However, when the contour is extracted from the captured image by the above-described conventional technique, if the noise included in the image increases, the edge amount is likely to be erroneously detected in a flat portion of the image, and the image is captured with particularly high imaging sensitivity. The accuracy of contour extraction can be greatly reduced for the processed image. Therefore, in order to suppress the influence of noise in the above-described conventional technology, for example, it is complicated to set parameters for noise countermeasures in consideration of imaging sensitivity or to perform noise removal processing according to imaging sensitivity. There is room for improvement in that it requires a special treatment.

  Accordingly, an object of the present invention is to provide means for accurately extracting a contour from a captured image by a process that is not easily affected by noise and does not require parameter adjustment.

A program according to one aspect causes a computer to execute an image reading process for acquiring a captured image, a local region setting process, a gradient information calculation process, and a contour extraction process. In the local region setting process, the computer designates two or more pixels of interest from among the pixels included in the captured image, and sets a local region of a predetermined size in the captured image based on the position of each pixel of interest. In gradient information calculation processing, the computer solves an eigenvalue problem of principal component analysis using the gradient information of pixel values obtained from a plurality of pixels included in the local region as a variable, the sum of the gradient information in the local region as a variable. Thus , an image gradient vector indicating the gradient of the image at the position of each target pixel is obtained. In the contour extraction process, the computer extracts a contour from the captured image using information on each image gradient vector.

  In the present invention, it is possible to accurately extract a contour from a captured image by using information on an image gradient vector related to an image gradient at the positions of a plurality of target pixels.

<Description of One Embodiment>
FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment. The image processing apparatus according to one embodiment is a personal computer in which an image processing program is installed. The computer 11 constituting the image processing apparatus can extract a contour from the input captured image by executing the image processing program.

  The computer 11 includes a data reading unit 12, a storage device 13, a CPU 14, a memory 15, an input / output I / F 16, and a bus 17. The data reading unit 12, the storage device 13, the CPU 14, the memory 15, and the input / output I / F 16 are connected to each other via a bus 17. Further, an input device 18 (keyboard, pointing device, etc.) and a monitor 19 are connected to the computer 11 via an input / output I / F 16. The input / output I / F 16 receives various inputs from the input device 18 and outputs display data to the monitor 19.

  The data reading unit 12 is used when reading captured image data or an image processing program from the outside. For example, the data reading unit 12 communicates with a reading device (such as an optical disk, a magnetic disk, or a magneto-optical disk reading device) that acquires data from a removable storage medium, or an external device in accordance with a known communication standard. It consists of communication devices (USB interface, LAN module, wireless LAN module, etc.) to be performed.

  The storage device 13 stores the image processing program and various data necessary for executing the program. The storage device 13 can also record captured image data acquired from the data reading unit 12. Note that the storage device 13 in one embodiment is configured by a hard disk, a nonvolatile semiconductor memory, or the like.

  The CPU 14 is a processor that comprehensively controls the operation of each part of the computer 11. Further, the CPU 14 obtains an image gradient vector indicating the gradient of the edge of the image at a predetermined position of the captured image by executing the image processing program. Then, the CPU 14 uses the information of the image gradient vector to extract a contour from the captured image (note that the details of the above arithmetic processing will be described later). The memory 15 temporarily stores the calculation result of the image processing program. The memory 15 is composed of, for example, a volatile SDRAM.

  Next, the outline extraction process of the captured image by the image processing program of one embodiment will be described.

  In general, noise in a captured image exists regardless of the structure of the image and the direction of the edge, so that the amount of edge due to the noise of the image has no directivity. On the other hand, in a normal captured image, it is known that a strong correlation occurs between pixel values in the structure direction in the local region due to the structure of the image. In the local region, the difference in pixel values in the direction with strong correlation (direction in which the image structure is similar) shows a small value, and the direction perpendicular to the direction with strong correlation (direction perpendicular to the edge tangent) The difference between the pixel values is a large value.

  Therefore, the image processing program according to the embodiment extracts the contour from the captured image based on the information of the image gradient vector indicating the edge gradient in the local region, which is obtained by using the above-described property and using the principal component analysis method. To do.

  Hereinafter, an example of the contour extraction process by the image processing program of the embodiment will be described with reference to the flowchart of FIG. 2 is started in response to a program execution instruction from the user.

  Step S101: The CPU 14 obtains data of a captured image to be processed from the outside via the data reading unit 12. The captured image data acquired in S101 is recorded in the storage device 13 or the memory 15 under the control of the CPU 14. Note that the CPU 14 may omit the processing of S101 when captured image data is stored in the storage device 13 in advance.

  Step S102: The CPU 14 designates the position of the pixel of interest X for which the image gradient vector is obtained from the plurality of pixels included in the captured image.

  Here, as an example, the CPU 14 in S102 is described as sequentially specifying all the pixels of the captured image as the target pixel X. However, even if the following processing is performed in a small region including predetermined pixels in the captured image. Good. Note that when changing the position of the pixel of interest X in S102, the CPU 14 sequentially designates the pixel of interest X from left to right one line at a time starting from the upper left corner of the image.

  Step S103: The CPU 14 sets a predetermined local region in the captured image with reference to the position of the target pixel X (see FIG. 3). Although the size and shape of the local area can be set arbitrarily, as an example, the CPU 14 in S103 sets a rectangular area (for example, a 9 × 9 pixel area) centered on the target pixel in the captured image as the local area. To do.

  Step S104: The CPU 14 obtains an image gradient vector at the position of the target pixel X using the gradient information of the pixel value of each pixel included in the local region. Here, as an example, the CPU 14 in S104 performs calculation by paying attention to the luminance of the captured image when obtaining the gradient of the pixel value. If the input captured image data is in the YCbCr format, the CPU 14 only needs to obtain the gradient of the pixel value of the Y component. However, if the captured image data is in the RGB format, the CPU 14 determines any pixel value of RGB. The gradient of the pixel value of the Y component may be obtained after color space conversion from RGB to YCbCr.

Next, the CPU 14 in S104 performs a calculation process for obtaining an image gradient vector in the following manner. For each pixel x _i in the local region (S103), the gradient of the pixel value is

Can be obtained by solving the eigenvalue problem. However, “C” in the above equation (2) indicates a matrix defined by the following equation (3) using the pixel value f (x) at each pixel x _i in the local region.

Note that “Σ” in Equation (3) above represents the sum of all the pixels included in the local region.

  Further, since the matrix C is a 2 × 2 matrix, the above equation (2) has two eigenvalues and two different unit eigenvectors.

  Step S105: The CPU 14 determines whether or not the current pixel of interest X is at the last position (pixel at the lower right corner of the image). If the above requirement is satisfied (YES side), the process proceeds to S106. On the other hand, if the above requirement is not satisfied (NO side), the CPU 14 returns to S102 and repeats the above operation. Note that the information on the image gradient vector obtained at each position of the captured image is stored in the memory 15 by the NO-side loop in S105.

  Step S106: The CPU 14 obtains the edge strength at each position of the captured image by using one of the following methods (a) and (b) (or a combination of both) using the information of the image gradient vector.

(A) The CPU 14 obtains the edge strength at each position of the captured image from the eigenvalue λ _l (obtained in S104) indicating the strength of the image gradient vector at the position of each target pixel.

Here, since the eigenvalue λ _l is an amount obtained by squaring the directivity of the edge gradient of the image, it is preferable to use the double root of the eigenvalue λ _l as the edge strength. Since the eigenvalue λ _l is a value proportional to the number of pixels in the local area, it is preferable to average the number of pixels in the local area. Therefore, the CPU 14 calculates the edge strength E for each pixel of interest X according to the following equation (4).

Note that “N” in Expression (4) indicates the number of pixels in the local region. In addition, as the intensity of the image gradient vector increases, the intensity of the edge of the image at the position of the target pixel increases. Therefore, it can be seen that the contour can be extracted from the image by using the value of the edge strength.

  (B) The CPU 14 obtains the strength of the edge at each position of the captured image based on the correlation of the direction of the image gradient vector between adjacent target pixels.

  Since the images are correlated as described above, the image gradient vectors of the target pixel and the surrounding pixels are high in the contour portion of the image, and the images are easily oriented in the same direction. On the other hand, since the above correlation disappears in a portion away from the contour in the image, the direction of the image gradient vector is not uniform compared to the contour portion. Therefore, if the correlation of the direction of the image gradient vector between adjacent target pixels is observed, the edge strength at each position of the image can be obtained.

  Step S107: The CPU 14 displays on the monitor 19 the distribution state of the edge intensity of the image obtained in S106. Thereby, the user can easily confirm the result of the contour extraction of the image from the display on the monitor 19.

  As an example, FIG. 4 shows an example of a captured image to be processed, and FIG. 5 schematically shows a display screen of the edge intensity distribution state of the image. For simplicity, in the display screen of FIG. 5, the subject is represented by a thicker line as the edge strength is higher. FIG. 4 shows an image in which the center person in focus has a clear outline, but the background tree is blurred. In this case, in the edge strength display screen shown in FIG. 5, the strength of the edge of the person is indicated by a high value, while the strength of the edge of the background tree is indicated by a low value. Above, description of the flowchart of FIG. 2 is complete | finished.

  Note that the CPU 14 may perform an area dividing process for dividing the captured image into a plurality of areas using edge strength information. For example, the CPU 14 obtains a contour line of an image by extracting pixels whose edge strength is a certain level or more. Then, the image processing unit 31 divides the captured image by dividing the captured image into a plurality of regions by the contour line. As a result, for example, when designating a range for performing various types of filter processing (for example, a blurring filter) or a trimming range, the user can designate the range in units of the above-described divided regions.

  Hereinafter, the operational effects of the embodiment will be described. In one embodiment, the CPU 14 obtains an image gradient vector in a local region of the captured image, and extracts a contour from the captured image based on the correlation between the strength of the image gradient vector and the direction of the image gradient vector. Since the above image gradient vector is obtained from the gradient of pixel values in the local region using the principal component analysis method, the influence of noise from the gradient component due to non-directional noise on the above image gradient vector is very small. It will be a thing. Therefore, in one embodiment, it is possible to extract the contour of the image with higher accuracy than the method of extracting the contour of the image by directly using the edge amount of the image.

  In one embodiment, it is not necessary to adjust parameters according to the imaging sensitivity of the captured image, and the convenience of the user when the computer 11 extracts the contour from the captured image is greatly improved.

<Description of other embodiments>
FIG. 6 is a block diagram illustrating a configuration example of an electronic camera according to another embodiment. The electronic camera 21 includes a focusing lens 22, a lens driving unit 23, an image sensor 24, a control unit 25, a ROM 26, a buffer memory 27, a monitor 28, a recording I / F 29, and a release button 30. doing. Here, the lens driving unit 23, the image sensor 24, the ROM 26, the buffer memory 27, the monitor 28, the recording I / F 29, and the release button 30 are connected to the control unit 25, respectively.

  The focusing lens 22 is a lens for performing focus adjustment. The lens position of the focusing lens 22 is adjusted in the optical axis direction by the lens driving unit 23.

  The image sensor 24 captures a subject image formed by an imaging optical system including the focusing lens 22 and generates an image signal of the captured image. The image signal output from the image sensor 24 is input to the control unit 25 via an A / D conversion circuit (not shown).

  Here, in the shooting mode of the electronic camera 21, the image sensor 24 captures a still image (main image) for recording in response to a full press operation of the release button 30. Further, the imaging element 24 in the shooting mode continuously captures images for observation (through images) at predetermined intervals even during standby for shooting. Here, the data of the through image acquired in time series is used for moving image display on the monitor 28 and various arithmetic processes by the control unit 25.

  The control unit 25 is a processor that comprehensively controls the operation of the electronic camera 21. The control unit 25 functions as an image processing unit 31 by executing a program stored in the ROM 26.

  The image processing unit 31 performs various types of image processing (color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment, color conversion processing, etc.) on the captured image data. Similarly to the above-described one embodiment, the image processing unit 31 obtains an image gradient vector in each local region of the captured image, and obtains edge strength using information on the image gradient vector.

  The ROM 26 stores a program executed by the control unit 25. An example of the operation in the shooting mode by this program will be described later. The buffer memory 27 temporarily stores image data in a pre-process or post-process of image processing by the image processing unit 31. The buffer memory 27 is composed of SDRAM which is a volatile storage medium. The monitor 28 displays various images according to instructions from the control unit 25.

  The recording I / F 29 is formed with a connector for connecting a nonvolatile storage medium 33. The recording I / F 29 executes data writing / reading with respect to the storage medium 33 connected to the connector. The storage medium 33 includes a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 6, a memory card is illustrated as an example of the storage medium 33.

  The release button 30 receives from the user an instruction input for starting an autofocus (AF) operation before shooting by a half-press operation and an instruction input for starting an imaging operation by a full-press operation.

(Operation example in shooting mode of electronic camera)
Next, an operation example in the shooting mode of the electronic camera 21 according to another embodiment will be described with reference to the flowchart of FIG. The electronic camera 21 in this shooting mode performs region segmentation of the through image using information of the image gradient vector, and performs scene discrimination based on the result of the region segmentation. 7 is started in response to a half-press operation of the release button 30.

  Step S201: The control unit 25 drives the image sensor 24 to capture a through image. The through image data output from the image sensor 24 is input to the control unit 25.

  Step S202: The image processing unit 31 designates the position of the pixel of interest X for which the image gradient vector is obtained from the plurality of pixels included in the through image. Note that the image processing unit 31 in S202 sequentially designates all pixels in the through image as the target pixel X in substantially the same manner as in the first embodiment.

  Step S203: The image processing unit 31 sets a local area of a predetermined size in the through image with reference to the position of the target pixel X. Note that the content of the processing in S203 corresponds to S103 in FIG.

  Step S204: The image processing unit 31 obtains an image gradient vector at the position of the pixel of interest X using the gradient information of the pixel value of each pixel included in the local region. Note that the content of the processing in S204 corresponds to S104 in FIG.

  Step S205: The image processing unit 31 determines whether or not the current target pixel X is the last position in the through image. If the above requirement is satisfied (YES side), the process proceeds to S206. On the other hand, if the above requirement is not satisfied (NO side), the image processing unit 31 returns to S202 and repeats the above operation.

Step S206: The image processing unit 31 obtains the strength of the edge at each position of the through image using the information of the image gradient vector. Similar to the processing in S106 of FIG. 2, the image processing unit 31 in S206 obtains the edge strength from the strength of the image gradient vector (eigenvalue λ _l ), or the unique unit vector between adjacent target pixels. What is necessary is just to obtain | require the intensity | strength of an edge from the inner product of. For this reason, redundant description of the calculation of the edge strength is omitted.

  Step S207: The image processing unit 31 divides the through image into a plurality of regions using the edge strength information. For example, the image processing unit 31 extracts a pixel whose edge strength is equal to or greater than a certain level and obtains an image outline. Then, the image processing unit 31 divides the through image by dividing the through image into a plurality of regions by the outline.

  Step S208: The image processing unit 31 determines whether or not there is a region corresponding to the skin of the person in the through image, based on the color information (for example, hue and brightness) in each divided region (S207). If the above requirement is satisfied (YES side), the process proceeds to S209. On the other hand, if the above requirement is not satisfied (NO side), the process proceeds to S210.

  Step S209: In this case, it can be estimated that the scene has a person in the shooting screen. Therefore, the control unit 25 automatically changes the mode setting at the time of shooting to a mode suitable for portrait shooting (portrait mode or the like). As a result, it is possible to shoot a person under shooting conditions suitable for portrait shooting.

  In the above-described mode suitable for human photography, the image processing unit 31 may adjust image processing parameters. For example, the image processing unit 31 performs color correction that lowers the saturation of the skin color region in order to express the skin white, or color correction that increases the saturation of the skin color region in order to reduce dullness of the skin color. May be performed.

  Then, in response to the user's full press of the release button 30, the control unit 25 drives the imaging device 24 to execute the imaging process for the main image. The data of the main image is recorded in the storage medium 33 through the recording I / F 29 after being subjected to predetermined processing by the image processing unit 31. Thereafter, the control unit 25 ends the operation in the shooting mode.

  Step S210: In this case, it can be inferred that the scene has no person in the shooting screen. Therefore, the control unit 25 automatically changes the mode setting at the time of shooting to a normal shooting mode (or a shooting mode suitable for landscape shooting).

  Then, in response to the user's full press of the release button 30, the control unit 25 drives the imaging device 24 to execute the imaging process for the main image. The data of the main image is recorded in the storage medium 33 through the recording I / F 29 after being subjected to predetermined processing by the image processing unit 31. Above, description of the flowchart of FIG. 7 is complete | finished.

The electronic camera 21 according to another embodiment performs image region division based on image gradient vector information, and switches shooting modes by scene discrimination using color information of each divided region.
Therefore, in the electronic camera 21 according to another embodiment, it is possible to suppress erroneous detection of a contour due to noise when performing image segmentation.

<Supplementary items of the embodiment>
(1) The apparatus for executing the program of the present invention is not limited to the examples of the computer 11 and the electronic camera 21 of the above-described embodiment, and is a general electronic device having an image display output function (camera mobile phone with camera or image Widely applicable to viewers).

  (2) In each of the above-described embodiments, an example has been described in which various arithmetic processes for contour extraction are realized by software using a program. Of course, these processes may be realized by hardware using an ASIC. It doesn't matter.

  (3) In the other embodiments described above, an example has been described in which image segmentation is performed based on image gradient vector information when the electronic camera 21 performs scene discrimination. However, as in the case of the first embodiment, the electronic camera 21 of the other embodiments can of course perform the contour extraction process using the main image as a processing target in a post-processing step after photographing.

  In addition, the area dividing process of the electronic camera 21 in the other embodiments described above can be applied to a case where a portrait image is generated by blurring only the background area by image processing when a person is a main subject, for example. .

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment. A flow chart showing an example of contour extraction processing by an image processing program of one embodiment Illustration of local region and image gradient vector The figure which shows the example of the captured image of a process target The schematic diagram which shows the example of the display screen of the intensity distribution state of the edge of the image corresponding to FIG. The block diagram which shows the structural example of the electronic camera of other embodiment. The flowchart which shows the operation example in the imaging | photography mode of the electronic camera of other embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Computer, 12 ... Data reading part, 13 ... Memory | storage device, 14 ... CPU, 15 ... Memory, 16 ... Input-output I / F, 17 ... Bus, 18 ... Input device, 19 ... Monitor, 21 ... Electronic camera, 22 DESCRIPTION OF SYMBOLS ... Focusing lens, 23 ... Lens drive part, 24 ... Image pick-up element, 25 ... Control part, 26 ... ROM, 27 ... Buffer memory, 28 ... Monitor, 29 ... Recording I / F, 30 ... Release button, 31 ... Image processing part 33 ... Storage medium

Claims (8)

An image reading process for acquiring a captured image;
A local area setting process for designating a target pixel from among pixels included in the captured image and setting a local area of a predetermined size in the captured image with reference to the position of the target pixel;
Using the gradient information of pixel values obtained from a plurality of pixels included in the local region, solving the eigenvalue problem of principal component analysis using the sum of the gradient information in the local region as a variable, the pixel of interest A gradient information calculation process for obtaining an image gradient vector indicating the gradient of the image at the position;
A contour extraction process for extracting a contour from the captured image using information of the image gradient vector obtained from at least two or more pixels of interest;
A program that causes a computer to execute. The program according to claim 1,
In the gradient information calculation process, the pixel x in the local region _i For the slope of the pixel value
However, “C” in Expression (2) is defined by Expression (3) below.
In the program according to claim 1 or 2 ,
In the contour extracting process, a contour is extracted by comparing information indicating the intensity of the image gradient vector. In the program according to claim 1 or 2 ,
In the contour extracting process, a contour is extracted based on the correlation of the direction of the image gradient vector between adjacent pixels of interest. In the program according to any one of claims 1 to 4 ,
A program further comprising an area dividing process for dividing the captured image into a plurality of areas using the information on the outline extracted by the outline extracting process. A camera comprising a computer that executes the program according to any one of claims 1 to 5 .   An acquisition unit for acquiring a captured image;
  A local region setting unit that specifies a target pixel from among pixels included in the captured image, and sets a local region of a predetermined size in the captured image with reference to the position of the target pixel;
  By using the sum of gradient information of pixel values obtained from a plurality of pixels included in the local region, by solving the eigenvalue problem of principal component analysis using the sum of the gradient information in the local region as a variable, A gradient information calculation unit for obtaining an image gradient vector indicating the gradient of the image at the position of the target pixel;
A contour extraction unit that extracts a contour from the captured image using information on the image gradient vector obtained from at least two or more pixels of interest;
  An image processing apparatus comprising: An image reading process for acquiring a captured image;
A local region setting step of designating a target pixel from among pixels included in the captured image and setting a local region of a predetermined size in the captured image with reference to the position of the target pixel;
Using the gradient information of pixel values obtained from a plurality of pixels included in the local region, solving the eigenvalue problem of principal component analysis using the sum of the gradient information in the local region as a variable, the pixel of interest A gradient information calculation step for obtaining an image gradient vector indicating the gradient of the image at the position;
A contour extraction step of extracting a contour from the captured image using information on the image gradient vector obtained from at least two or more pixels of interest;
A method for extracting an outline of an image, comprising: