JP2011253354A - Image processing apparatus, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy of recognizing category of object in an image.SOLUTION: A distance detecting section 32 acquires information on a distance of an object from an imaging device at capturing an image, which is acquired by a image acquisition section 31, per pixel unit in the image. A region dividing section 41 included in an object recognition section 36 divides the image into a foreground region which is an object and a background region other than that in the image based on the image and the distance information. A background-effect reducing feature amount extraction section 42 extracts a feature amount, which reduces an effect on category recognition of the object by the background, based on the information either one or all of the image, the distance information and the divided background regions. A recognition processing section 43 recognizes a category of the foreground of the object in the image based on the extracted feature amount. The invention is applicable to an image processing apparatus.

Description

  The present invention relates to an image processing device, method, and program, and more particularly, to an image processing device, method, and program that improve the recognition accuracy of the type of an object that is a subject in an image.

  Conventionally, as an object recognition technique, there is a technique for identifying the shape of a hand such as goo, choki, or par with a recognition target as a human palm.

  In addition, in order to improve the object recognition accuracy of these object recognition technologies, there are technologies that use specific conditions and additional information in combination with images, and as an example, the hand area can be reduced by using skin color information. A technique has been proposed that facilitates acquisition and improves the accuracy of hand shape recognition (see Patent Document 1).

JP 2007-148663 A

  However, in the technique of Patent Document 1, since the skin color information is used, the accuracy of the recognition process is easily affected by lighting or the like.

  The present invention has been made in view of such circumstances, and in particular, makes it possible to recognize with high accuracy the type of an object that is a subject in an image even for images taken under various environments. Is.

  An image processing apparatus according to an aspect of the present invention includes: an image acquisition unit configured to acquire an image; and information about a distance of an object captured in the image from an imaging device when the image is captured. A distance acquisition means for acquiring in units, an area dividing means for dividing an area into a foreground area as a subject in the image and a background area other than the image based on the image and the distance information; the image; A background influence reducing feature quantity that extracts a feature quantity that reduces the influence of the background on the recognition of the type of subject based on distance information and / or all of the background area information divided by the area dividing means Extraction means, and recognition means for recognizing the type of the subject as the foreground in the image based on the feature amount extracted by the background effect reduction feature amount extraction means.

  The background influence reducing feature amount extraction means includes a smoothing means for smoothing pixels corresponding to the background area in the image based on the image and information on the background area divided by the area dividing means. A feature amount that reduces the influence of the background on the recognition of the type of the subject is extracted from the image in which the pixels corresponding to the background region are smoothed by the smoothing unit. can do.

  The smoothing means smoothes the pixels with the intensity corresponding to the ratio of the background pixels and foreground pixels of the neighboring pixels to the pixels in the background area divided by the area dividing means. Can do.

  The background effect reduction feature quantity extraction means sets the weight based on the feature quantity extraction means for extracting the feature quantity of the image, the image, distance information, and information on the background area divided by the area division means. A weight setting unit for adding the weight to the feature amount, and the feature amount to which the weight set by the weight setting unit is attached can be used for recognition of a subject type by the background. It can be made to extract as a feature-value which reduces influence.

  The feature amount extraction means can extract HOG feature amounts.

  An image processing method according to an aspect of the present invention includes: an image acquisition unit configured to acquire an image; and information on a distance of an object captured in the image from an imaging device at the time of imaging the image. A distance acquisition means for acquiring in units, an area dividing means for dividing an area into a foreground area as a subject in the image and a background area other than the image based on the image and the distance information; the image; A background influence reducing feature quantity that extracts a feature quantity that reduces the influence of the background on the recognition of the type of subject based on distance information and / or all of the background area information divided by the area dividing means An image including extraction means and recognition means for recognizing the type of the subject as the foreground in the image based on the feature amount extracted by the background effect reduction feature amount extraction means An image processing method for a physical apparatus, wherein the image acquisition unit acquires the image, and the distance acquisition unit captures an image of the subject captured in the image at the time of capturing the image. A distance acquisition step of acquiring information on the distance from the device in units of pixels of the image, the image in the region dividing means, and a foreground region that is a subject in the image based on the distance information; A region dividing step of dividing the region into background regions, and all of the image, the distance information, and the information on the background region divided by the region dividing step in the background influence reducing feature amount extraction unit, or Based on either of them, a background influence reducing feature quantity that extracts a feature quantity that reduces the influence of the background on recognition of the type of subject is extracted. And step out, in the recognition unit, on the basis of the feature value extracted by the processing of the background impact reduction feature extraction step, and a recognition step of recognizing the type of the subject to be the foreground in the image.

  According to another aspect of the present invention, there is provided a program for acquiring, in pixel units of the image, image acquisition means for acquiring an image, and information on the distance of the subject imaged in the image from the imaging device at the time of imaging the image. A distance acquisition means for acquiring, an area dividing means for dividing the area into a foreground area as a subject in the image and a background area other than the image based on the image and the distance information; the image; and the distance information. And background influence reducing feature amount extraction means for extracting a feature amount that reduces the influence of the background on recognition of the type of subject based on all or one of the information of the background region divided by the region dividing means. And an image processing unit for recognizing a type of a subject as a foreground in the image based on the feature amount extracted by the background effect reduction feature amount extraction unit. An image acquisition step for acquiring the image in the image acquisition means, and a subject captured in the image in the distance acquisition means from an imaging device at the time of imaging the image to a computer that controls the apparatus A distance acquisition step of acquiring distance information in units of pixels of the image, the image in the region dividing means, and a foreground region as a subject in the image based on the distance information, and other background regions And / or all of the information of the image, the distance information, and the background region divided by the processing of the region dividing step in the background effect reducing feature amount extraction unit. Based on the above, a background influence reducing feature quantity for extracting a feature quantity for reducing the influence of the background on the recognition of the type of subject And a recognition step of recognizing the type of the subject as the foreground in the image based on the feature amount extracted by the processing of the background effect reduction feature amount extraction step in the recognition unit. .

  In one aspect of the present invention, an image is acquired, and distance information of the subject captured in the image from the imaging device at the time of capturing the image is acquired in pixel units of the image. And based on the distance information, an area is divided into a foreground area as a subject in the image and a background area other than that, and all of the image, the distance information, and information of the divided background area, or Based on either of them, a feature amount that reduces the influence of the background on the recognition of the subject type is extracted, and the type of the subject that is the foreground in the image is recognized based on the extracted feature amount.

  The image processing apparatus of the present invention may be an independent apparatus or a block that performs image processing.

  According to one aspect of the present invention, it is possible to improve the identification accuracy of the type of an object that is a subject included in images captured under various environments.

It is a block diagram which shows the structural example of one Embodiment of the image processing apparatus to which this invention is applied. It is a flowchart explaining the object recognition process by the image processing apparatus of FIG. It is a figure explaining the object recognition process by the image processing apparatus of FIG. 3 is a flowchart for explaining background effect reduction feature amount extraction processing by the image processing apparatus of FIG. 1; 3 is a flowchart for explaining image processing by the image processing apparatus of FIG. 1. It is a flowchart explaining the feature-value extraction process by the image processing apparatus of FIG. It is a figure explaining a feature-value extraction process. It is a block diagram which shows the structural example of other embodiment of the image processing apparatus to which this invention is applied. It is a flowchart explaining the background influence reduction feature-value extraction process by the image processing apparatus of FIG. FIG. 20 is a block diagram illustrating a configuration example of still another embodiment of an image processing device to which the present invention has been applied. It is a flowchart explaining the feature-value extraction process by the image processing apparatus of FIG. And FIG. 11 is a diagram illustrating a configuration example of a general-purpose personal computer.

Hereinafter, the best mode for carrying out the invention (hereinafter referred to as an embodiment) will be described. The description will be given in the following order.
1. First embodiment (example in which reduction in recognition accuracy is reduced by image processing processing and feature amount extraction processing)
2. Second embodiment (example in which a reduction in recognition accuracy is reduced only by feature amount extraction processing)
3. Third embodiment (example in which a reduction in recognition accuracy is reduced only by image processing)

<1. First Embodiment>
[Configuration example of image processing apparatus]
FIG. 1 shows a configuration example of an embodiment of hardware of an image processing apparatus to which the present invention is applied. The image processing apparatus 11 in FIG. 1 recognizes the type of an object that is a subject included in images captured under various environments.

  The image processing apparatus 11 includes an image acquisition unit 31, a distance detection unit 32, a target position detection unit 33, enlargement / reduction units 34 and 35, and an object recognition unit 36.

  The image acquisition unit 31 captures a plurality of images like a stereo image in real time by an imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and detects the distance detection unit 32 and target position detection. To the unit 33. The image acquisition unit 31 may acquire at least one image captured in advance, instead of the stereo image captured in real time. In this case, the acquired image is used as the target position detection unit 33. Supply only to.

  When a plurality of images such as a stereo image are captured in real time by the image acquisition unit 31, the distance detection unit 32 obtains parallax at corresponding points by a corresponding point search process between the captured images. Then, the distance detection unit 32 obtains the distance from the image acquisition unit 31 to the subject in units of pixels of the subject included in the image captured by the parallax at the obtained corresponding point, and the target position detection unit 33 as distance information. And to the enlargement / reduction unit 35. In addition, the distance detection unit 32 cannot acquire a stereo image or the like in real time from the image acquisition unit 31, and is different from a method using an image when a pre-captured image is supplied, for example, TOF (Time of Flight). ) Method to detect the distance to the subject at the time of imaging, and supplies it to the target position detection unit 33 and the enlargement / reduction unit 35 as distance information. Furthermore, the distance detection unit 32 obtains distance information from the imaging position to the subject in the image detected by a distance measuring sensor or the like at the timing when the image acquired by the image acquisition unit 31 is captured, for example. And supplied to the target position detection unit 33 and the enlargement / reduction unit 35.

  The target position detection unit 33 detects a target position existing in the image based on the image supplied from the image acquisition unit 31 and the distance information supplied from the distance detection unit 32, and information on the target position The necessary area in the image is cut out based on the above and supplied to the enlargement / reduction unit 34. Here, the target is an object as a subject to be recognized, and the target position is a coordinate indicating the position of the subject as a target in the three-dimensional coordinate space. For example, when the image acquisition unit 31 is an image sensor, the target position detection unit 33 sequentially selects a region (or a point) closest to the image sensor or the distance measuring sensor, or a region that satisfies a predetermined distance condition from the Monte Carlo. The target position is obtained by estimation using an arbitrary probability model such as a method. The target position detection unit 33 obtains a required area from the image acquired by the image acquisition unit 31, and cuts out the calculated area. The purpose is to specify a processing area. Therefore, it is not always necessary. In particular, when it is assumed that the area of the subject of the acquired image is larger than the predetermined area, the processing is not necessarily required, so that the target position detection unit 33 is omitted as a configuration. It may be.

  The enlargement / reduction units 34 and 35 supply the image and distance information to the object recognition unit 36 with the size enlarged or reduced according to the necessity of processing. More specifically, the enlargement / reduction units 34 and 35 enlarge or reduce the image and distance information by the nearest neighbor method, the bilinear method, the bicubic method, the Lanczos filter, or the like. Note that the enlargement or reduction of the size is a process performed as necessary, and therefore the enlargement / reduction units 34 and 35 may output the same size without enlargement or reduction.

  Based on the image supplied from the enlargement / reduction unit 34 and the distance information supplied from the enlargement / reduction unit 35, the object recognition unit 36 identifies and recognizes the type of the object that is the subject in the image, Output as recognition result. More specifically, the object recognition unit 36 includes a region division unit 41, a background effect reduction feature amount extraction unit 42, and a recognition processing unit 43.

  Based on the distance information supplied from the enlargement / reduction unit 35, the region division unit 41 divides the image supplied from the enlargement / reduction unit 34 into a foreground region and a background region, and reduces background influence as region division information. This is supplied to the feature quantity extraction unit 42. More specifically, for example, the region dividing unit 41 obtains the median value of the distances of all the pixels included in the distance information, sets the pixels that are closer than the median as the foreground region, and uses the pixels that are far away as the background. This is an area. However, the foreground area and the background area may be divided by other methods. For example, pixels closer than a predetermined distance may be used as the foreground area, pixels farther than this may be used as the background area, and image segmentation or the like may be used. In combination with the extraction of an arbitrary object and the distance, for example, a pixel having a distance shorter than the distance of a predetermined object may be set as the foreground area, and the other may be set as the background area. The area division information is flag information indicating whether each pixel belongs to the divided foreground area or background area. Accordingly, when the entire area of the image is divided into two areas, the foreground area and the background area, the area division information is information including a flag indicating the foreground area and a flag indicating the background area. Further, the area dividing unit 41 may divide the area into more areas according to the distance in addition to the foreground area and the background area. In this case, the area division information indicates an area to be divided according to the distance. This is information composed of flags.

  Based on the image supplied from the enlargement / reduction unit 34, the distance information supplied from the enlargement / reduction unit 35, and the region division information supplied from the region division unit 41, the background influence reduction feature amount extraction unit 42 A feature amount for reducing the reduction in recognition of the type of the object that is the foreground region due to the influence is extracted and output to the recognition processing unit 43. More specifically, the background influence reducing feature amount extraction unit 42 includes an image processing unit 51 and a feature amount extraction unit 52.

  Based on the region division information and the distance information, the image processing unit 51 performs a smoothing process on the pixels belonging to the background region in the image. More specifically, the image processing unit 51 includes a ratio calculation unit 61, a ratio intensity calculation unit 62, a distance intensity calculation unit 63, and a smoothing unit 64. The ratio calculation unit 61 sets a local area in association with each pixel belonging to the background area, and obtains the ratio of the pixels belonging to the background area of the pixels belonging to the local area as the ratio. The ratio strength calculation unit 62 calculates the smoothing strength in the smoothing unit 64 as the ratio strength based on the ratio obtained by the ratio calculation unit 61. The distance strength calculation unit 63 calculates the smoothing strength in the smoothing unit 64 as the distance strength based on the distance information. The smoothing unit 64 smoothes pixels belonging to the background region with an intensity corresponding to the ratio intensity and the distance intensity.

  The feature amount extraction unit 52 uses a HOG (Histogram of Oriented Gradients) feature amount as a feature amount necessary for recognition processing from an image obtained by smoothing the background region by the image processing unit 51 based on the region division information and the distance information. To extract. More specifically, the feature amount extraction unit 52 includes a gradient calculation unit 71, a ratio calculation unit 72, a ratio weight calculation unit 73, a distance weight calculation unit 74, a histogram generation unit 75, and a normalization unit 76. The gradient calculation unit 71 obtains the magnitude and direction of the gradient for each pixel in the image supplied from the image processing unit 51. Similar to the ratio calculation unit 61, the ratio calculation unit 72 sets a local area in association with each pixel, and obtains the ratio of the pixels belonging to the background area of the pixels belonging to the local area as the ratio. The ratio weight calculation unit 73 sets a ratio weight to be added to the magnitude of the gradient of each pixel based on the ratio calculated by the ratio calculation unit 72. The distance weight calculation unit 74 sets a distance weight to be added to the gradient of each pixel according to the distance of each pixel based on the distance information. The histogram generating unit 75 obtains the frequency by adding the ratio weight and the distance weight to the magnitude of the gradient for each gradient direction in a cell unit composed of a plurality of pixels, and generates a histogram based on the obtained frequency. The normalizing unit 76 normalizes the generated histogram for each block composed of a plurality of cells, and supplies this to the recognition processing unit 43 as a HOG feature amount. In this example, an example of extracting a HOG feature value as a feature value is described. However, the feature value is not limited to this as long as the feature value can be used for object recognition. For example, a Haar-like feature value is used. SIFT feature value, Gabor feature value, or the like may be used.

  The recognition processing unit 43 performs recognition processing for identifying the type of an object by using, for example, SVM (Support Vector Machine) or Boosting based on the background effect reduction feature amount supplied from the background effect reduction feature amount extraction unit 42. Execute and output the recognition result.

[Object recognition processing]
Next, the object recognition process will be described with reference to the flowchart of FIG.

  In step S <b> 11, the image acquisition unit 31 acquires an image including an object that is a subject and supplies the acquired image to the distance detection unit 32 and the target position detection unit 33.

  In step S <b> 12, the distance detection unit 32 detects pixel-unit distance information corresponding to each of the objects as subjects included in the image acquired by the image acquisition unit 31, the target position detection unit 33, and the enlargement / reduction processing To the unit 35. More specifically, for example, when the image acquisition unit 31 includes an image sensor and captures the same subject from a plurality of angles such as a stereo image in real time, the distance detection unit 32 captures the plurality of images. By the corresponding point search process, a distance is obtained for each pixel using the parallax and detected as distance information.

  In step S13, the target position detection unit 33 detects a target position existing in the image based on the image supplied from the image acquisition unit 31 and the distance information supplied from the distance detection unit 32, In step S <b> 14, necessary areas in the image are cut out and adjusted based on the target position information, and supplied to the enlargement / reduction unit 34.

  In step S <b> 15, the enlargement / reduction units 34 and 35 enlarge or reduce the image adjusted by the target position detection unit 33 based on the target position and the distance information to a size suitable for processing, and then send the image to the object recognition unit 36. Supply. Note that the images supplied from the enlargement / reduction units 34 and 35 whose size is enlarged or reduced and the distance information are only adjusted in size for an area suitable for processing, and so on. Are simply referred to as images and distance information.

  In step S16, the region dividing unit 41 in the object recognition unit 36 divides the image into a foreground region and a background region in units of pixels based on the distance information. More specifically, for example, the area dividing unit 41 classifies each pixel as a foreground area based on distance information, and classifies other pixels as background areas by classifying other pixels as background areas. Divide into foreground and background areas. That is, for example, when an image as shown by the image A1 in FIG. 3 is acquired and the distance information A2 in FIG. 3 is detected, the area division information A3 in FIG. 3 is obtained. Here, the image A1 is an image in which an object Z1 that is a subject made up of a palm exists on the background Z2. Further, the distance information A2 is drawn by a color scheme Z12 so that the entire area where the palm object Z1 exists is drawn with a color scheme Z11 indicating a predetermined distance, and becomes black as the distance of each pixel in the space in the background Z2 increases. Has been. Further, in the area division information A3, a palm slightly closer than a predetermined distance behind the area where the palm object Z1 is present is displayed as a white foreground area Z21, and the subsequent area is indicated as a background area Z22. Shown in gray. As a result, the area corresponding to the palm object Z1 is divided as the foreground area Z21, and the other area is divided into the background area Z22.

  In step S <b> 17, the background effect reduction feature value extraction unit 42 executes background effect reduction feature value extraction processing, and extracts a background effect reduction feature value from the image based on the distance information and the region division information.

[Background effect reduction feature extraction processing]
Here, the background influence reduction feature amount extraction processing will be described with reference to the flowchart of FIG. The background effect reduction feature value extraction process includes an image processing process executed in step S31 in the image processing unit 51 and a feature value extraction process executed in step S32 in the feature value extraction unit 52.

[Image processing]
Here, the image processing will be described with reference to the flowchart of FIG.

  In step S51, the image processing unit 51 initializes the counter y of the vertical coordinate constituting the image, that is, the y coordinate, to zero.

  In step S52, the image processing unit 51 initializes a horizontal coordinate constituting the image, that is, an x coordinate counter x to zero.

  In step S53, the image processing unit 51 determines whether or not the pixel I (x, y) that is the processing target on the image is a background region based on the distance information. For example, the image processing unit 51 uses a flag indicating the background region. If there is, the process proceeds to step S54.

  In step S54, the image processing unit 51 controls the ratio calculation unit 61 to set a local region corresponding to the pixel I (x, y). That is, the ratio control unit 61 sets, for example, a 6 pixel × 6 pixel square area centered on the pixel I (x, y) to be processed as a local area.

  In step S55, the ratio calculator 61 calculates a ratio q between the number of pixels belonging to the foreground area and the number of pixels belonging to the background area of each pixel belonging to the local area.

  In step S56, the image processing unit 51 controls the ratio intensity calculation unit 62 to calculate the ratio intensity St_q based on the smoothing ratio q of the pixel I (x, y) to be processed. That is, for example, when the ratio q indicates that the ratio of the foreground region is high, there are few pixels in the background region that affect the object recognition processing around the pixel I (x, y) to be processed. Therefore, the ratio strength calculation unit 62 calculates the ratio strength St_q to be weak. On the other hand, when the ratio q indicates that the ratio of the background region is high, it is considered that there are many pixels in the background region that affect the object recognition processing around the pixel I (x, y) to be processed. Therefore, the ratio strength calculation unit 62 performs calculation so as to increase the ratio strength St_q.

  In step S57, the image processing unit 51 controls the distance intensity calculation unit 63 to calculate the distance intensity kk based on the distance of the smoothed distance information of the pixel I (x, y) to be processed. That is, as the distance of the pixel I (x, y) to be processed increases, the distance region is considered to be the background region. Therefore, the distance intensity calculation unit 63 calculates the distance intensity St_d to be stronger, and conversely, the distance is As the distance is closer, the distance strength St_d is weakened and smoothed.

  In step S58, the image processing unit 51 controls the smoothing unit 64 to smooth the pixel I (x, y) to be processed with an intensity corresponding to the ratio intensity St_q and the distance intensity St_d. As described above, since the smoothing strength depends on the ratio q and the distance, for example, the smoothing unit 64 performs the smoothing based on the multiplication result St_q × St_d of the ratio strength St_q and the distance strength St_d. The strength may be set.

  On the other hand, if it is determined in step S53 that the pixel I (x, y) to be processed is not a background region flag, the processing in steps S54 to S58 is skipped.

  For example, when the pixel I (x, y) to be processed is the pixel B1 in the divided region information A4 in FIG. 3, the pixel B1 belongs to the foreground region, and therefore, in step S53, the pixel to be processed is the background region. It is determined that the flag is not, and the processing of steps S54 to S58 is skipped.

  On the other hand, for example, when the pixel I (x, y) to be processed is the pixel B2 or B3 in the divided region information A4 in FIG. 3, the pixels B2 and B3 belong to the background region, and thus in the local regions Z102 and Z103. Each is smoothed based on the proportion of the background area. That is, all of the local area Z103 is a background area, but only a part of the local area Z102 is a background area. For this reason, the pixel B3 is smoothed with a stronger intensity than the pixel B2.

  In step S59, the image processing unit 51 increments the horizontal coordinate constituting the image, that is, the x-coordinate counter x by one.

  In step S60, the image processing unit 51 determines whether or not the counter x is smaller than the horizontal size Sx of the image. If it is determined that the counter x is smaller, the process returns to step S53. That is, the processes in steps S53 to S60 are repeated until the counter x becomes smaller than the horizontal size Sx. If it is determined in step S60 that the counter x is smaller than the horizontal size Sx, the process proceeds to step S61.

  In step S61, the image processing unit 51 increments a horizontal coordinate constituting the image, that is, an x-coordinate counter x by one.

  In step S62, the image processing unit 51 determines whether or not the counter y is smaller than the vertical size Sy of the image. If it is determined that the counter y is smaller, the process returns to step S52. That is, the processes in steps S52 to S62 are repeated until the counter y is smaller than the vertical size Sy. If it is determined in step S62 that the counter y is smaller than the vertical size Sy, the process ends.

  That is, on the basis of the area division information, the pixels corresponding to the background area are smoothed with the intensity corresponding to the distance between the background area and the foreground area in the local area and the distance information. As a result, the influence of the pixels in the background area that affects the recognition of the object type is reduced, and in the subsequent stage, the feature amount is extracted from the image in which the influence of the pixels in the background area that affects the recognition process is reduced. As a result, the recognition rate in the object type recognition process can be improved. Note that the smoothing intensity can be controlled only by the ratio or by the distance alone, so that the effect of suppressing the reduction of the recognition accuracy of the recognition process by the background region can be achieved. You may make it control the intensity | strength of smoothing only by either.

[Feature extraction processing]
Next, the feature amount extraction processing will be described with reference to the flowchart of FIG.

  In step S <b> 71, the feature amount extraction unit 52 initializes the counter y of the vertical coordinate constituting the image, that is, the y coordinate, to 0.

  In step S72, the feature amount extraction unit 52 initializes a horizontal coordinate constituting the image, that is, an x coordinate counter x to zero.

  In step S73, the feature amount extraction unit 52 controls the gradient calculation unit 71 to calculate the angle and magnitude of the gradient for the pixel I (x, y) to be processed. More specifically, for example, when the processing target pixel I (x, y) is the pixel C1 in FIG. 7, the gradient calculating unit 71 sets the pixels I (x, y + 1) and I ( The gradient is obtained by calculating the following equation (1) using x, y−1) and the pixels I (x + 1, y) and I (x−1, y) adjacent in the horizontal direction 132. In the upper left part of FIG. 7, the gradient arrangement of the pixel I (x, y) to be processed is shown. In the upper center part of FIG. 7, the pixel 121 that is the processing target pixel C <b> 1 is included. A configuration example of the cell 102 composed of m pixels × n pixels is shown. Further, in the upper right part of FIG. 7, an example of the configuration of the block 101 having M × N cells 111 corresponding to the cell 102 is shown, and in the lower part, the gradient of each pixel constituting the cell 102 is shown. An example of the histogram 103 including the frequencies for the directions L1 to LL is shown.

  Here, gx (x, y) and gy (x, y) are a horizontal gradient and a vertical gradient, respectively.

  And the gradient calculation part 71 calculates the magnitude | size and angle of a gradient, respectively by calculating the following formula | equation (2) and Formula (3).

  Here, r (x, y) is the magnitude of the gradient, and θ (x, y) indicates the angle of the gradient.

  In step S74, the feature amount extraction unit 52 controls the ratio calculation unit 72 to set a local region corresponding to the pixel I (x, y) that is the processing target on the image. That is, by the processing in step S74, the ratio calculation unit 72 sets, for example, a square region of neighboring u pixels × v pixels centered on the pixel I (x, y) to be processed as a local region.

  In step S75, the ratio calculation unit 72 calculates a ratio q between the number of pixels belonging to the foreground area and the number of pixels belonging to the background area of each pixel belonging to the local area.

  In step S76, the feature amount extraction unit 52 controls the ratio weight calculation unit 73 to calculate the ratio weight wk based on the ratio q.

  In step S77, the feature amount extraction unit 52 controls the distance weight calculation unit 74 to calculate the distance weight wd based on the distance of the pixel I (x, y) to be processed in the distance information.

  In step S78, the feature quantity extraction unit 52 increments the horizontal coordinate constituting the image, that is, the x-coordinate counter x by one.

  In step S79, the feature amount extraction unit 52 determines whether or not the counter x is smaller than the horizontal size Sx of the image. If it is determined that the counter x is smaller, the process returns to step S73. That is, the processes in steps S73 to S79 are repeated until the counter x becomes smaller than the horizontal size Sx. If it is determined in step S79 that the counter x is smaller than the horizontal size Sx, the process proceeds to step S80.

  In step S80, the feature amount extraction unit 52 increments the horizontal coordinate constituting the image, that is, the x-coordinate counter x by one.

  In step S81, the feature quantity extraction unit 52 determines whether or not the counter y is smaller than the size Sy in the vertical direction of the image. If it is determined that the counter y is smaller, the process returns to step S72. That is, the processes in steps S72 to S81 are repeated until the counter y is smaller than the vertical size Sy. If it is determined in step S81 that the counter y is smaller than the vertical size Sy, the process proceeds to step S82.

  In step S82, the histogram generation unit 75 sets an unprocessed cell as a cell to be processed. That is, as shown in the upper center portion of FIG. 7, a cell composed of a plurality of pixel groups is set in the image, and an unprocessed cell is set as a processing target cell. In the upper center of FIG. 7, an example in which a cell 102 of m pixels × n pixels is set is shown.

  In step S83, the histogram generation unit 75 sets the frequency including the ratio weight wk and distance weight wd for each gradient direction in the cell to be processed. That is, the HOG feature amount is based on a histogram obtained from the frequency belonging to the gradient direction of each pixel by dividing the gradient direction into directions L1 to LL for each pixel of the cell 102 shown in the upper center of FIG.

  That is, the histogram generation unit 75 sets the frequency fl ′ in the gradient direction L1 of each pixel as represented by the following expression (4).

  Here, fl ′ is the frequency in the direction L1, and wk is a ratio weight based on the ratio between the foreground area and the background area in the local area of each pixel obtained in the process of step S76. Further, wd is a distance weight set according to the distance of the distance information of each pixel obtained in step S77, and rl (x, y) is the magnitude of the gradient.

  That is, the ratio weight wk is set based on the ratio b between the foreground area and the background area set by the following equation (5).

  Here, δ (D) is a generalized function of the region division information D (x, y) set by the following equation (6). U and v indicate that the local area is an area of u pixels × v pixels.

  In other words, the function δ (D) is a function that is 1 in the foreground area and 0 in the background area.

  For example, when the ratio weight wk is set in k stages, the ratio weight calculator 73 sets the frequency including the weight as shown in the following formula (7) based on these conditions, for example.

  Here, th1, th2,..., Thk are thresholds for setting k-stage weights, and the ratio weight wk and the threshold thk have a relationship represented by the following expression (8).

  That is, as the ratio b increases, the ratio weight wk also increases. That is, as the ratio of the foreground area increases, the ratio weight wk is set larger. Conversely, as the ratio of the foreground area decreases, the ratio weight wk increases. The weight wk is set small. For this reason, the ratio weight wk is set so that the frequency of the histogram by the pixels in the background area that affects the object identification is reduced, so that the influence of the pixels in the background area is eliminated from the feature amount and the recognition accuracy is improved. It becomes possible to make it.

  Similarly, the distance weight wd is set to be smaller as the distance is larger for each pixel, and conversely, is set to be larger as the distance is smaller. For this reason, the distance weight wd is set so that the frequency of the histogram of the pixels in the background area that affects the object identification is reduced, so that the influence of the pixels in the background area is eliminated from the feature amount, and the recognition accuracy is reduced. Can be improved.

  In step S84, the histogram generation unit 75 generates a histogram for each cell to be processed based on the frequency including the ratio weight wk and the distance weight wd thus determined. That is, the histogram is a sum of the frequency for each gradient direction classified into a predetermined number of gradient directions of each pixel, and is represented by the following equation (9), for example.

  Here, F is a histogram of each cell, and fL1, fL2,... FLL indicates the frequency for each gradient direction LL.

  In step S85, the histogram generation unit 75 determines whether there is an unprocessed cell. If there is an unprocessed cell, the process returns to step S82. That is, the processes in steps S82 to S85 are repeated until histograms are generated for all cells. In step S85, histograms are obtained for all the cells, and if it is determined that there are no unprocessed cells, the process proceeds to step S86.

  In step S86, the normalization unit 76 sets an unprocessed block as a processing target block. That is, the block is an area composed of a plurality of cells. For example, the block is an area of M × N cells as shown in the upper right part of FIG.

  In step S87, the normalizing unit 76 normalizes the obtained histogram in units of cells, in units of blocks to be processed set around each cell, and uses the normalized histogram in units of blocks to generate HOG feature values. Is output to the recognition processing unit 43. That is, the histogram V in block units set in cell units is expressed by the following equation (10).

  The normalization unit 76 normalizes the histogram V in the block set in cell units for all cells by calculating the following equation (11). As a result, the normalization result in units of blocks set for each cell is obtained as the HOG feature amount.

  Here, the V bar indicates a normalized histogram V in block units. Further, ε is a constant for preventing the histogram V in block units from becoming divergent because it becomes zero.

  In step S88, the normalization unit 76 determines whether there is an unprocessed block. If there is an unprocessed block, the process returns to step S86. That is, the processes in steps S86 to S88 are repeated until the normalization process is performed on all blocks. If it is determined in step S88 that the normalization process for all the blocks has been completed and there are no unprocessed blocks, the process ends.

  That is, when only two types of foreground areas and background areas are set in the image, the ratio weight wk is a process when k = 2. Therefore, even when an area other than the foreground area and the background area is set based on the distance information and k> 2, the ratio weight wk is increased as the ratio of the areas close to each other is high, and the pixels with the short distance are set. By increasing the distance weight wd, it is possible to eliminate pixels that are far away from each other, that is, elements that are considered to belong to the background area and that affect the recognition process. It becomes possible to identify and recognize.

  Here, it returns to the flowchart of FIG.

  In step S18, the recognition processing unit 43 identifies the type of the object included in the image by, for example, SVM (Support Vector Machine) or Boosting based on the background effect reduction feature amount supplied by the above-described processing. Recognize and output the recognition result.

  With the above processing, the pixels belonging to the background area in the image are smoothed with an intensity according to the ratio of the foreground area and the background area in the local area, and for each pixel, the foreground area and the background area in the local area The feature amount is extracted by adding the ratio weight set according to the ratio and the distance weight set based on the distance information of each pixel. As a result, in the object recognition process, it is possible to reduce the influence of the background region that reduces the recognition accuracy, and it is possible to improve the object recognition accuracy.

<2. Second Embodiment>
[Other image processing apparatus configuration examples]
In the above, an example has been described in which both the image processing process and the feature amount extraction process reduce the influence of the pixels in the background region, which causes a reduction in object recognition accuracy. Even if there is, it is effective. Therefore, even if the image processing unit 51 is deleted as in the image processing apparatus 11 shown in FIG. 8, it is possible to reduce the influence of the pixels in the background area by the feature amount extraction processing. In this case, the background effect reduction feature quantity extraction process is only the feature quantity extraction process shown in step S101 as shown in FIG. Note that the feature amount extraction processing is the same as the processing described with reference to the flowchart of FIG.

<3. Third Embodiment>
[Further configuration example of other image processing apparatus]
Further, as in the image processing apparatus 11 shown in FIG. 10, it is possible to reduce the influence of the pixels in the background area that cause the object recognition accuracy to be lowered only by the image processing. That is, in the image processing apparatus 11 in FIG. 10, the ratio calculation unit 72, the ratio weight calculation unit 73, and the distance weight calculation unit 74 are deleted from the feature amount extraction unit 52 in the background influence feature amount extraction unit 52. In this case, as shown in FIG. 11, the feature amount extraction process does not include the process of setting the ratio weight and distance weight in steps S75 to S77 in FIG. 6, and the ratio weight and distance weight in step S130. A frequency that does not take into account is set. Note that the processing in steps S121 to S129 and S131 to S135 shown in the flowchart of FIG. 11 is the same as the processing of steps S71 to S74, S78 to S82, and S84 to S88 in FIG.

  According to the above, the feature amount can be extracted by excluding the elements of the background area that cause the recognition accuracy in the recognition process for recognizing the type of the object as the subject included in the image, and thus included in the image. It is possible to improve the recognition accuracy for recognizing the type of an object as a subject.

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 12 shows a configuration example of a general-purpose personal computer. This personal computer incorporates a CPU (Central Processing Unit) 1001. An input / output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

  The input / output interface 1005 includes an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of the processing result to a display device, programs, and various types. A storage unit 1008 including a hard disk drive for storing data, a LAN (Local Area Network) adapter, and the like, and a communication unit 1009 for performing communication processing via a network represented by the Internet are connected. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 1010 for reading / writing data from / to a removable medium 1011 such as a memory is connected.

  The CPU 1001 is read from a program stored in the ROM 1002 or a removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 to the RAM 1003. Various processes are executed according to the program. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

  In this specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series in the order described, but of course, it is not necessarily performed in time series. Or the process performed separately is included.

  DESCRIPTION OF SYMBOLS 11 Image processing apparatus, 31 Image acquisition part, 32 Distance detection part, 33 Target position detection part, 34, 35 Enlarging / reducing part, 36 Object recognition part, 41 Area division part, 42 Background influence reduction feature-value extraction part, 43 Recognition processing , 51 image processing unit, 52 feature quantity extraction unit, 61 ratio calculation unit, 62 ratio intensity calculation unit, 63 distance intensity calculation unit, 64 smoothing unit, 71 gradient calculation unit, 72 ratio calculation unit, 73 ratio weight calculation unit , 74 Distance weight calculator, 75 Histogram generator, 76 Normalizer

Claims (7)

Image acquisition means for acquiring images;
Distance acquisition means for acquiring information on the distance of the subject imaged in the image from the imaging device at the time of imaging of the image in units of pixels of the image;
Area dividing means for dividing the area into a foreground area as a subject in the image and a background area other than the image based on the image and the distance information;
A background for extracting a feature amount that reduces the influence of the background on the recognition of the type of the subject based on the image, the distance information, and / or all of the background area information divided by the area dividing unit. An influence reduction feature amount extraction means;
An image processing apparatus comprising: a recognizing unit that recognizes a type of a subject as a foreground in the image based on the feature amount extracted by the background effect reducing feature amount extracting unit. The background influence reducing feature amount extraction means includes:
Smoothing means for smoothing pixels corresponding to the background area of the image based on the image and information on the background area divided by the area dividing means;
The image processing apparatus according to claim 1, wherein a feature amount that reduces an influence on recognition of a subject type by the background is extracted from an image obtained by smoothing pixels corresponding to the background region by the smoothing unit. The said smoothing means smoothes a pixel with the intensity | strength according to the ratio of the background pixel of the pixel of the vicinity, and a foreground pixel with respect to the pixel of the background area | region divided | segmented by the said area | region division means. Image processing apparatus. The background influence reducing feature amount extraction means
Feature quantity extraction means for extracting the feature quantity of the image;
Weight setting means for setting a weight based on the image, distance information, and information on a background area divided by the area dividing means, and attaching the weight to the feature amount;
The image processing apparatus according to claim 1, wherein the feature amount to which the weight set by the weight setting unit is added is extracted as a feature amount that reduces the influence of the background on recognition of the type of subject. The image processing apparatus according to claim 1, wherein the feature amount extraction unit extracts a HOG feature amount. Image acquisition means for acquiring images;
Distance acquisition means for acquiring information on the distance of the subject imaged in the image from the imaging device at the time of imaging of the image in units of pixels of the image;
Area dividing means for dividing the area into a foreground area as a subject in the image and a background area other than the image based on the image and the distance information;
A background for extracting a feature amount that reduces the influence of the background on the recognition of the type of the subject based on the image, the distance information, and / or all of the background area information divided by the area dividing unit. An influence reduction feature amount extraction means;
An image processing method of an image processing apparatus, comprising: a recognition unit that recognizes a type of a subject that is a foreground in the image based on the feature amount extracted by the background effect reduction feature amount extraction unit.
An image acquisition step of acquiring the image in the image acquisition means;
A distance acquisition step of acquiring distance information from the imaging device at the time of imaging the image of the subject imaged in the image in the distance acquisition unit, in units of pixels of the image;
A region dividing step of dividing the region into a foreground region to be a subject in the image and a background region other than the image based on the image and the distance information in the region dividing unit;
Based on the background, the distance information, and / or all of the information of the background region divided by the region dividing step in the background influence reducing feature amount extraction unit, the type of the subject by the background is determined. A background effect reduction feature quantity extraction step for extracting a feature quantity that reduces the influence on recognition; and
A recognition step of recognizing the type of the subject as the foreground in the image based on the feature amount extracted by the processing of the background effect reduction feature amount extraction step in the recognition means. Image acquisition means for acquiring images;
Distance acquisition means for acquiring information on the distance of the subject imaged in the image from the imaging device at the time of imaging of the image in units of pixels of the image;
Area dividing means for dividing the area into a foreground area as a subject in the image and a background area other than the image based on the image and the distance information;
A background for extracting a feature amount that reduces the influence of the background on the recognition of the type of the subject based on the image, the distance information, and / or all of the background area information divided by the area dividing unit. An influence reduction feature amount extraction means;
A computer that controls an image processing apparatus including: a recognition unit that recognizes a type of a subject that is a foreground in the image based on the feature amount extracted by the background effect reduction feature amount extraction unit;
An image acquisition step of acquiring the image in the image acquisition means;
A distance acquisition step of acquiring distance information from the imaging device at the time of imaging the image of the subject imaged in the image in the distance acquisition unit, in units of pixels of the image;
A region dividing step of dividing the region into a foreground region to be a subject in the image and a background region other than the image based on the image and the distance information in the region dividing unit;
Based on the background, the distance information, and / or all of the information of the background region divided by the region dividing step in the background influence reducing feature amount extraction unit, the type of the subject by the background is determined. A background effect reduction feature quantity extraction step for extracting a feature quantity that reduces the influence on recognition; and
And a recognition step of recognizing the type of the subject as the foreground in the image based on the feature amount extracted by the processing of the background effect reduction feature amount extraction step in the recognition means.

Images