JP2008040724A - Image processing device and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device and an image processing method capable of dividing the region of an image with a high degree of precision even when the amount of information on the image is small. <P>SOLUTION: The image processing device acquires data of an image taken with an imaging device that takes the image of a periphery by the frame units in a time-series manner, and divides it into partial regions. The image processing device extracts the feature quantity based on the image data acquired from the imaging device, and acquires the coordinate values on a screen of the feature region where the extracted feature quantity meets a predetermined condition. The image processing device acquires movement information concerning the quantity of movement per unit time on the screen and the direction of movement for each pixel corresponding to the feature region. It acquires color information and/or brightness information for each pixel corresponding to the feature region. Based on the coordinate values on the screen of the feature region, the movement information, and the color information and/or the brightness information, the image processing device generates a feature space made up by a plurality of dimensions, divides the feature space into the partial space that meets a predetermined condition, specifies the coordinate values on the screen corresponding to the divided partial space, and divides the image into a plurality of partial regions. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method capable of accurately dividing an area of an image based on image data captured by an imaging apparatus that captures the periphery.

  When detecting an object to be recognized from an image captured by an imaging device capable of capturing the surroundings, the image is divided based on specific conditions in order to improve recognition accuracy, A method of performing object detection processing on the divided areas is often used. A method of dividing an image into a plurality of regions is based on region movement information (motion amount), color or luminance information, initial edge tracking, and the like.

  For example, as disclosed in Non-Patent Document 1, the amount of motion between frames for each pixel is grasped by calculating the deviation between the position of the previous frame and the position of the current frame in pixel units. A method of extracting an area larger than a predetermined amount as a partial area has been developed.

Further, as disclosed in Non-Patent Document 2, a luminance value or a Y, U, V signal value (or R, G, B signal value) is obtained in pixel units, and a signal value is obtained in the case of a color image. A method has been developed for extracting candidate regions by grouping pixels within a predetermined range.
N. Vasconcelos, 1 other, “Empirical Bayesian motion segmentation”, IEEE paper “Transactions on Pattern Analysis and Machine Intelligence”, p. 217-221, February 2001 D. Comaniciciu and one other, “Average Shift: A Robust Approach to Feature Domain Analysis”, IEEE Journal “Transactions on Pattern Analysis and Machine Intelligence”, p. 603-619, May 2002

  However, as disclosed in Non-Patent Document 1, in the image segmentation method based on the amount of motion between frames for each pixel, for example, when the amount of information is small, such as an image captured by a far-infrared imaging device However, there is a problem that the amount of motion cannot be accurately grasped, and there is a possibility that the area division of the image cannot be performed with high accuracy.

  Further, as disclosed in Non-Patent Document 2, the region dividing method for extracting a pixel group having similar luminance values for each pixel as one region is not a pixel that should originally be included in the region. In other words, there is a possibility that pixels having similar luminance values and the like may be included in the area, and there is a possibility that erroneous area division may be performed.

  The present invention has been made in view of such circumstances. For example, even an image with a small amount of information, such as an image captured by a far-infrared imaging device, can perform image segmentation with high accuracy. An object is to provide an image processing apparatus and an image processing method.

In order to achieve the above object, an image processing device according to a first aspect of the present invention obtains image data captured by an imaging device that images the periphery in time series in units of frames, and a portion in an image in which an object exists. In the image processing device for detecting a region, based on the image data acquired from the imaging device, movement information acquisition means for acquiring movement information about a movement amount and a movement direction per unit time on the screen for each pixel, and Color / luminance information acquisition means for acquiring color information and / or luminance information for each pixel based on image data acquired from the imaging device, coordinate values on the screen, movement information, and color information and / or luminance Based on the information, the feature space generating means for generating a feature space composed of a plurality of dimensions, and the feature space generated by the feature space generating means are divided into partial spaces having predetermined conditions. Identifying a coordinate value on the screen corresponding to the divided partial space, characterized in that it comprises an image dividing means for dividing an image into a plurality of partial regions.

  An image processing apparatus according to a second invention is the image processing apparatus according to the first invention, wherein, for the plurality of partial areas divided by the image dividing means, a means for calculating a similarity between adjacent partial areas, and a similarity calculated by the means Means for determining whether or not is greater than a predetermined value, and means for integrating adjacent partial areas into one partial area when it is determined that the means is larger.

  An image processing method according to a third aspect of the present invention is an image processing method for detecting a partial region in an image in which an object is present by acquiring image data captured by an imaging device that images the periphery in time series in units of frames. In the above, based on the image data acquired from the imaging device, movement information regarding a movement amount and a movement direction per unit time on the screen is acquired for each pixel, and the pixel is determined based on the image data acquired from the imaging device. Color information and / or luminance information is acquired for each, and a feature space composed of a plurality of dimensions is generated and generated based on the coordinate values, movement information, and color information and / or luminance information on the screen. The feature space is divided into partial spaces having predetermined conditions, and coordinate values on the screen corresponding to the divided partial spaces are specified to divide the image into a plurality of partial regions.

  In the first invention and the third invention, image data captured by an imaging device that images the periphery in a time-series manner is acquired, and a partial region in an image in which an object is present is detected. Based on the image data acquired from the imaging device, movement information regarding the movement amount and movement direction per unit time on the screen is acquired for each pixel. On the other hand, based on the image data acquired from the imaging device, color information and / or luminance information is acquired for each pixel. A feature space composed of a plurality of dimensions is generated based on coordinate values on the screen, movement information, and color information and / or luminance information, and the generated feature space is converted into a partial space having a predetermined condition. The image is divided, the coordinate value on the screen corresponding to the divided partial space is specified, and the image is divided into a plurality of partial areas. Thus, by generating a multi-dimensional feature space considering not only movement information between frames for each pixel but also color information and / or luminance information for each pixel, acquisition of movement information, color information and / or Alternatively, even when luminance information acquisition reliability is low, it is possible to perform region segmentation of an image with high accuracy by using these in a complementary manner.

  In the second invention, for a plurality of divided partial areas, the similarity between adjacent partial areas is calculated, and if the calculated similarity is greater than a predetermined value, the adjacent partial areas are integrated into one partial area. . Thereby, it becomes possible to update the partial area specified by being excessively divided to an appropriate size, for example, a size close to the physical size of the object to be detected.

  According to the first and third aspects of the invention, by generating a multi-dimensional feature space that considers not only movement information between frames for each pixel but also color information and / or luminance information for each pixel, Even when the acquisition of information and the reliability of acquisition of color information and / or luminance information is low, it is possible to perform image segmentation with high accuracy by using these in a complementary manner.

  According to the second aspect of the present invention, it is possible to update the partial area specified by being excessively divided to an appropriate size, for example, a size close to the physical size of the object to be detected.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an image captured by the far-infrared imaging device while the vehicle is traveling is divided into partial areas, and obstacles existing in front of the vehicle, for example, using the divided partial areas, for example, A case where the presence of a pedestrian, a bicycle, or the like is detected will be described as an example.

  FIG. 1 is a block diagram showing a configuration of an image processing apparatus 1 according to an embodiment of the present invention. The image processing apparatus 1 according to the embodiment of the present invention includes at least an LSI 11, a RAM 12, an image memory 13, a video input unit 14, a video output unit 15, and an alarm output unit 16.

The LSI 11 reads out the image data stored in the image memory 13 in units of frames and extracts a feature amount for each pixel. The LSI 11 uses, for example, the L ^* u ^* v ^* color specification as the coordinate values x and y on the screen, the motion amounts Vx and Vy in units of frames, and color information for an area where the extracted feature value satisfies a predetermined condition. The system signal value L ^* u ^* v ^* signal value is acquired, and a seven-dimensional feature space is generated. Hereinafter, the case of the 7-dimensional case will be described. However, when a gray image is targeted and luminance information is used instead of color, one-dimensional information on luminance information is used instead of L ^* u ^* v ^* . Therefore, the feature space is five-dimensional. The LSI 11 divides the feature space into a plurality of partial spaces by grouping points corresponding to pixels on the feature space for each predetermined distance range. An area on the screen is divided into a plurality of partial areas based on the divided partial spaces.

  The RAM 12 is an SRAM, a flash memory, or the like, and stores time-series position data of an area in which data generated in the course of arithmetic processing and a feature amount satisfy a predetermined condition, that is, a movement amount, a movement vector, and the like. The image memory 13 is an SRAM, flash memory, SDRAM or the like, and stores image data input from the far-infrared imaging devices 2 and 2 via the video input unit 14.

  The video input unit 14 receives input of image data captured by the far-infrared imaging devices 2 and 2 via a video cable 7 compatible with an analog video system such as NTSC or a digital video system. The image data acquired by the video input unit 14 is stored in the image memory 13 in synchronization with each frame, for example.

  The video output unit 15 outputs image data to the display device 4 such as a liquid crystal display via a cable 8 compatible with a video system such as NTSC, VGA, DVI, and the alarm output unit 16 has voice, sound effects, etc. Thus, an output signal such as a synthesized sound is transmitted to the alarm device 5 that emits an audible warning via the in-vehicle LAN cable 6 compliant with CAN.

  In the present embodiment, the far-infrared imaging devices 2 and 2 that capture an image around the vehicle are mounted in a front grill near the center in front of the vehicle. The far infrared imaging devices 2 and 2 are imaging devices using infrared light having a wavelength of 7 to 14 micrometers. In addition, it is not limited to using the far-infrared imaging devices 2 and 2 as an imaging device, The near-infrared imaging device using the infrared light whose wavelength is 0.8-3 micrometers may be sufficient, A visible light imaging device may be used.

FIG. 2 is a block diagram showing the configuration of the far-infrared imaging device 2 used in the image processing device 1 according to the embodiment of the present invention. The image pickup unit 21 includes image pickup elements that convert optical signals into electric signals in a matrix. As an image sensor for infrared light, an infrared sensor such as a vanadium oxide bolometer type using a micromachining technique or a pyroelectric type BST (Barium-Strontium-Titanium) is used. The image capturing unit 21 reads an infrared light image around the vehicle as a luminance signal, and transmits the read luminance signal to the signal processing unit 22.

The signal processing unit 22 is a LSI, converts the luminance signal received from the image capturing unit 21 to the ^{L * u * v * L *} u * v * digital signal such as a signal a color system signals of the imaging device The process of correcting the variation of the defect, the correction process of the defective element, the gain control process, and the like are performed and stored in the image memory 23 as image data. Needless to say, it is not essential to temporarily store the image data in the image memory 23, and the image data may be transmitted directly to the image processing apparatus 1 via the video output unit 24.

  The video output unit 24 is an LSI, and outputs video data to the image processing apparatus 1 via a video cable 7 compatible with an analog video system such as NTSC or a digital video system.

  Hereinafter, detailed processing in the LSI 11 of the image processing apparatus 1 will be described. FIG. 3 is a flowchart showing a processing procedure of the LSI 11 of the image processing apparatus 1 according to the embodiment of the present invention.

  The LSI 11 reads out the image data acquired from the far infrared imaging devices 2 and 2 stored in the image memory 13 (step S301).

  The LSI 11 calculates the amount of motion d (x, y, t) from the previous frame for each pixel (for each coordinate value (x, y)) based on the image data stored in the image memory 13 (step) S302). The method of calculating the amount of motion in units of frames for each pixel is not particularly limited, and for example, a well-known Kanade-Lucas-Tomasi tracker (hereinafter referred to as KLT) is used.

  In KLT, when the brightness of the coordinate value (x, y) at time t is I (x, y, t), the motion amount d (x, y, t) between frames can be expressed by (Equation 1). it can. In (Equation 1), τ represents the frame interval, and n represents the noise generated by the movement of the imaging device and the object in the image.

I (x, y, t + τ) =
I (x−d (x, y, t), yd (x, y, t), t) + n (x, y, t)
... (Equation 1)

  Therefore, by searching for the motion amount d (x, y, t) such that the noise n (x, y, t) in (Equation 1) is smaller than a predetermined threshold, the coordinate position (x, y) is supported. Corresponding feature points.

  However, in estimating the amount of motion using KLT, only the amount of motion of the associated feature point can be estimated. In order to obtain the motion amount for all the pixels, in this embodiment, it is assumed that the motion amount of the feature points that are not associated with each other is equivalent to the motion amount of the spatially closest feature point. Complement the amount of movement. In this way, it is possible to estimate the amount of motion for pixels from which feature points have not been extracted.

FIG. 4 is a diagram illustrating an example of a motion amount extraction result of the image processing apparatus 1 according to the embodiment of the present invention. As shown in FIG. 4, for example, it can be seen that movement vectors having the same direction and amount of movement are distributed in the vicinity of the feature point 41. Therefore, it is possible to divide the region by grouping pixels having similar movement vectors only from the amount of motion. However, in an image (frame) in which it is difficult to detect the amount of motion, since the reliability of the amount of motion is low, it is not possible to accurately divide the region.

Next, the LSI 11 acquires color information and / or luminance information for each pixel (for each coordinate value (x, y)) based on the image data stored in the image memory 13 (step S303). When the acquired image is a color image, for example, L ^* u ^* v ^* signal value, which is a signal value of L ^* u ^* v ^* color system, is used as color information, and as luminance information when the image is a monochrome image. Each luminance value is acquired.

The LSI 11 is configured in a plurality of dimensions based on the coordinate values (x, y), the motion amount d (x, y, t) on the screen of the extracted feature region, and color information and / or luminance information. A feature space is generated (step S304). For example, when the image data stored in the image memory 13 is color image data, the coordinate value (x, y), the amount of motion (Vx, Vy) on the screen of the extracted feature region, and L that is color information. ^* u ^* v ^* Generates a seven-dimensional feature space of signal values (l, u, v).

FIG. 5 is a diagram schematically illustrating a feature space generation method. As shown in FIG. 5A, with respect to the coordinate value (x, y) of an arbitrary pixel P of the image data stored in the image memory 13, the amount of motion (Vx, Vy) and L ^* which is color information ^. u ^* v ^* signal values (l, u, v) are calculated. Using the calculated motion amount (Vx, Vy) and color information L ^* u ^* v ^* signal values (l, u, v), as shown in FIG. Corresponding points P ′ are plotted on the converted seven-dimensional feature space (x ′, y ′, Vx ′, Vy ′, l ′, u ′, v ′). By doing so, the image data can be mapped to the seven-dimensional feature space. That is, each of the corresponding points in FIG. 5B corresponds to the coordinate value (x, y) of an arbitrary pixel of the image data stored in the image memory 13, and the feature space of each corresponding point is It is possible to calculate the spatial distance from the origin.

Note that the normalized seven dimensions (x ′, y ′, Vx ′, Vy ′, l ′, u ′, v ′) are arbitrary coordinate values (x, y), the amount of motion (Vx, Vy), and L ^* u ^* v ^* signal values (l, u, v), which are color information, are values divided by weighting coefficients, respectively.

  The LSI 11 divides into one or a plurality of partial spaces depending on whether or not a predetermined condition is satisfied in the specified feature space, for example, whether or not the distance on the feature space is shorter than the predetermined distance ( Step S305). Specifically, the LSI 11 calculates the spatial distance in the feature space for each corresponding point corresponding to the coordinate value (x, y) of the pixel, thereby corresponding points whose spatial distance is close in the feature space. A subspace is specified with a set of as a group. The LSI 11 specifies a partial area on the screen corresponding to each partial space using the coordinate values (x, y) included in the divided partial space (step S306).

  FIG. 6 is a diagram schematically illustrating a partial region dividing method. As shown in FIG. 6B, when the feature space is divided into a plurality of partial spaces 61, 61,..., The coordinate values of each partial space 61 are seven-dimensional coordinate values. Since the two-dimensional coordinate value (x, y) indicating the position on the screen is included therein, the partial spaces 61, 61,... Correspond one-to-one as shown in FIG. Can be mapped to the pixel groups 62, 62,... On the screen. Therefore, the area formed by the pixel groups 62, 62,... Can be divided as partial areas.

If the amount of information is small, such as images captured by the far-infrared imaging devices 2 and 2, the partial area cannot be divided with high accuracy only by the amount of pixel movement between frames. In addition, in the method of dividing a pixel group having similar color information for each pixel as a partial region, there is a risk of erroneously recognizing that pixels having similar brightness values belong to the same partial region even though they are different pixels. is there. On the other hand, as shown in FIG. 6B, by generating one feature space including the amount of motion and color information, and dividing it into partial spaces according to the spatial distance in the generated feature space. Even when the acquisition of the amount of motion and the reliability of the acquisition of color information and / or luminance information are low, it is possible to perform image segmentation with high accuracy by using these complementarily. .

  FIG. 7 is a diagram illustrating an example of the case where the image is divided into partial areas using only color information and / or luminance information. Since it is difficult to distinguish the background image from the background image, it cannot be divided as a partial region where the object can be detected. On the other hand, FIG. 8 is a diagram illustrating a result of performing image processing according to the present embodiment on the same image data. As shown in FIG. 8, one feature space including the amount of motion and color information is generated, and an object is detected by dividing the feature space into subspaces according to the spatial distance in the generated feature space. It is possible to accurately identify a partial area where the image can be captured.

  The LSI 11 determines whether or not an object such as a pedestrian or a bicycle exists for the divided partial area by a known method, and transmits warning display data to the display device 4 according to the determination result. Or an alarm signal is transmitted to the alarm device 5.

  Note that, when a partial area is divided from a multi-dimensional feature space, it may be assumed that the partial area is excessively subdivided and specified depending on the partial space division algorithm or the size of the threshold. Therefore, the LSI 11 calculates a similarity between the divided partial areas, and integrates the partial areas determined to be similar based on the calculated similarity to obtain a new partial area.

Specifically, the LSI 11 calculates the mean square of the difference between the motion amount (Vx, Vy) and the color information L ^* u ^* v ^* signal value (l, u, v) for adjacent partial regions on the image. Is calculated. The LSI 11 determines whether or not the calculated root mean square is smaller than a predetermined threshold. If the LSI 11 determines that the LSI 11 is smaller than the predetermined threshold, the LSI 11 determines that the similarity between both partial areas is high, and Merge areas.

  By doing so, it is possible to prevent the partial area from being excessively subdivided, and it is possible to identify the optimal partial area for detecting an object present in the image.

  Note that the method is not limited to the method of integrating the partial regions based on the similarity, and the partial regions separated by using, for example, morphological processing may be integrated. The morphological process uses, for example, Opening.

  As described above, according to the present embodiment, by generating not only the movement information between frames for each pixel but also the multi-dimensional feature space considering the color information and / or luminance information for each pixel, Even when the acquisition of information and the reliability of acquisition of color information and / or luminance information is low, it is possible to perform image segmentation with high accuracy by using these in a complementary manner.

It is a block diagram which shows the structure of the image processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the far-infrared imaging device used with the image processing apparatus which concerns on embodiment of this invention. It is a flowchart which shows the procedure of the process of LSI of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the extraction result of the motion amount of the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows typically the production | generation method of feature space. It is a figure which shows typically the division | segmentation method of a partial region. It is a figure which shows an example at the time of dividing | segmenting into a partial area using only color information and / or luminance information. It is a figure which shows the result of having performed the image process which concerns on this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Far-infrared imaging device 4 Display apparatus 5 Alarm apparatus 6 Car-mounted LAN cable 7 Video cable 8 Cable 11 LSI
12 RAM
13 Image memory 14 Video input unit 15 Video output unit 16 Alarm output unit

Claims (3)

In an image processing device that acquires image data captured by an imaging device that images the periphery in time series in units of frames, and detects a partial region in an image in which an object exists,
Based on the image data acquired from the imaging device, movement information acquisition means for acquiring movement information regarding a movement amount and a movement direction per unit time on the screen for each pixel;
Color / luminance information acquisition means for acquiring color information and / or luminance information for each pixel based on image data acquired from the imaging device;
Feature space generating means for generating a feature space composed of a plurality of dimensions based on coordinate values on the screen, movement information, and color information and / or luminance information;
The feature space generated by the feature space generation means is divided into partial spaces having predetermined conditions, and coordinate values on the screen corresponding to the divided partial spaces are specified to divide the image into a plurality of partial regions. An image processing apparatus comprising: an image dividing unit. Means for calculating a similarity between adjacent partial areas for a plurality of partial areas divided by the image dividing means;
Means for determining whether the similarity calculated by the means is greater than a predetermined value;
The image processing apparatus according to claim 1, further comprising: means for integrating adjacent partial areas into one partial area when the means determines that the area is large. In an image processing method for acquiring image data captured by an imaging device that captures the periphery in a time-series manner in units of frames and detecting a partial region in an image in which an object exists,
Based on the image data acquired from the imaging device, to acquire movement information about the movement amount and movement direction per unit time on the screen for each pixel,
Based on the image data acquired from the imaging device, color information and / or luminance information is acquired for each pixel,
Based on coordinate values on the screen, movement information, and color information and / or luminance information, a feature space composed of a plurality of dimensions is generated,
Dividing the generated feature space into partial spaces having a predetermined condition, specifying coordinate values on the screen corresponding to the divided partial spaces, and dividing the image into a plurality of partial regions Image processing method.

Images