JP5013319B2 - Stereo image processing apparatus and image processing program - Google Patents

Description

  The present invention relates to a stereo image processing apparatus and a program for image processing for recognizing a foreign object having a height from a floor surface by a stereo camera having parallel optical axes.

As a method for extracting a three-dimensional foreign object in stereo image processing, for example, as described in Patent Document 1, a fine parallax between left and right images is associated with each other to obtain a parallax, thereby calculating a distance from the camera, Some require a three-dimensional distance data image. A three-dimensional foreign object such as an intruder or an obstacle can be extracted from the three-dimensional image thus obtained. However, there is a problem that this processing is complicated and the processing load on the processing apparatus is very heavy. Although not described in the patent literature, a method of extracting a three-dimensional foreign object by comparing left and right images obtained by projective transformation from an image captured by a stereo camera whose optical axes are not parallel is also known.
JP 2005-270484 A

  The present invention has been made to solve the problem of the processing burden of the prior art, and its purpose is to perform a complicated process such as mapping conversion as in the conventional apparatus using a stereo camera with parallel optical axes. It is to be able to accurately extract three-dimensional foreign objects such as intruders and obstacles by simply drawing (or placing) a single line on the floor without processing.

The invention according to claim 1 is a stereo image processing apparatus for detecting a foreign object, and among the left and right images obtained by imaging the reference line drawn on the floor surface of the monitoring area, the reference line of one image is used as the reference image of the other image. Means for obtaining a shift amount necessary for matching with a reference line, first shift processing means for shifting the one image according to the shift amount, and a difference between the shifted image and the other image A means for storing the partial image as a background image, a means for recording the acquired shift amount in the storage means, and one of the left and right images of the monitoring area is shifted according to the shift amount recorded in the storage means. A second shift processing means; a means for forming a monitoring area difference image between the image of the monitoring area subjected to the shift process and the image of the other monitoring area; and a difference image between the monitoring area difference image and the background image. And having means for, means for extracting foreign image from the difference image.
According to a second aspect of the present invention, in the stereo image processing apparatus according to the first aspect, the means for obtaining the necessary shift amount is the image in the one image when the reference lines of the left and right images are matched. Means for calculating the shift amount of the edge image for each of a plurality of lines crossing the line.
A third aspect of the present invention, the stereo image processing apparatus according to claim 1 or 2, characterized in that it comprises means for binarizing the foreign object extraction image obtained by means for extracting the foreign image.
A fourth aspect of the present invention is a stereo image processing program for causing a computer of a stereo image processing apparatus to function as each means of the stereo image processing apparatus according to any one of the first to third aspects.

  According to the present invention, a three-dimensional foreign object can be easily extracted by a relatively simple process. That is, in a stereo camera with parallel optical axes, it is possible to extract a three-dimensional foreign object by simply drawing a single reference line on the floor without performing projective transformation.

  The present invention is a stereo image processing system for recognizing a foreign object having a height from the floor surface by a stereo camera having parallel optical axes, and calculates a matching of line luminance change for each frame as in the prior art. Focusing on the fact that the ground is a flat surface instead of obtaining three-dimensional data, mechanical conversion is performed so that the left and right parallax disappear on that surface, that is, the difference between one image and the other image that has been converted in advance. Only foreign matter floating from the ground is extracted using an image.

Embodiments of the present invention will be described with reference to the drawings.
FIG. 1A is a block diagram schematically illustrating a stereo image processing system according to an embodiment of the present invention, and FIG. 1B is a block diagram schematically illustrating a stereo image processing apparatus.
The stereo image processing system 1 includes a pair of digital stereo cameras 10 and an image processing device 20. The digital stereo camera 10 may be commercially available, and image data captured by the digital stereo camera 10 is sent to the image processing apparatus 20 via a known interface unit 30 such as a USB (Universal Serial Bus) cable.

As shown in FIG. 1B, the image processing apparatus 20 includes a control unit 22 that is a computer for controlling the whole, an input unit 24 for inputting data and commands, and a display unit used for confirming input and output. It consists of 26.
The control unit 22 temporarily stores processing data and the like as a CPU (Central Processing Unit) 200 that performs data processing and calculation, a ROM (Read Only Memory) 202 that records programs for operating the CPU 200, and the work area of the CPU 200. RAM (Random Access Memory) 204 and image memory 206 made up of NV (Non-Volatile) RAM.
In this embodiment, since the burden for image processing can be significantly reduced as compared with the conventional case, the CPU 200 is a combination of a field programmable gate array (FPGA) and a low performance CPU, or a low level instead of the CPU 220. A cost DSP (Digital Signal Processor) can be used, thereby reducing the cost. Here, these things are called CPU.

In the above configuration, first, the principle in the setting stage of the stereo image processing system will be schematically described.
The stereo image processing system (1) obtains a stereo image by placing one marker line ML as a reference line at the center of the screen in the absence of foreign matter. Here, the marker line (reference line) ML is arranged at the center of the image. However, the reference line may be arranged at any position as long as it is reflected in the left and right camera images, not the center of the image. Also good.
(2) Of the acquired left and right images, paying attention only to the center of the screen, the marker line ML is placed on the right for each of a plurality of lines (for example, about 30) that are horizontally projected on the entire screen. A shift amount necessary to match the marker line ML of the image is calculated, and a table of the shift amount is created.
(3) Referring to the table, a left image is created by shifting the left image for each line (that is, a kind of affine transformation is performed).
(4) A difference image between the right image and the shift-converted left image is created (stereo difference background image).

The above is the outline of the stereo image processing. Next, it will be described in more detail.
The stereo image processing is composed of two stages, that is, a setting stage for preparation and an operation stage after setting when extracting a foreign object from a stereoscopic image.
First, the setting stage will be described. In the setting stage, two fixed digital cameras 10 for imaging the monitoring area are installed on the left and right sides (or a single stereo camera). As shown in FIG. The vertical marker line ML of the book is drawn, and then the marker line ML of the image (see FIG. 3A) taken by the left camera and the image (see FIG. 3B) taken by the right camera are sent to the image processing device 20. In the image processing device 20, the left image is shifted here for each of the plurality of lines attached to the image, as will be described later, so that the marker line ML in the video is at the same position by the processing of the control unit 22. (See FIG. 3C). The shift amount is recorded in a table provided in the NVRAM 206. In this state, the difference image (see FIG. 3D) between the left and right images is stored in the NVRAM 206 as a background image.
Finally, the marker line ML is removed to complete the setting. This setting only needs to be performed once at the time of installation.

FIG. 4 is a flowchart for explaining processing performed at the time of setting.
In the installation setting, the shift amount for each line is calculated. First, the installer installs two cameras (uses a stereo camera with parallel optical axes: a single stereo camera) so as to be parallel to the floor surface, and places a marker line ML on the floor ( (Or pull) to prepare for installation.
Next, the marker line ML is imaged with the installed stereo camera, and the left image and the right image are input (S101). Next, distortion of the input image, that is, distortion correction is performed (S102). After correcting the distortion of the input image, the density of the left image is detected for each pixel, and the detected value is compared with a predetermined threshold value. An edge of the image is detected, and a binarized image of the edge is created (S103). For the obtained edge image, a plurality of lines that divide the image in the horizontal direction are defined, and the shift amount of the edge image for each line in the left image when the left image and the marker line ML of the right image are matched is calculated. Then, the calculated shift amount for each line is recorded in a table (S105), and the background left image and the right image are stored in the image memory (S106).

Here, an example of a method for calculating the shift amount for each line performed by the CPU 200 at the time of installation will be described. If it is assumed that the marker line ML is near the center of the image, the pixel at the center position of the binarized image of the edge of the created left image is scanned from the left to the right to search for the edge image. If an edge is found, the part corresponding to the block of the left camera image (for example, a block of 9 × 9 pixels) centered on the pixel of the edge is detected how many blocks are shifted horizontally in the right camera image, The shift amount is set as the shift amount of the line.
As the evaluation method, for each pixel in the block, a method is adopted in which the block that is the smallest in the search range in which the absolute values of the luminance differences are summed is used as a pixel coincidence point. The same processing is performed for all lines.

FIG. 5A is a diagram showing the scanning range of the edge of the marker line performed when acquiring the shift amount, and FIG. 5B is a flowchart for explaining the process of recording the shift amount.
First, an edge point is found within a certain scanning range of the left image (here, within the width of the treatment including the marker line ML) (S201: see FIG. 5A). Once the edge point is found, which point in the right image matches that point, the right block on the line and the block that is the smallest in the search range with the sum of the absolute values of the luminance differences as described above. By searching (S202), the number of blocks that are shifted is counted. The shift amount (number of blocks) of the block is recorded as a shift amount. This is performed for all lines, and the shift amount for each line is recorded in the table (S203).

Next, processing during operation after setting will be described.
In operation, the table is referred to, and a three-dimensional foreign object is extracted by taking a difference between an image obtained by shifting and deforming the left image and a right image and a background image stored at the time of setting.

The processing at the time of operation will be schematically described below. FIG. 6 shows a captured image and a processed image during operation.
(1) A stereo image is taken by two stereo cameras.
(2) An image (FIG. 6B) obtained by shift-converting the left image (FIG. 6A) based on the table created at the time of setting is created.
(3) A difference image between the right image and the shift-converted left image is created (FIG. 6C).
(4) A difference image between the stereo background left image at the time of installation setting and the difference image is created (FIG. 6D). When binarization processing is performed on this difference image, an object floating above the ground appears.
(5) Further, foreign matter is extracted with, for example, about 40 × 30 pixels by noise removal, blocking, labeling processing, and the like.

Next, the process of extracting a moving solid foreign object based on the background image difference described above will be described with reference to the flowchart of FIG.
In the figure, as in the setting, first, a left image and a right image captured by a stereo digital camera are input (S301), distortion correction of the input image is performed (S302), and then a background left image and a current left image A shift-converted image obtained by shifting the background image and the current left image by the shift amount read from the table for each line is created (S303: see FIG. 6B).
A background stereo difference image obtained by subtracting the background right image from the obtained background shift converted image is obtained (S304), and a current stereo difference image obtained by subtracting the current right image from the current left shift converted image is obtained (S305). : Refer to FIG. 6C), a foreign object extraction image is acquired by subtracting the current stereo difference image from each acquired background stereo difference image (S306: refer to FIG. 6D), and the acquired foreign object extraction image is obtained by the above-described processing. Binarization is performed (S307).
Here, noise is removed from the obtained binarized image by a normal method, binarized pixels are blocked, labeling processing is performed, contraction and expansion processing are performed as necessary, and a foreign object image is extracted. The process ends (S308).

  In the above description, the marker line ML does not have to be shown on all lines. If all the floors shown are flat or have a certain inclination angle (FIG. 2), As shown in FIG. 8, after calculating the shift amount of only the line in which the marker line ML is shown, an approximate linear equation or approximate curve formula of the shift amount is obtained by the least square method, and the shift of the portion where the marker line ML is not shown The quantity can be estimated and applied from the obtained approximate expression. However, if the angle of inclination of the reflected floor is not constant, it cannot be estimated, for example, on a hill-plane-slope or stairs.

  In the above description, the case where the left image is shifted has been described as an example, but conversely, the right image may be processed so as to be shifted in accordance with the left image.

  As described above, according to the present embodiment, a small amount of image processing is performed on the floor surface using the parallax of the left and right images based on one marker line drawn (or placed) on the floor surface. Can be easily extracted.

FIG. 1A is a block diagram schematically showing a stereo image processing system according to an embodiment of the present invention, and FIG. 1B is a block diagram schematically showing a stereo image processing apparatus. The camera arrangement and the marker line drawn on the floor in the setting stage are shown. The image captured by the left and right cameras and the processed image are shown. It is a flowchart for demonstrating the process performed at the time of a setting. FIG. 5A is a diagram illustrating the scanning range of the edge of the marker line that is performed when acquiring the shift amount at the time of setting, and FIG. 5B is a flowchart for explaining the process of recording the shift amount at the time of setting. Shows captured images and processed images during operation It is a flowchart which shows the process of extracting a movement solid foreign material by a background image difference. It is a figure explaining the method of calculating | requiring shift amount calculation by an approximate expression.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Stereo image processing system, 10 ... Digital stereo camera, 20 ... Image processing apparatus, 22 ... Control part, 24 ... Input part , 26 ... Display part , 200 ... CPU , 202 ... ROM, 204 ... RAM, 206 ... NVRAM (image memory) , ML.

Claims (4)

A stereo image processing device for detecting foreign matter,
Means for acquiring a shift amount necessary to match the reference line of one image with the reference line of the other image among the left and right images obtained by imaging the reference line drawn on the floor surface of the monitoring area;
First shift processing means for shifting the one image according to the shift amount;
It means for storing the difference of image and the shift processed image and the other image as the background image,
Means for recording the acquired shift amount in a storage means;
A second shift processing means for performing a shift process on one of the left and right images of the monitoring area according to the shift amount recorded in the storage means;
Means for forming a monitoring area difference image between the shift-processed monitoring area image and the other monitoring area image;
Means for forming a difference image between the monitoring region difference image and the background image;
It means for extracting foreign image from the difference image,
A stereo image processing apparatus comprising: The stereo image processing apparatus according to claim 1,
The means for acquiring the necessary shift amount includes means for calculating the shift amount of the edge image for each of a plurality of lines crossing the image in the one image when the reference lines of the left and right images are matched. A stereo image processing apparatus. The stereo image processing apparatus according to claim 1 or 2,
Stereo image processing apparatus characterized by comprising means for binarizing the foreign object extraction image obtained by means for extracting the foreign image. A stereo image processing program for causing a computer of a stereo image processing apparatus to function as each of the means of the stereo image processing apparatus according to claim 1.

Images