JPH0229878A - Image processing method - Google Patents

Abstract

PURPOSE:To simplify image processing by expanding data corresponding to picture elements constituting the boundary of a texture area to its inside. CONSTITUTION:Since a certain kind of constitutional elements exist in the texture area in comparatively high density and the elements have almost equal characteristics in a restricted range, the data corresponding to the picture elements constituting the boundary of the texture area are expanded to the inside. Since the processing required for said expansion is only the processing for finding out correlation between the picture elements of the 1st texture area and the picture elements of the 2nd texture area indicated by the corresponding data included in picture elements adjacent to said picture elements or their peripheral picture elements and the processing for finding out data corresponding to the picture elements of the 1st texture area from the correlation, the processing is extremely simple.

Description

[Detailed Description of the Invention] [Object of the Invention] (Industrial Field of Application) The present invention relates to a technique for associating component pixels between a plurality of texture areas, for example, for mapping between corresponding areas in a stereo image. The present invention relates to a technique for detecting parallax in composed images.

(Prior Art) A typical image includes regions having various properties. Texture plays an important role in visually determining the identity of each area.
For example, in an image containing a melon, the texture (pattern) of the melon separates the area of the melon from other areas, and the distant texture appears fine and the nearby texture appears coarse, so the perspective, or three-dimensional It expresses the shape.

In other words, for example, the same object is imaged using two cameras whose imaging planes are substantially the same and their optical axes are parallel to each other, and the pixels constituting each texture area are associated with each other to find the parallax. This allows the distance to the target to be determined and reproduced in a three-dimensional image.

BACKGROUND ART Conventionally, a method has been known in which constituent elements are extracted as line segments from each image, feature amounts of each are determined, and the extracted line segments are associated with each other.

(ll1I to be solved by the invention) However, this method is only effective for simple images, and when applied to an area containing a large number of constituent elements at high density, such as a texture area, each constituent element In order to extract each line segment as independently distinguishable line segments, a huge amount of memory is required. Furthermore, even if this problem is solved, the number of line segments that are correspondence candidates increases due to the high spatial density of the constituent elements, and it is virtually impossible to consider correspondence for each line segment.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing method that appropriately associates constituent pixels of a plurality of texture areas with each other through simple processing.

[Structure of the invention]

(Means for Solving Problem 11) In order to achieve the above object, in the present invention, the first texture area and the second texture area are divided from an original image composed of pixels of minute sections.

A feature amount of a boundary line of a first texture area.

Correspondence data indicating the correspondence between pixels forming the border of the first texture area and pixels forming the border of the second texture area by comparing the feature amount of the border of the second texture area is determined in association with each of the pixels forming the boundary line of the first texture area, and the pixels in the first texture area and the pixels in the second texture area indicated by the corresponding data of at least the pixels adjacent thereto. It is assumed that correspondence data of pixels in the first texture area is determined from the correlation, and corresponding pixels of the second texture area are determined from correspondence data of each pixel in the first texture area.

(Effect) According to this, in a texture area, certain types of constituent elements exist at a relatively high density and have approximately equal characteristics in a limited range. The corresponding data is expanded inside it.

What is the process required for this?

A process for determining a correlation between a pixel in a first texture area and a pixel in a second texture area or its surrounding pixels indicated by corresponding data of a pixel adjacent to the pixel, and a process for determining the correlation between a pixel in the first texture area and a pixel in the second texture area or its surrounding pixels; The only process required is to obtain the corresponding data of the pixels, and is extremely simple. In this case, the reliability of the processing is high because the present invention is based on the characteristics of the texture area described above.

In a preferred embodiment of the present invention, the first texture region and the second texture region are each divided from different original images having the same object. Furthermore, the correspondence data of pixels in the first texture area is obtained by sequentially focusing on the constituent pixels in the first texture area from a certain direction, and then sequentially observing the constituent pixels from the opposite direction. Therefore, we are correcting it. This greatly increases processing accuracy.

It should be noted here that the present invention does not intend to limit the number of texture regions for which correspondence is sought to two.

That is, when there are three or more texture areas, if any one of them is set as the first texture area and the others are set as the second texture area, the corresponding pixels can be found using exactly the same procedure.

Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the drawings.

(Embodiment) FIG. 1a shows the configuration of an image processing apparatus that implements the present invention by way of example.

This device consists of a computer 1. ITV camera unit 2
and 39 image memory 4. Display 5. It consists of a printer 6, a floppy disk 7, a keyboard terminal 8, etc.

The ITV camera units 2 and 3 are ITV cameras 21 and 31 with the same specifications, respectively, which image the scene in front and output analog data in 512 x 512 pixel divisions.
and A/D converters 22 and 32 that digitally convert the output analog data of the ITV camera 21 or 31 to generate original image data of the 256th floor g.

The ITV' cameras 21 and 31 are shown in FIG. They are arranged horizontally with the ITV camera 31 on the right, their optical axes 211 and 311 are parallel, and their imaging planes 212 and 312 are on the same plane. Therefore, the top side of each imaging plane is the XL axis, yXR axis, and the left side is the YL axis, respectively.
When the YR axis is used, the XL axis and the XR axis are on the same straight line, and the YL axis and the 7th generation axis are parallel. Note that in the following, the image captured by the ITV camera 21 will be referred to as the left image and the ITV camera 21.
The image captured by the V camera 31 will be referred to as a right image.

The image memory 4 is readable and writable, and stores various processed data including original image data of the left image and right image.

The display unit 5 and printer 6 output the processing results of the computer 1, and the floppy disk 7 registers the processing results. Further, the keyboard terminal 8 is operated by an operator to input various instructions.

A host computer is further connected to the computer 1, and controls each section according to instructions given from the host computer or instructions given from the keyboard terminal 8, and performs corresponding processing.

Among these processes, "stereoscopic processing" for displaying three-dimensional graphics of the scene in front will be described below.

FIG. 2 is a general flow showing an outline of a stereoscopic processing routine that is activated by a stereoscopic processing execution instruction given from the keyboard terminal 8, and FIGS. 3a to 3h are subflows showing important steps therein in detail. It is.

At the start of the three-dimensional processing, the computer 1 uses Sl (
In S2, the image memory 4 and the registers used in this process are initialized, and in S2 the ITV camera unit 2 is initialized.
and 3, the original image data of the left image and the right image are written into the image memory 4. After this, in S3, each image is divided.

This image division will be explained with reference to FIG. 3a.

In 5101, forward raster scan is set for the left image. Forward raster scanning is performed within the imaging area 212 (312) shown in Figure 1b, with the XL (XR) axis as the main scanning axis and the YL (YR) axis as the sub-scanning axis, from the upper left pixel to the lower right pixel. This is a scan that focuses on each pixel along the path, and here the scan start position is initialized. Incidentally, the reverse raster scan, which will be described later, refers to a scan in which each pixel is focused on a path from the lower right pixel to the upper left pixel.

In steps 8102 to 5105, differential data and slope data are generated while performing forward raster scanning, and a histogram of the differential data is created.

The differential data is (pz +P2 +P8) - (pa +ps +pa) when the original image data of the pixel of interest and its 8 neighboring pixels are called as shown in Figure 4.
The main scanning direction differential data given by .

(P2 +Pa +P4) - (P6 +P7 +ps
) is the sum of the sub-scanning direction differential data given by

The tilt data, which indicates the amount of spatial change in the original image data, is the ratio of the above-mentioned main scanning direction differential data and sub-scanning direction differential data, i.e., sub-scanning direction differential data l main scanning direction differential data to 8 The computer 1 shows the direction in which the original image data spatially changes.

These data are associated with each pixel and stored in the image memory 4.
write to.

The histogram is obtained by determining the size and appearance frequency of the differential data. When the computer 1 completes the forward rask scan, the computer 1 sets a threshold value from this histogram in step 5106, binarizes the differential data, and converts the binarized data into individual data. It is stored in association with the pixel.

As a result of binarization in this manner, a region where the original image data changes rapidly spatially, that is, a part edge is detected. It is preferable that these edges be continuous when they indicate the outline or texture of an object existing in the front scene, but they are not necessarily continuous depending on the imaging conditions, etc. Therefore, forward rask scanning is performed again in 5107. When set, processing is performed to connect discontinuous edges below 3108.

In 8108, when an end point pixel of an edge (edge break pixel) that has not yet been subjected to connection processing is detected, the forward rask scan is interrupted, that pixel is set as the pixel of interest, and in 5109, the pixel of interest is selected from its 8 neighboring pixels. A pixel in the direction indicated by the tilt data is selected as an extension candidate pixel, and the determination is made in 5110. Here, the extension candidate pixel and the differential of two pixels on both sides connected to it in a direction orthogonal to the tilt data of the pixel of interest are selected. Compare the data and
If the differential data corresponding to the extension candidate pixel is maximum, it is determined to be a ridge.

If it is determined that the edge is a ridge, the edge is extended to that extension candidate pixel in step 5111. At this time, if the extended pixel becomes an endpoint pixel again, the process returns from 5112 to 5109 and the above is repeated, paying attention to it, but if not;
Since the edge has been connected, the process proceeds to 5115 and restarts the forward rask scan. If it is determined that the pixel is not a ridge, the extension candidate pixel is updated in step 5113 to the pixel on the mountain side of the two pixels on both sides of the extension candidate pixel.

At this time, if the updated extension candidate pixel is in the 8 neighborhood of the pixel of interest, the process returns to 5110 and performs the above determination; however, if it is outside of □, the edge is assumed to be cut and the process returns to 5110.
Proceed to step 5 and restart the forward rask scan.

After completing the above image segmentation for the left image, image segmentation for the right image is performed using exactly the same procedure.

Figures 5a and 5b show the results of image segmentation for a certain left image and right image, respectively. From this, the contours of the pot, melon, cup, and sawyer and the texture of the melon are clearly represented in the line drawings. I can see that I am being ignored.

When the computer 1 finishes dividing one image, it extracts high edge density regions of each image at 84, which will be described with reference to FIG. 3b.

Here, a forward rask scan is set for the left image in step 8201, and a loop consisting of steps S202 to S5213 is executed.

5202, nXn pixels (
In this example, n=21) is set, register Me is cleared in 5203, and 520
4, a forward rask scan within the window is set.

8205 to 5208 are loops for counting edge constituent pixels within the window, and the number is stored in register M6.

3209 to 8211 are all pixels within the window (n
The edge density is determined from the ratio (M e/n x n) of the edge constituent pixels (value of the register Me) to Assuming that the pixel is a pixel in a high density area, "high" (a label indicating a pixel in a high edge density area) is written corresponding to the pixel of interest.

When the extraction of the high edge density region for the left image is completed, the extraction of the high edge density region for the right image is performed in exactly the same procedure. These are the results of extracting high edge density regions for each, and the extracted regions are shown as black pixels. In these figures, an area indicating a melon, which is naturally expected to be a high edge density area from FIGS. 5a and 5b, is extracted, but its outline is not very clear.

When the computer 1 finishes extracting the high edge density regions, it labels the regions of each image in step 85. This will be explained with reference to FIG. 3C.

In step 5301, forward raster scan is set for the left image, and in step 5302, label data is set to an initial value L'' for the region of the --like surface.

If the pixel of interest is an edge constituent pixel and has not been labeled, forward raster scanning is interrupted, registers Mt and Mr are cleared in 5305, and in a loop consisting of 5306 to 5311,
While tracking the edge, label data L is written in association with the pixels that make up the edge, the number of pixels that make up the edge is counted in the register Mr, and the number of pixels included in the high edge density area of the edge is written in the register Mt. Count the number of pixels. In this edge tracking, an edge in a certain direction with respect to the tracking direction is selected every time there is a branch. Therefore, as shown in FIG. 7a, when edges constitute a plurality of closed regions, the periphery of the smallest unit of the closed regions (for example, R1) is tracked. In addition, the count value of the register Mr indicates the number of peripheral pixels of the closed region, and the count value of the register Mt indicates the number of peripheral pixels of the closed region included when images (indicated by hatching) extracted from the high edge density region are superimposed. Each number represents a number.

At 5312, the ratio between the count value of register Mr and the count value of register Mt is determined. This ratio indicates the proportion of traced edges that are included in the high edge density region, and when this ratio is less than a predetermined value Ms" (0.5 in this example), the --like surface is After determining that the area is a texture area and incrementing the label data L by 1 in 5314, forward raster scanning is restarted, but when it exceeds a predetermined value Ms, it is determined that it is a texture area, and the label data L is incremented by 1 in 5314.
15 to 5317, while tracing the edge again, the label data written in association with the pixels constituting the edge is rewritten to t'', and then the forward raster scan is resumed.

As a result of the above processing, the edges of the --like surface area are labeled with labels that identify them, and the edges of the texture area are assigned a common label t'. This is because the texture area has an infinite number of edges as shown in FIGS. 5a and 5b, and it was determined that it would be impractical to label each edge independently. For example, in the case of FIG. 7a, all or most of the edges constituting each small area are included in the high edge density area, so the label data corresponding to all the pixels is t''.

In the loop consisting of 5321 to 5327, the same label as the edge is attached to the region closed by the edge while performing forward raster scanning.

In other words, when the label data of the pixel of interest remains initialized (0), label data attached to consecutive images above, below, left, and right is read in each direction, and when there is common label data in each direction, writes that label data as the label data of the pixel of interest, and if there is no common label data, label data # indicating that it is a background pixel is written.
Write A11 as label data of the pixel of interest. However, in this case, if there is an area closed by another edge within the area closed by an edge, and there is more than one label data common in each direction, the order in which the multiple common label data appear in the raster scan Let the newest one be the label data of the pixel of interest.

For example, when there is an area A2 surrounded by edges with label data L2 within an area A1 surrounded by edges with label data L1, in raster scanning, pixels outside area A1, pixels within area A1 and areas Pixels outside A2,
Pay attention to pixels in area A2, pixels in area A1 but outside area A2, and pixels outside area Ai in that order. When paying attention to pixels outside area A1, there is no common label data, so label data "A'' is written as label data of the pixel of interest, and when paying attention to a pixel within area Ai but outside area A2, common label data L1 is detected, so it is written as label data of the pixel of interest, and area A2
When paying attention to the pixels within, the common label data L1
and L2 are detected, and the label data L2 that appears later is taken as the label data of the pixel of interest.

Processing similar to the above processing for the left image is executed for the right image.

FIGS. 8a and 8b show the results of performing the above processing using the data constituting the aforementioned FIGS. 5a and 6a or FIGS. 5b and 6b, and showing only the texture area. These figures and figures 6a and 6b
Comparing with the figure, it can be seen that the area indicating the melon is clearly extracted. This is shown in Figure 7b,
This is because the boundary is continuous because the high edge density region was extracted based on the edges.

When the computer 1 finishes labeling the area, it returns 86
In this step, the texture regions of each image are integrated and labeled, which will be described with reference to FIG. 3d.

In 5401, forward raster scan is set for the left image, and in 5402, label data is set to an initial value T'' for the texture area.

In the texture area, the common value t0 is written as label data by the above-mentioned labeling, so 540
3 and 5404, the pixels of the texture area are searched for while looking at it. If the pixel is a pixel at the boundary of the area and has not been processed, forward scan is interrupted and S
In the loop consisting of steps 40 to 5409, label data is assigned to each pixel while tracking region boundary pixels.
Rewrite it to .

Finishing this for one texture area leaves 54
Since the label data T is incremented by one in step 10, when the forward raster scan is completed, a label for identifying each texture area is attached corresponding to the boundary pixel of each texture area.

Next, in a loop consisting of steps 8414 to 5419, while forward raster scanning is performed, the labels attached corresponding to the boundary pixels of each texture area are attached to all pixels in that area. Each time a pixel in each texture area that can be distinguished by label data t'' is detected, the label data corresponding to the pixel to the left of the pixel (hereinafter left pixel: peripheral pixel or processed pixel) is read and the label data is read. This is done by rewriting the label data corresponding to the pixel of interest using .

Processing similar to the above is also performed on the right image.

In this way, as a result of executing the processes from 83 to S6, the --like surface area and texture area are divided into units that are recognized as independent areas for each left and right image.

Subsequently, in S7, the computer 1 associates the boundaries of each area of the left image with the boundaries of each area of the right image using feature amounts. This will be explained with reference to FIG. 3e.

In 5501, the following 5502 to 5 for the left image
The forward raster scan process of the loop consisting of 512 is set.

At 5502, the label data corresponding to the pixel of interest is read, and if it is not "A", the pixel of interest is -
Since it belongs to a similar surface area or texture area, 55
04, the pixel counter associated with the label data is incremented by 1, and the original image data of the pixel of interest (
gradation data) is added, and the square of the original image data of the pixel of interest is added to a register that stores the sum-of-squares gradation associated with the label data.

If the pixel of interest at this time is an unprocessed boundary pixel, the forward raster scan is interrupted, and in a loop consisting of 8507 to 5509, the boundary pixel is traced while looking at the area to the left, and the pixel of interest and its four Vector data of the pixel of interest is obtained from the coordinates of the previous pixel (in tracking) by linear approximation, and is written in association with the pixel of interest. When you finish this,
In step 5510, the vector data obtained at that time is referred to, and the boundary line of the area is segmented based on the proximity of the vector data and the continuity of pixels, and the label of each segment, the coordinates of the start point and end point, and the label of the area to which it belongs are placed in a table. Summarize.

After this, restart the forward rask scan and repeat the above to complete the segmentation of the boundaries of each region,
Since the number of constituent pixels and the sum and square sum of the original image data corresponding to the constituent pixels are determined for each region, in step 5513, these data are used to find the average gradation and gradation variance of each region.

The number of constituent pixels, average gradation, and variance of each region are the feature amounts of each region.

In step 5514, segmentation of the boundaries of each area in the right image and feature amount detection of each area are performed in the same manner as above.

, S515, each region of the left image is associated with each region of the right image. At this time, the condition is that the feature amount of the region of the left image and the feature amount of the region of the right image that is a candidate thereof are equal within a permissible range.

In step 8516, segments of the boundaries of mutually correlated regions in the left and right images are correlated.

Here, the start and end points of both segments are on the same epipolar line within a tolerance range, the slope and length of the line segment connecting them are equal within a tolerance range, and the direction in which the area exists with respect to the line segment is equal. The condition is that. In this way, when the border segments of the left and right images are associated, the parallax between the pixels that make up the border segment of the left image and the pixels that make up the border segment of the right image (when the images are overlapped) deviation in the main scanning direction)
is determined and stored in association with the former (pixels forming the segment of the boundary line of the left image).

Further, in S8, the computer 1 correlates the boundaries of each texture area of the left image with the boundaries of each texture area of the right image.

This will be explained with reference to FIG. 3f.

In 5601, the following 5602 to 5 for the left image
The forward rask scan process of the loop consisting of 618 is set.

When a boundary pixel of an unprocessed texture area is detected in steps 8602 to 5605, the forward rask scan is interrupted and the process proceeds to steps 8606 and subsequent steps.

Since the unprocessed boundary pixel has parallax data with the corresponding pixel in the boundary line association using the feature amount described above, it is read in 8606.

Store in register D.

5607, the sum of original image data (hereinafter m m' sum gradation) is calculated and stored in registers E and e□, and in S 6011, 2 is subtracted from the parallax data D (value of register D - the same symbol is used for convenience; the same meaning for the others). The obtained value is stored in register i, and the value obtained by adding 2 to it is stored in register I, respectively.

In 5609, m X m of the pixel of the right image whose parallax with the pixel of interest is i (hereinafter referred to as the parallax 1 pixel of the right image)
'Determine the Japanese gradation and store it in register ai.

In S 610, the value of register eO and register e
The difference in the value of i, that is, the difference between the m x m ' sum gradation of the pixel of interest and the m x m ' sum gradation of the pixel with one parallax, is compared with the value of register E. At this time, if the former is less than the latter, the former is stored in register E and the value of register i is stored in register D in S611.

Thereafter, in step 5612, register i is incremented by 1, and the above process is repeated until the value exceeds the value of register (2).

In other words, in this embodiment, pixels that are highly correlated with each other in the left and right images are on the same epipolar line.

And assuming that the surrounding gradations are approximately equal, 3609
~ 5613, the right pixel (the right pixel with the highest correlation) having the mXm'Xm harmonic closest to the m x m' sum gradation of the pixel of interest is selected as the parallax pixel of the right image and the four pixels before and after it along the main scanning axis. It is selected from pixels.

In 5614, the value of register E, that is, the difference in each mXm/sum gradation between the pixel of interest and the right pixel selected at this time, is used as correlation data, and the value of register D, that is, the parallax of the right pixel, is used as parallax data. It is stored in association with the pixel of interest.

At step 5615, the boundary pixels are updated while looking at the area to the left, and the processing from step 8606 is repeated.

When the correlation based on the above-mentioned correlation is completed for the boundary line of one texture area, the forward rask scan is restarted, and the boundary line correspondence of all texture areas is performed.

FIG. 9a shows the boundary line of the texture area of the left image, and FIG. 9b shows the right image as a result of correlation-based correspondence.
That is, after completing the correspondence between the boundaries of the texture areas of the left and right images in the manner shown in 6 or more indicating the right pixel indicated by the parallax data of the pixels forming the boundary of the left image, next, in S9, the forward direction While performing the rask scan, the pixels inside each texture area are associated with each other based on the parallax data of the left boundary line of each texture area.

First, the concept of the process will be explained with reference to FIGS. 10a, 10b, and 10c.

For example, while performing a forward rask scan of the left image and moving the pixel of interest Po (symbols shown in Fig. 4 apply mutatis mutandis; the same applies to the others), the configuration of the left boundary line B- of a certain texture area Assume that the pixel to the right of the pixel is detected. In this case, the pixel to the left of the pixel of interest Pa (left pixel)
ps is a constituent pixel of the left border line BL, and the pixel immediately above (upper pixel) ps is a pixel that has completed processing, each having parallax data Q or U. In this embodiment, adjacent pixels are Assuming that they have substantially equal parallax data, these parallax data a and U are applied to the pixel of interest Pa to obtain the parallax Ω pixel po' of the right image, the four pixels before and after it along the main scanning axis, and the right image. The above parallax pixel po" and the four pixels before and after it along the main scanning axis are determined, and the correlation with the pixel of interest Po is examined for these.Furthermore, inside this, the same examination is repeated using the processed parallax data. .

This will be explained in more detail with reference to FIG. 3g.

In step 5701, forward rask scan processing is set for the left image, and in steps 8702 to 5705, unprocessed pixels in the texture area are searched for. When this is detected, 8
Proceed to 706 and below.

In 8706, the parallax data of the pixel to the left of the pixel of interest is read and stored in register a, and the parallax data of the upper pixel is read and stored in register U. In 8707, the m x m ' sum gradation of the pixel of interest is calculated. Register E
and eg, and in 870B, parallax data U
The value obtained by subtracting 2 from 2 is stored in register i.

The loop from 8710 to S714 is the above-mentioned S6
The same processing as the loop consisting of 09 to 5613 is performed, and m
m' Right pixel with Japanese gradation (right pixel with the highest correlation)
of.

Select from the right pixel specified by the parallax Q and the four pixels before and after it along the main scanning axis, and store the m x m ' sum gradation difference between the pixel of interest and the pixel to its right in register E, and register the pixel to its right. The parallaxes of are stored in register D, respectively.

Next, in the loop consisting of 5717 to 5721, exactly the same processing is performed using the parallax data U, and thereby, the parallax a pixel of the right image and the 4 pixels before and after it, the pixel on the parallax of the right image, and the 4 pixels before and after it. The right pixel with the highest correlation is selected, and the correlation data is stored in register E, and the parallax data of that pixel is stored in register D. In 5722, these data are associated with the pixel of interest. Remember.

As explained above, this process can be said to propagate the parallax data of the left boundary line to each pixel inside the left boundary line. Therefore, as the pixel of interest moves away from the left border, the uncertainty of its disparity data increases.
In S10, the correspondence is corrected by reverse rask scanning.

Please refer to Figures 11a-11c. For example, while performing reverse rask scan of the left image and moving the pixel of interest Po, the right border B of a certain texture area
It is assumed that the pixel to the left of the constituent pixel No. 8 is detected. This pixel of interest p. has the parallax data D obtained in the above-mentioned initial correspondence process, and the pixel to the right of it (right pixel) ps
is a constituent pixel of the right boundary line BR, so it naturally has parallax data r. Normally, these parallax data should be approximately equal, but the pixel of interest P at this time
When a is far away from the left boundary line BL, it may be different as described above. In that case,
Parallax pixel Pa of the right image and parallax 1 pixel Po of the right image
II and the four pixels before and after it along the main scanning axis are the pixel of interest P.

Consider the correlation with Further inside, a similar study is performed using processed parallax data.

This will be explained in more detail with reference to FIG. 3h.

In 5801, reverse rask scan processing is set for the left image, and in 5802 to 5805, unprocessed pixels in the texture area are searched for. ;88 when detecting
Proceed to 06 and below.

At 8806, correlation data of the pixel of interest is read and stored in register E, and its parallax data is read and stored in register D.
Then, the parallax data of the pixel to the right of the pixel of interest is read and stored in the register r, and in step 5807, the mXm'Xm harmonic calculation of the pixel of interest is performed and stored in the register eQ.

At 8808, the disparity data D of the pixel of interest and the disparity data r of the right pixel are compared, and the difference between them is set to a predetermined value Ps.
If it is less than (2 in this embodiment), the correlation data E and parallax data D of the pixel of interest are determined, but if not, the processing from 1i8809 onwards is performed.

The steps below 8809 are the same as above, and the parallax pixels of the right image, the 1 parallax pixel, and 4 pixels before and after it are calculated using the parallax data r.
The right pixel with the highest correlation with the pixel of interest is selected from among the pixels, and the correlation data and parallax data stored corresponding to the pixel of interest are each corrected.

After completing the processing up to S10, the computer 1
calculates the distance from the installation position of each ITV camera unit to the point on the object corresponding to each pixel using parallax data associated with each pixel. In this regard, please refer again to FIG. 1b.

As a result of imaging the point Q on the same object with the ITV cameras 21 and 31 and performing the above processing on the image,
Assume that it is found that the pixel QL of the left image corresponds to the pixel QR of the right image.

In other words, point Q is a straight line connecting the focal point OL of the ITV camera 21 and the pixel QL of the imaging surface 211, and the point Q
The focus of? 58 and the straight line connecting the pixel QR of the imaging surface 311.

Therefore, if the distance 2a between the optical axes of the ITV cameras 21 and 31 and their focal length f are used, the pixels QL and QR
From the parallax with Z = 2 a f / D ”・
As (1), the distance Z to the point Q on the object is found.

In Stt, distance data to the corresponding point on the object is obtained for each pixel according to equation (1), and in S12, a three-dimensional display of the object is performed on the display 5 using the distance data. Figure 12 shows an example.
The processing result of the same original image as shown in FIG. Referring to this diagram:
The spherical curved surface of the melon is well expressed, and it can be seen that the correspondence result between the left image and the right image according to this example is correct.

To summarize the above, in this embodiment, the ITV camera 2
The following processing is applied to the left image and right image captured in steps 1 and 31.

(1) Perform image segmentation by differentiating each image and extracting edges; (2) Extract regions with high spatial density of edges for each image; (3) Extract areas with low edge density in each image Each region is given an individual label, and regions with high edge density are given a common label. (4) Among the regions with high edge density in each image, a group of regions that can be recognized as independent regions Individually label the region as a texture region; (5) Segment the border of each extraction region in each image, and associate the segments of the left image and the segments of the right image based on the feature amount: (6 ) Correlate the boundary line of the texture area in the left image with the boundary line of the texture area in the right image based on correlation; (7) Use the disparity data of the left boundary line of the texture area in the left image to find the pixels in that area. Obtain parallax data; (8) Correct the parallax data of the pixels in the area using the parallax data of the right border of the texture region in the left image; (9) Calculate the parallax data of each pixel from the camera installation position using the parallax data of each pixel. Find the distance to the point on the object.

Performs 3D display.

That is, in this embodiment, since the image is divided into texture areas and non-texture areas, appropriate processing can be performed on each area. In this division, attention is paid to the fact that elements forming the basis of the texture area are arranged with a certain degree of density within the texture area, so the processing is simple and appropriate.

In addition, when determining the correspondence between the left image and the right image, note that the parallax between adjacent pixels is approximately equal, first correlate the boundaries of the regions, and then use the parallax of the left border to After determining the parallax within, this is corrected based on the parallax of the right border. This can be done easily and accurately even when there are many constituent elements, such as a texture area, and the correspondence based on feature amounts is inverted.

Note that in the above embodiment, two images, left and right, are used, but if more images are to be processed, the same processing as described above may be performed for each.

It should also be noted that techniques such as edge extraction, labeling of pixels within a region surrounded by edges (intra-region propagation of edge labels), and boundary segmentation are merely examples.

〔Effect of the invention〕

As explained above, according to the present invention, it is noted that certain constituent elements exist in a relatively high density in a texture region and have substantially equal characteristics in a limited spatial range, and Since the corresponding data found for the pixels forming the boundary line is expanded inside the boundary line, the processing is extremely simple and highly reliable.

In addition, as shown in the example description, the corresponding data of pixels in the first texture area is obtained by sequentially focusing on the constituent pixels in the first texture area from a certain direction, and then the data is obtained in the opposite direction. If this is corrected by sequentially paying attention to the constituent pixels from the direction, the processing accuracy will be extremely high.

Note that if there are three or more texture regions for which correspondence is sought, any one of them is set as the first texture region, and the others are set as the second texture region, and the corresponding pixels are obtained using exactly the same procedure. I can do it.

[Brief explanation of the drawing]

FIG. 1a is a block diagram showing the configuration of an image processing apparatus implementing the present invention as an example, and FIG. 1b is a schematic diagram showing a model of imaging by the ITV cameras 21 and 31 shown in FIG. 1a. Figures 2, 3a, 3b, 3c, and 3d. 3rd al! 1, 3f, 3g and 3h are flowcharts showing the operation of the computer 1 shown in FIG. 1a. FIG. 4 is a plan view showing the spatial relationship between the pixel of interest and eight neighboring pixels. Figures 5a, 5b, 6a, 6b, 7a,
Figure 7b, Figure 8a, Figure 8b, Figure 9a, Figure 9b,
10a, 10b, 10c, 11a, 11b, 11c, and 12 are plan views showing specific examples of the processing of the computer 1 shown in FIG. 1a. 1: Computer 2.3: ITV camera unit 21.31: ITV camera 22.32: A/D converter 4: Image memory 5: Display 6: Printer 7: Floppy disk 8: Keyboard terminal Figure b Figure a 2 Figure Laa Figure a Figure b Figure '+IJ Figure 8a [' to 1 Figure a [S1 Figure 81'1 Figure b Figure

Claims (4)

[Claims] (1) For the first texture area and the second texture area divided from the original image composed of pixels of minute sections, the feature amount of the boundary line of the first texture area and the boundary of the second texture area By comparing the line feature amounts, correspondence data indicating the correspondence between pixels forming the boundary line of the first texture area and pixels forming the boundary line of the second texture area is obtained from the first texture area. It is determined in association with each of the pixels constituting the boundary line, and from the correlation between the pixels in the first texture area and the pixels in the second texture area indicated by the corresponding data of at least the pixels adjacent to the first texture area. An image processing method that calculates corresponding data of pixels in a texture area, and calculates corresponding pixels of a second texture area based on the corresponding data of each pixel in the first texture area. (2) The correspondence data of the pixels in the first texture area is obtained by sequentially focusing on the constituent pixels in at least the first texture area from a certain direction. Image processing method described. (3) The corresponding data of the pixels in the first texture area is obtained by sequentially focusing on the constituent pixels in at least the first texture area from a certain direction, and then sequentially from the opposite direction. The image processing method according to claim 1, wherein the image processing method is performed by focusing on constituent pixels and correcting them. (4) Claims (1), (2), or (3), wherein the first texture area and the second texture area are each divided from different original images having the same object. Image processing method described in section.