WO2017187950A1 - Image processing device and image processing method - Google Patents

Abstract

The purpose of the present invention is to provide an image processing device and an image processing method which are capable of obtaining parallax stably with high speed. In an image processing device according to one embodiment of the present invention, a geometric region is extracted on the basis of a first parallax calculated using a reduced image, associated with the image (original image) before reduction, and displayed. Accordingly, a user can comprehend the parallax calculation state, and specify a desired region (for instance, a region for which parallax could not be calculated satisfactorily). Furthermore, a second parallax is calculated for a specified processing region in the original image. Accordingly, the amount of processing of the original image can be reduced, and processing can be performed with high speed. Moreover, the second parallax may be calculated using, as the specified processing region, a portion of the image (original image) before reduction.

Description

Image processing apparatus and image processing method

The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method for calculating parallax from an image of a subject.

Conventionally, measurement of a subject (acquisition of two-dimensional information or three-dimensional information) has been performed directly by an inspector using a measuring tool, but in recent years, image measurement has been performed based on an image acquired by an imaging device. ing. In such image measurement, it is known to use parallax calculated from a stereo image for measurement. For example, Patent Document 1 describes that in a road surface state measurement system, stereo images are matched to calculate parallax, and three-dimensional information (such as crack depth and size) of the road surface is measured from the obtained parallax. Has been. Patent Document 2 describes that stereo images are reduced to perform matching, and parallax is acquired from the matching result.

JP 2008-82870 A JP 2013-65247 A

However, in Patent Document 1, since the original image (the image that has not been reduced) is matched, the accuracy is high but the processing time is long. Further, in Patent Document 2, since the reduced image is matched, the processing is fast but the accuracy is low, and when the measurement area is small or exists at the boundary with the adjacent area, the parallax cannot be obtained or the correct parallax is obtained. I couldn't. As described above, the conventional technique cannot acquire the parallax at a high speed and stably with respect to a desired region of the subject.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing apparatus and an image processing method capable of acquiring parallax for a desired region of a subject at high speed and stably.

In order to achieve the above-described object, an image processing apparatus according to the first aspect of the present invention includes an image input unit that inputs a plurality of images obtained by photographing one subject from a plurality of viewpoints, and a plurality of images. Respectively, a reduced image generation unit that generates a plurality of reduced images, a first parallax calculation unit that calculates a first parallax by searching for a corresponding position between the plurality of reduced images, and a first Based on the parallax and the pixel positions of the plurality of reduced images, a geometric region extraction unit that extracts a geometric region in the plurality of reduced images, and a display unit that displays the extracted geometric region in association with the plurality of images are displayed. And detecting a user's instruction input for the plurality of images, and specifying an area designating unit for designating a partial region of the plurality of images as a processing region, and searching for a corresponding position between the plurality of images for the designated processing region. To calculate the second parallax Comprising a second parallax calculating section, the.

In the image processing apparatus according to the first aspect, the geometric region is extracted based on the first parallax calculated using the reduced image and displayed in association with the image before being reduced (original image). By grasping the calculation status, a desired area (for example, an area where the parallax cannot be calculated sufficiently) can be designated. In addition, since the second parallax is calculated for the processing region designated in the original image, the processing amount for the original image can be reduced, and the processing can be performed at high speed. Note that the processing region may be specified for a part of the image before reduction (original image) to calculate the second parallax.

Thus, according to the first aspect, parallax can be acquired at high speed and stably for a desired region of the subject. In the first aspect, the first and second parallaxes can be calculated by various methods such as feature-based matching and region-based matching. In the first aspect, “corresponding position” refers to the same position (corresponding point) captured in a plurality of images. A “geometric area” refers to a region of a subject belonging to the same plane or curved surface, and one or more arbitrary numbers of one subject may exist. In the first aspect, the image input unit may input an image acquired by photographing a subject or an already acquired image.

In the image processing apparatus according to the second aspect, in the first aspect, the area designating unit selects an area designated by the user in a plurality of images or an area selected by the user from candidate areas displayed in the plurality of images. Specify as processing area. In the second aspect, the area designating unit designates the area designated by the user in the plurality of images or the area selected by the user from the candidate areas displayed in the plurality of images as the processing area. Can be easily specified.

The image processing apparatus according to a third aspect further includes a grouping processing unit that groups pixels included in a plurality of images for each area in the second aspect, and the area designating unit sets the grouped area as a candidate area. Display on the display. According to the third aspect, since the pixels are grouped for each area and displayed as candidate areas, it is easy to specify the area.

In any one of the first to third aspects, the image processing device according to the fourth aspect is the first geometric equation, wherein the geometric area extraction unit determines a first geometric equation that is a geometric equation representing the geometric area. A geometric region is extracted based on the geometric equation. The fourth mode prescribes an example of geometric region extraction from a reduced image.

The image processing apparatus according to the fifth aspect is any one of the first to fourth aspects, wherein the geometric region extraction unit extracts pixels whose distance from the geometric region is equal to or less than a threshold as pixels belonging to the geometric region. The fifth aspect shows a criterion for pixel extraction. A pixel whose distance from a certain geometric region exceeds a threshold value is a pixel belonging to a geometric region different from the geometric region. Note that the threshold value can be set in consideration of extraction accuracy requirements and the like.

The image processing device according to a sixth aspect is the first noise removal according to any one of the first to fifth aspects, wherein noise removal is performed on at least one of the calculated first parallax and the extracted geometric region. The unit is further provided. In the sixth aspect, parallax can be calculated accurately and stably by removing noise. In addition, as described above with respect to the first to fifth aspects, the parallax can be calculated by designating the processing area for the area in which the parallax calculation or the geometric area extraction cannot be correctly performed by noise removal.

In any one of the first to sixth aspects, the image processing apparatus according to the seventh aspect further includes a second noise removal unit that removes noise from the calculated second parallax. In the seventh aspect, the parallax can be accurately and stably calculated by removing the noise.

The image processing apparatus according to an eighth aspect is the image processing apparatus according to any one of the first to seventh aspects, wherein the reduced image generation unit converts the plurality of images into a grayscale image, and the image that parallelizes the plurality of images. At least one of the processes is performed, and a plurality of reduced images are generated for the image subjected to the at least one image process. The eighth aspect prescribes the content of so-called “pre-processing”. By generating a reduced image for an image that has undergone such image processing, the amount of parallax calculation processing can be reduced, and high speed and high speed can be achieved. The parallax can be calculated with high accuracy.

In any one of the first to eighth aspects, the image processing apparatus according to the ninth aspect further includes an optical system that acquires a plurality of images, and the image input unit inputs the images acquired via the optical system. To do. The optical system included in the ninth aspect can be a stereo optical system including a plurality of optical systems including a photographic lens and an imaging element corresponding to each of a plurality of viewpoints.

In any one of the first to ninth aspects, the image processing apparatus according to the tenth aspect calculates the two-dimensional information or the three-dimensional information of the measurement target included in the processing region based on the calculated second parallax. And a measuring unit. In the tenth aspect, high-accuracy and stable measurement can be performed based on the parallax calculated by any one of the first to ninth aspects. In each aspect of the present invention, examples of the two-dimensional information and the three-dimensional information may include the position, length, width, and area of the measurement target, but are not limited to these examples.

In an image processing apparatus according to an eleventh aspect, in the tenth aspect, the measurement unit calculates a second geometric equation representing a processing area based on the second parallax, and calculates the second geometric equation and the measurement. Two-dimensional information or three-dimensional information is calculated based on the target pixel position. The eleventh aspect defines a method for calculating two-dimensional information and three-dimensional information.

In the image processing apparatus according to the twelfth aspect, in the tenth or eleventh aspect, the subject is a concrete structure, and the measurement target is damage to the concrete structure. The concrete structure is damaged, and the shape and size of the generated damage change with time. The image processing apparatus according to the twelfth aspect is used to measure the damage of the concrete structure. By applying, two-dimensional information or three-dimensional information of a measurement target (that is, damage) can be calculated at high speed and with high accuracy. Examples of concrete structures include bridges, tunnels, roads, and buildings, and examples of damage include cracks and free lime. Concrete structures to which the twelfth aspect can be applied and Damage is not limited to these examples.

In order to achieve the above-described object, an image processing method according to a thirteenth aspect of the present invention includes an image input step of inputting a plurality of images obtained by photographing one subject from a plurality of viewpoints, and a plurality of images. A reduced image generation step of generating a plurality of reduced images by reducing each of the image, a first parallax calculation step of calculating a first parallax by searching for a corresponding position between the plurality of reduced images, Based on the parallax and the pixel positions of the plurality of reduced images, a geometric region extraction step for extracting a geometric region in the plurality of reduced images, and a display step for displaying the extracted geometric region in association with the plurality of images are displayed. Detecting an instruction input by a user for a plurality of images and designating a part of the plurality of images as a processing region, and searching for a corresponding position between the plurality of images in the designated processing region. And second Comprising a second parallax calculating step of calculating a difference, a. According to the thirteenth aspect, the parallax can be calculated with high accuracy and stability in the same manner as in the first aspect. Note that the thirteenth aspect may further include a configuration similar to the second to twelfth aspects. In addition, a program that causes an image processing apparatus to execute the image processing method according to these aspects, and a non-transitory recording medium on which a computer-readable code of such a program is recorded can also be cited as one aspect of the present invention.

As described above, according to the image processing apparatus and the image processing method of the present invention, parallax can be acquired at high speed and stably for a desired region.

FIG. 1 is a diagram showing a bridge as an example of an application target of the image processing apparatus and the image processing method of the present invention. FIG. 2 is a block diagram showing a configuration of the image processing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram illustrating a functional configuration of the processing unit. FIG. 4 is a diagram illustrating information stored in the storage unit. FIG. 5 is a flowchart showing the procedure of the image processing method according to the embodiment of the present invention. FIG. 6 is a conceptual diagram illustrating a state in which a vertical shift exists between the left and right images. FIG. 7 is a conceptual diagram showing how the left and right images are parallelized. FIG. 8 shows left and right reduced images. FIG. 9 is a diagram illustrating a state where the reduced image is subjected to block matching. FIG. 10 is a diagram illustrating the extracted parallax. FIG. 11 is a diagram illustrating a state in which noise is removed from a parallax image. FIG. 12 is a diagram showing the extracted geometric region. FIG. 13 is a diagram illustrating a state in which noise removal is performed on the extracted geometric region. FIG. 14 is a diagram illustrating a state in which an original image and a geometric area are displayed in association with each other. FIG. 15 is a diagram illustrating a state in which the original image and the geometric region are superimposed and displayed. FIG. 16 is a diagram illustrating a state in which a processing area is specified in the original image. FIG. 17 is another diagram showing a state in which a processing area is specified in the original image. FIG. 18 is still another view showing a state in which a geometric region is designated in the original image. FIG. 19 is still another diagram showing a state in which a geometric region is designated in the original image. FIG. 20 is a diagram illustrating how the parallax is calculated for the processing region. FIG. 21 is a diagram illustrating a state where calculated parallax noise is removed. FIG. 22 is a diagram illustrating how a crack is measured.

Hereinafter, embodiments of an image processing apparatus and an image processing method according to the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a perspective view showing a structure of a bridge 1 (concrete structure) which is an example of an application target of an image processing apparatus and an image processing method according to the present invention. The bridge 1 shown in FIG. 1 has a main girder 3, and the main girder 3 is joined by a joint 3A. The main girder 3 is a member that is passed between the abutment and the pier and supports the load of the vehicle on the floor slab 2. In addition, a floor slab 2 for driving a vehicle or the like is placed on the main girder 3. The floor slab 2 is generally made of reinforced concrete. The bridge 1 has members such as a horizontal girder, a tilted frame, and a horizontal frame (not shown) in addition to the floor slab 2 and the main girder 3.

<Acquisition of image>
When measuring damage to the bridge 1, the inspector uses the digital camera 102 (see FIG. 2) to photograph the bridge 1 from below (direction C in FIG. 1), and includes an image (left image and right image) of the inspection range. Stereo image; multiple images). The photographing is performed while appropriately moving in the extending direction of the bridge 1 (A direction in FIG. 1) and the orthogonal direction (B direction in FIG. 1). If it is difficult for the inspector to move due to the surrounding conditions of the bridge 1, the digital camera 102 may be installed on a movable body that can move along the bridge 1 to perform imaging. Such a moving body may be provided with a lifting mechanism and / or a pan / tilt mechanism of the digital camera 102. Examples of the moving body include a vehicle, a robot, and a flying body, but are not limited to these.

<Configuration of image processing apparatus>
FIG. 2 is a block diagram illustrating a schematic configuration of the image processing apparatus 10 (image processing apparatus) according to the present embodiment. The image processing apparatus 10 includes a digital camera 102 (an image input unit, an optical system), a processing unit 110 (an image input unit, a reduced image generation unit, a first parallax calculation unit, a geometric region extraction unit, a region designation unit, a second designation unit, A parallax calculation unit, a grouping processing unit, a first noise removal unit, a second noise removal unit, a measurement unit, a display unit), a storage unit 120, a display unit 130 (display unit), and an operation unit 140, It is connected so that necessary information can be sent and received.

The function of each unit can be realized by a control device such as a CPU (Central Processing Unit) executing a program stored in a ROM (Read Only Memory) or the like. In this case, a computer-readable code of a program for causing the image processing apparatus to execute the image processing method according to the present invention is recorded in the ROM or the like. The processing unit 110 includes a wireless communication antenna and an input / output interface circuit, and the storage unit 120 includes a non-temporary recording medium such as an HDD (Hard Disk Drive). The display unit 130 includes a display device such as a liquid crystal display, and the operation unit 140 includes an input device such as a keyboard and a mouse. Note that these are examples of the configuration of the image processing apparatus according to the present invention, and other configurations can be adopted as appropriate.

As described above, an image photographed using the digital camera 102 is input to the processing unit 110 by wireless communication and subjected to measurement processing (described later). The digital camera 102 includes a left image optical system 102L for acquiring a left viewpoint image and a right image optical system 102R for acquiring a right viewpoint image, and the same subject (the bridge 1 in the present embodiment) is captured by these optical systems. You can shoot from multiple viewpoints. The left image optical system 102L and the right image optical system 102R include a photographing lens and an image sensor (not shown). Examples of the image sensor include a CCD (Charge-Coupled Device) type image sensor and a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor. An R (red), G (green), or B (blue) color filter is provided on the light receiving surface of the image sensor, and a color image of the subject can be acquired based on the signals of each color.

<Functional configuration of processing unit>
FIG. 3 is a diagram showing a main functional configuration of the processing unit 110. The processing unit 110 includes an image acquisition unit 110A, a reduced image generation unit 110B, a parallax calculation unit 110C, a geometric region extraction unit 110D, a region specification unit 110E, a grouping processing unit 110F, a noise removal unit 110G, a measurement unit 110H, and display control. Part 110I. These functions (each processing of the image processing method) are performed by devices such as a CPU (Central Processing Unit) and various electronic circuits, images and information stored in the storage unit 120, and EEPROM (Electronically Erasable and Programmable Read Only Memory): This is performed by executing a program stored in a ROM or the like while appropriately referring to data stored in a (non-temporary recording medium) or the like. In this case, a computer-readable code of a program for causing the image processing apparatus to execute the image processing method according to the present invention is recorded in the ROM or the like. In processing, a RAM (Random Access Memory) or the like is used as a temporary storage area or a work area. In FIG. 3, these devices are not shown.

The image acquisition unit 110A acquires an image of the bridge 1 by controlling the digital camera 102. The digital camera 102 and the image acquisition unit 110 </ b> A constitute an image input unit in the image processing apparatus 10. The reduced image generation unit 110B reduces the image input via the image acquisition unit 110A and generates a reduced image. The parallax calculation unit 110C (first parallax calculation unit, second parallax calculation unit) calculates the first parallax based on the reduced image and the second parallax based on the original image (image before reduction). The geometric region extraction unit 110D extracts a geometric region in the reduced image based on the first parallax based on the reduced image and the pixel position of the reduced image. The area designation unit 110E detects a user instruction input via the operation unit 140, and designates a processing area based on the detection result. The grouping processing unit 110F groups pixels included in the original image for each region. The noise removing unit 110G (first noise removing unit, second noise removing unit) performs noise removal on the first parallax, the second parallax, and the geometric region. The measurement unit 110H calculates two-dimensional information or three-dimensional information about the subject based on the second parallax. The display control unit 110I (display unit) performs display control on the display unit 130 such as an image, a parallax, a geometric region, and a measurement result.

<Configuration of storage unit>
The storage unit 120 is configured by a non-temporary recording medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), a hard disk (Hard Disk), or various semiconductor memories, and stores the images and information illustrated in FIG. 4 in association with each other. . The crack image 120A is an image obtained by capturing a crack generated in the bridge 1 (for example, the floor slab 2) with the digital camera 102 and inputting it with the image acquisition unit 110A. Note that not only images input by the digital camera 102 and the image acquisition unit 110A but also crack images acquired via a network or a recording medium may be stored. Further, not only an image (original image) captured by the digital camera 102 but also an image (for example, an image after preprocessing or a reduced image) subjected to processing described later may be stored. The first parallax 120B is parallax (first parallax) calculated based on an image (reduced image) generated by reducing an image (original image) captured by the digital camera 102. The second parallax 120C is the parallax calculated for the designated processing region (second parallax). The measurement result 120D is a measurement result (two-dimensional information or three-dimensional information) of the subject (measurement target). The storage unit 120 stores the extracted geometric region information and the result of the grouping process in addition to the image and information described above.

<Configuration of display unit and operation unit>
The display unit 130 includes a display device (not shown) such as a liquid crystal display, and displays input images, images and information stored in the storage unit 120, parallax obtained by the processing unit 110, measurement results, and the like. Can do. The operation unit 140 includes a pointing device such as a mouse and an input device (not shown) such as a keyboard, and the user can operate an image, a button, or the like displayed on the display unit 130 with the operation unit 140.

<Measurement procedure>
Next, measurement of a subject using the image processing apparatus 10 having the above-described configuration will be described. FIG. 5 is a flowchart showing a procedure of measurement processing (image processing method) according to the present embodiment. In addition, this embodiment demonstrates the case where the crack which arose in the floor slab 2 of the bridge 1 which is a concrete structure is measured.

<Image acquisition>
First, the stereo image of the bridge 1 photographed by the digital camera 102 as described above is input to the processing unit 110 (image acquisition unit 110A) by wireless communication (step S100; image input process). A plurality of images of the bridge 1 are input in accordance with the inspection range, and information on the shooting date and time is added to the input image by the digital camera 102. Note that the shooting date and time of the input image does not necessarily have to be the same for all images, and may be for a plurality of days. As the image, a plurality of images may be input at a time, or one image may be input at a time. The image of the bridge 1 may be input not via wireless communication but via a non-temporary recording medium such as various memory cards, or image data that has already been captured and recorded may be input via a network. .

FIG. 6 shows an example of the left image iL0 and the right image iR0 input in this way. FIG. 6 shows an example of an image when a portion where the plate-like member PM is provided in a portion (corner portion) where three planes intersect in the bridge 1 is photographed. The three planes intersect at boundary lines E1, E2, and E3, and these boundary lines E1, E2, and E3 coincide at a point E0. Further, the corners of the plate member PM are set as points E4 and E5. In the left image iL0 and the right image iR0, the horizontal direction in FIG. 6 is the horizontal direction u, and the vertical direction is the vertical direction v. The number of channels and the image size of the left image iL0 and the right image iR0 are not particularly limited. For example, a color image (R, R) of 4,800 pixels (horizontal direction u) × 3,200 pixels (vertical direction v) is used. G, B 3 channels). The left image iL0 and the right image iR0 are images obtained by editing and / or combining a plurality of images (for example, an image showing the entire measurement range generated by combining images obtained by capturing a part of the measurement range). Also good.

<Pretreatment>
In the present embodiment, it is assumed that the left image iL0 and the right image iR0 are color images and are shifted by a distance δ in the vertical direction v as shown in FIG. Accordingly, the processing unit 110 (reduced image generation unit 110B) converts the left image iL0 and the right image iR0 into a grayscale image in step S110 (preprocessing step) prior to generation of a reduced image described later. In step S110, the processing unit 110 shifts the left image iL0 and / or the right image iR0 in the vertical direction v to correct (parallelize) the shift of the distance δ described above. Examples of images obtained by performing these pre-processing (left image iL1 and right image iR1) are shown in FIG. In the image processing apparatus 10 and the image processing method according to the present embodiment, the parallax can be calculated and measured at high speed and stably by performing such preprocessing. In the present embodiment, in addition to the input left image iL0 and right image iR0, the left image iL1 and right image iR1 that have been pre-processed in this way are also referred to as “original images”.

<Reduced image generation>
Next, the processing unit 110 (reduced image generation unit 110B) generates a reduced image based on the left image iL1 and the right image iR1 that are images after preprocessing (step S120; reduced image generation step). The degree of reduction is not particularly limited. For example, the horizontal direction u and the vertical direction v of the left image iL1 and the right image iR1 are respectively reduced to 1/16 to obtain 300 pixels (horizontal direction u) × 200 pixels (vertical direction v ) Can be generated. Let this reduced image be the left image iL2 and the right image iR2 (see FIG. 8).

<Calculation of first parallax>
Next, the processing unit 110 (parallax calculation unit 110C) calculates a first parallax by searching for a corresponding position between the left image iL2 and the right image iR2 that are reduced images (step S130: first parallax). Calculation step). “Corresponding position” refers to the same position (corresponding point) captured in a plurality of images. For example, points E0, E4, and E5 can be used in FIGS. The first parallax can be calculated by block matching (region-based matching) between reduced images, for example, as described below. Region-based matching is a technique for matching a local block image of a reference image and a local block image of a comparative image using a measure (correlation value) of difference or similarity.

Block matching sets a block including a plurality of pixels in one image (reference image) of the left image iL2 and the right image iR2, and sets a block having the same shape and size as the block in the other image (comparison image) Then, the block in the comparative image is moved in the horizontal direction u pixel by pixel, and the correlation value for the two blocks is calculated at each position. In the example of FIG. 9, the reference image is the left image iL2, the comparison image is the right image iR2, and the block AR having the same shape and size as the block AL set in the left image iL2 is moved in the horizontal direction u pixel by pixel. Since the left and right images are parallelized in the preprocessing in step S110, if the position of the block AL is determined, it is only necessary to move the block AR in the horizontal direction u at the time of block matching. When the left and right images are not parallelized in the preprocessing, in the block matching, the movement in the horizontal direction u is repeated while shifting the position in the vertical direction v.

As described above, the correlation value is calculated while moving the block AR, and when the position of the block AR (position corresponding to the block AL) having the highest correlation with respect to the position of the block AL is specified, the target pixel (for example, the block AL) The distance between the center pixel) and the corresponding pixel (for example, the center pixel) of the block AR at the specified position is calculated as the parallax. Then, such processing is executed for all the pixels of the left image iL2, which is the reference image, and parallax is obtained for each pixel position to generate a parallax image. As a method of calculating similarity or dissimilarity in the calculation of parallax, for example, methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared intensity Difference), NCC (Normalized Cross Correlation) can be cited. Among these methods, SAD and SSD calculate the degree of difference between images, and NCC calculates the degree of similarity between images.

An example of the parallax image obtained in this way is shown in FIG. In the parallax image iP1 shown in FIG. 10, the density corresponds to the magnitude of the parallax, the white portion has a small parallax, and the black portion has a large parallax. Note that an area where the parallax cannot be calculated accurately (an area where the parallax value is significantly different from the surrounding area) is shown as a noise area NA. The parallax cannot be accurately calculated for the plate-like member PM described above, and the region R0 where the parallax is calculated is different from the shape of the original member.

In addition, although the case where a corresponding position between images is searched for by calculating a parallax by block matching (region-based matching) has been described here, a corresponding point may be searched for by feature-based matching. In the case of feature-based matching, for example, feature points (edges, corners, etc.) are extracted from the left image iL2 and the right image iR2, and a feature amount is calculated from an area around the feature points to search for (match) corresponding positions. Do. In this case, examples of the feature points include points E0, E4, E5 and boundary lines E1, E2, E3 in FIGS.

<Noise removal>
Next, the processing unit 110 (noise removal unit 110G) performs a process of removing the noise area NA on the first parallax calculated in step S130 (step S140: noise removal step). The removal of the noise area NA can be performed, for example, by performing a low-pass filter process on the parallax image. An example of the parallax image after such noise processing is shown in FIG. In the parallax image iP1A in FIG. 11, the noise area NA that existed in FIG. 10 is lost.

<Extraction of geometric area>
Next, the processing unit 110 (geometric region extraction unit 110D) extracts a geometric region (step S150: geometric region extraction step). The extraction of the geometric region can be performed using, for example, a RANSAC (RANDom Sample Consensus) algorithm. The RANSAC algorithm can obtain an optimum evaluation value by calculating a model parameter (a parameter representing a plane) using randomly sampled data (three if it is a plane) and evaluating the correctness of the calculated parameter. It is an algorithm that repeats until. A specific procedure will be described below.

(Step S1)
Three points are randomly extracted from the parallax image after noise removal. For example, points f1 (u ₁ , v ₁ , w ₁ ), f2 (u ₂ , v ₂ , w ₂ ), and f3 (u ₃ , v ₃ , w ₃ ) are extracted from the parallax image iP1A in FIG. Shall. The points to be extracted here are points for determining the geometric equation of each geometric region, and the number of points to be extracted is changed depending on the assumed geometric region type (plane, cylindrical surface, spherical surface, etc.). Good. For example, in the case of a plane, representative points of 3 points or more (assuming that they are not on the same straight line) are extracted. The horizontal coordinate of the image is represented by u _i , the vertical coordinate is represented by v _i , and the parallax (distance direction) is represented by w _i (i is an integer of 1 or more representing a point number).

(Step S2)
Next, a plane equation (first geometric equation) is determined from the extracted points f1, f2, and f3. The plane equation F in the three-dimensional space (u, v, w) is generally expressed by the following (Expression 1) (a, b, c, d are constants).

F = a * u + b * v + c * w + d (Formula 1)
(Step S3)
For all the pixels (u _i , v _i , w _i ) of the parallax image, the distance to the plane represented by the plane equation F in (Expression 1) is calculated. If the distance is less than or equal to the threshold value, it is determined that the pixel belongs to the plane represented by the plane equation F.

(Step S4)
If the number of pixels existing on the plane represented by the plane equation F is larger than the number of pixels for the current optimal solution, the plane equation F is determined as the optimal solution.

(Step S5)
Steps S1 to S4 are repeated a predetermined number of times.

(Step S6)
One plane is determined by using the obtained plane equation as a solution.

(Step S7)
The pixels on the plane determined up to step S6 are excluded from the processing target (plane extraction target).

(Step S8)
Steps S1 to S7 are repeated, and the process ends when the number of extracted planes exceeds a certain number or the number of remaining pixels is less than a specified number.

FIG. 12 shows an example of the geometric region extracted by the above procedure. In the example of FIG. 12, three geometric regions (planes) G1, G2, and G3 are extracted from the image iG, but the region of the plate-like member PM (region R0 in FIG. 11) is correctly extracted. Not a noise (region N1 and region N2).

<Noise reduction for geometric region extraction results>
Next, the processing unit 110 (noise removal unit 110G) performs noise removal on the extracted geometric region (step S152: noise removal step). Noise removal from the geometric region extraction result can be performed, for example, by the following expansion / contraction process.

<Expansion and shrinkage treatment>
The expansion / contraction process can be performed on the binarized image (monochrome image). In this case, if there is even a white pixel around the pixel of interest, the process of replacing the pixel of interest with white is called “dilation”. If there is even a black pixel around the pixel of interest, the pixel of interest is black. The process of replacing with is called “Erosion”. Then, small pattern noise (region N1 in the example of FIG. 12) and linear pattern noise (region N2 in the example of FIG. 12) are removed by appropriately repeating contraction and expansion. For example, small pattern noise can be removed by expanding the image after contracting, and linear pattern noise can be removed by contracting after expanding the image. Such expansion and contraction can be repeated, and the process of expanding and contracting the same number of times is called closing, and the process of contracting and expanding the same number of times is called opening.

The expansion / contraction process can be performed not only on a binarized image but also on a grayscale image. In this case, the above-described “expansion” replaces the luminance value of the target pixel with the maximum luminance value near the target pixel, and “shrink” replaces the luminance value of the target pixel with the minimum luminance value near the target pixel.

FIG. 13 shows an example of the geometric region after removing the noise by the above-described expansion / contraction process. FIG. 13 shows an image iG2 from which the regions N1 and N2 in FIG. 12 have been removed as noise.

As described above, since the image is reduced in the first parallax calculation, and noise is removed from the parallax image and the geometric area, the parallax is reduced at the boundary between small members and planes like the plate member PM. Calculation and area extraction may not be performed correctly. However, in this case, the image processing apparatus 10 and the image processing method according to the present embodiment specify the desired region based on the region extraction result as described below, so that the second can be performed at high speed and stably. The parallax can be calculated and measurement can be performed based on the second parallax. In the above-described example, noise removal is performed on both the parallax image and the geometric region. However, depending on conditions such as the amount of noise and the region where the noise is generated, noise removal is performed on only one of them. May be performed.

Here, the geometric equation (plane equation) when the geometric region is a plane has been described, but a geometric equation representing another type of geometric region, such as a cylindrical surface (cylindrical surface) or a spherical surface, is determined according to the shape of the subject. May be. This is because the shape of a structure such as a bridge pier or a tunnel is often expressed not only by a plane but also by a cylindrical surface or a spherical surface. Specifically, a cylinder whose central axis is the z-axis and whose radius is r is expressed by the following (formula 2) (z is an arbitrary value), and a sphere whose center is the origin of the coordinate system and whose radius is r is (Equation 3).

x ² + y ² = r ² (Formula 2)
x ² + y ² + z ² = r ² (Formula 3)
<Display of original image and geometric area>
When the geometric region is extracted by the processing up to step S150, the processing unit 110 (display control unit 110I) displays the original image and the geometric region in association with the display unit 130 (step S160: display step). For example, as shown in FIG. 14, the left image iL1 and the corresponding image iG2 are displayed in parallel. The right image iR1 may be displayed in addition to or instead of the left image iL1. FIG. 15 is a diagram showing another example of the display of the original image and the geometric area. FIG. 15 shows an image iLG (the left image iL1 is displayed on the front) in which the left image iL1 and the image iG2 are superimposed. In the superimposed display, one image may be made translucent so that the other image can be seen through, or the image displayed on the front side can be switched. In FIG. 15, the plate-like member PM is shown in white because no region is extracted.

As described above, in the image processing apparatus 10 and the image processing method according to the present embodiment, the original image and the geometric region are displayed in association with each other, and these images are compared, so that the region extraction result such as the presence of an unextracted region is obtained. Can be easily grasped. In addition to the geometric region, a parallax image (for example, the parallax image iP1A in FIG. 11) may be displayed.

<Processing area specification>
Next, the processing unit 110 (region specifying unit 110E) detects a user instruction input and specifies a processing region for a part of the original image (step S170: region specifying step). FIG. 16 is a diagram showing an example of processing area designation. In the example of FIG. 16, the user operates a device such as a mouse provided in the operation unit 140 to fill the region R1 of the plate member PM in the left image iL1, and the region specifying unit 110E detects the region R1 painted by the user. And specify it as a processing area. FIG. 17 is a diagram showing another example of processing area designation. In the example of FIG. 17, the user operates the device provided in the operation unit 140 in the left image iL1 to specify the region R2 surrounding the plate member PM, and the region specifying unit 110E detects the region R2 surrounded by the user. To specify as a processing area. As a processing area, a desired area such as an area that has disappeared at the time of area extraction or an area in which parallax is not correctly calculated, or an area in which a crack to be measured exists can be specified.

The processing area is specified by the processing section 110 (grouping processing section 110F) grouping the original images for each area and displaying them as candidate areas, and the area specifying section 110E detects the area selected by the user. You may go. In this case, the grouping can be performed, for example, by applying a watershed algorithm or edge detection to the original image. FIG. 18 is an example of an image iG3 showing a grouped region, and each region is displayed in a different color (however, since the color is difficult to show, the color is displayed in characters in FIG. 18). In FIG. 18, when the user specifies the yellow region R3, the processing unit 110 (region specifying unit 110E) detects the user's specification, and specifies the region R3 as the processing region.

In the above-described processing area designation example, the processing area is designated by detecting the area designation or area selection instruction input by the user. However, the processing area designation is designated by detecting the measurement point instruction input by the user. Also good. For example, when the user designates the measurement point T3 via the operation unit 140 as shown in FIG. 19, the processing unit 110 (region designation unit 110E) detects this designation and designates the region R6 surrounding the measurement point T3 as the processing region. To do. Such measurement points and areas are not limited to one, and a plurality of measurement points and areas may be designated. For example, when detecting the length of a crack, the user designates the start point and end point of the crack as the measurement point T3 described above, and the area designation unit 110E detects the designation of those points, and each of the start point and end point is detected. Alternatively, a processing area may be specified. When performing such area designation, the size of the area R6 is not particularly limited. For example, 480 pixels (horizontal direction u) × 320 pixels (vertical direction v) or 300 pixels (horizontal direction u) × 200 pixels ( It can be the vertical direction v).

Note that the above-described operations for filling, enclosing, and measuring point specification can be performed via operation buttons and tool boxes displayed on the display unit 130 by the processing unit 110 (region specifying unit 110E and display control unit 110I). .

<Second parallax calculation>
When the processing region is designated in step S170, the processing unit 110 (parallax calculation unit 110C) calculates a second parallax from the original image for the designated processing region (step S180: second parallax calculation step). Similar to the first parallax in step S130, the second parallax calculated in step S180 can be calculated from block matching of the left image iL1 and the right image iR1 and the distance between feature points. A parallax image iP2 indicating the second parallax calculated in this way is shown in FIG. FIG. 20 shows a state including a noise region NA in addition to the region R4 corresponding to the plate member PM, and noise is removed by the noise removing unit 110G (step S190: noise removing step) as shown in FIG. A parallax image iP3 is obtained. Noise removal in step S190 can be performed by, for example, a low-pass filter process as in step S140.

As described above, in the present embodiment, the second parallax is calculated for the processing region specified for a part of the original image, so that the parallax can be calculated at high speed without targeting the entire image, and one pixel. It is possible to avoid the problem that the parallax cannot be calculated for a small area due to the parallax calculation failure due to acquiring only the parallax or the image reduction and noise removal.

<Calculation of 2D information or 3D information>
When the second parallax is calculated for the desired processing region by the above-described processing, the processing unit 110 (measurement unit 110H) performs two-dimensional information or three-dimensional information on the measurement target included in the processing region based on the second parallax. Is calculated (step S200: measurement step). Examples of two-dimensional information or three-dimensional information can include position, length, width, and area, but items to be calculated are not limited to these examples, and depend on the nature of the subject and the measurement target. Other items such as volume and cross-sectional area may be calculated.

<Extraction of cracks>
In this embodiment, since it is assumed that damage (crack) of the bridge 1 is measured, the processing unit 110 (measurement unit 110H) first extracts a crack from an image (for example, the left image iL1 or the right image iR1). Crack extraction can be performed by various methods. For example, a crack detection method described in Japanese Patent No. 4006007 can be used. This method calculates wavelet coefficients corresponding to the two concentrations to be compared, calculates each wavelet coefficient when each of these two concentrations is changed, creates a wavelet coefficient table, and detects cracks. A wavelet image is created by wavelet transforming the input image of the target concrete surface, and in the wavelet coefficient table, the wavelet coefficients corresponding to the average density of neighboring pixels in the local area and the density of the target pixel are calculated. This is a crack detection method comprising a step of determining a crack area and a non-crack area by comparing a wavelet coefficient of a target pixel with a threshold value as a threshold value.

Crack extraction can also be performed using the method described in Non-Patent Document 1 below. In the method described in Non-Patent Document 1, an area composed of pixels having a luminance value less than the threshold value is set as a percolated area (percolation area), and the threshold value is sequentially updated according to the shape of the percolation area. The crack is detected from the surface image. Note that the percolation method is a method in which the region is sequentially enlarged in general imitating water penetration (percolation) in nature.

[Non-Patent Document 1] Tomoyuki Yamaguchi, “A Study on Image Processing Method for Crack Inspection of Real Concreate Surfaces”, MAJOR IN PURE AND APPLIED PHYSICS, GRADUATE SCHOOL OF SCIENCE AND ENGINEERING, WASEDA UNIVERSITY, February 2008 In the above description, the crack is extracted after the geometric equation is determined. However, the crack extraction is performed in parallel with the processing from the preprocessing (step S110) to the noise removal (step S190) or prior to these processing. You may go. FIG. 22 is a diagram illustrating a state in which cracks Cr are extracted in the region R5 of the plate-like member PM. The crack Cr is a crack from the end point T1 to the end point T2. Instead of extracting cracks by image processing as described above and obtaining the length, as described in relation to FIG. The designation of the end point of the crack (for example, clicking on the end point with the mouse) may be received by inputting an instruction, and the length may be calculated based on the designated end point.

<Measurement of cracks>
When calculating the length of the crack Cr, the parallax (corresponding to the w coordinate) at the end points T1 and T2 is obtained because the parallax (second parallax) is calculated for the region R5. Then, if necessary, the (u, v, w) coordinates of the end points T1, T2 are converted into (x, y, z) coordinates in the real space based on the position and shooting direction of the digital camera 102, and the following (formula 4) ) To obtain the length L (distance between end points T1 and T2). If the crack Cr is a curved crack, the crack Cr is divided into a plurality of sections so that each section can be regarded as a straight line, and the length of each section is calculated to obtain the length of the crack Cr. Can do.

L = {(x ₁ −x ₂ ) ² + (y ₁ −y ₂ ) ² + (z ₁ −z ₂ ) ² } ^1/2 (Formula 4)
However, the coordinates in the real space of the end points T1 and T2 are (x ₁ , y ₁ , z ₁ ) and (x ₂ , y ₂ , z ₂ ), respectively.

<Measurement of crack based on plane equation and pixel position>
The length of the crack Cr can be estimated using a plane equation in addition to directly calculating from the parallax calculated for the region R5 (second parallax) as described above. Specifically, parallax is calculated based on block matching, distance between feature points, and the like between the left image iL1 and the right image iR1 with respect to three or more representative points set in the region R5, and the region based on the calculated parallax. Estimate the plane equation (second geometric equation) of R5. At this time, the representative point extraction and the plane equation estimation may be repeated by the RANSAC algorithm described above. Then, the parallax (corresponding to the w coordinate) of the end points T1 and T2 can be obtained from the pixel positions (u and v coordinates) of the end points T1 and T2 and the plane equation of the region R5. When the parallax of the end points T1 and T2 is calculated, as described above, the (u, v, w) coordinates of the end points T1 and T2 are converted into (x, y, z) coordinates in the real space, The length L (distance between the end points T1 and T2) can be obtained.

As described above, in the image processing apparatus 10 and the image processing method according to the present embodiment, the geometric region extracted based on the first parallax and the original image are displayed in association with each other, so that the region extraction result can be easily grasped. It is possible to calculate the second parallax in the processing area designated for a part of the original image even for the area where the geometric area is not correctly extracted (the area where the first parallax cannot be calculated correctly). it can. As a result, it is possible to measure parallax at a high speed and stably for a desired region.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the image processing apparatus and the image processing method of the present invention can calculate and measure parallax for various subjects other than cracks in a concrete structure. For example, the parallax can be calculated for the structure itself, not the cracks, obstacles, or cracks on the road, and the shape and size can be measured.

DESCRIPTION OF SYMBOLS 1 Bridge 2 Floor slab 3 Main girder 3A Joint part 10 Image processing apparatus 102 Digital camera 102L Left image optical system 102R Right image optical system 110 Processing part 110A Image acquisition part 110B Reduced image generation part 110C Parallax calculation part 110D Geometric area extraction Unit 110E area designation unit 110F grouping processing unit 110G noise removal unit 110H measurement unit 110I display control unit 120 storage unit 120A image 120B first parallax 120C second parallax 120D measurement result 130 display unit 140 operation unit AL block AR block E1 Boundary line E2 Boundary line E3 Boundary line N1 Region N2 Region NA Noise region PM Plate member R0 Region R1 Region R2 Region R3 Region R4 Region R5 Region R6 Region S1 to S8 Steps S100 to S200 of geometric region extraction Processing Steps T1 End Point T2 End Point T3 Measurement Point iG Image iG2 Image iG3 Image iL0 Left Image iL1 Left Image iL2 Left Image iLG Image iP1 Parallax Image iP1A Parallax Image iP2 Parallax Image iP3 Parallax Image iR0 Right Image iR1 Right Image iR2 Right Image u Horizontal v Vertical δ Distance

Claims (13)

1. An image input unit for inputting a plurality of images obtained by photographing one subject from a plurality of viewpoints;
   A reduced image generator that reduces the plurality of images to generate a plurality of reduced images;
   A first parallax calculation unit that calculates a first parallax by searching for a corresponding position between the plurality of reduced images;
   A geometric region extraction unit that extracts a geometric region in the plurality of reduced images based on the first parallax and the pixel positions of the plurality of reduced images;
   A display unit that displays the extracted geometric region in association with the plurality of images;
   An area designating unit for detecting a user instruction input for the plurality of displayed images and designating a partial area of the plurality of images as a processing area;
   A second parallax calculating unit that calculates a second parallax by searching for a corresponding position between the plurality of images for the designated processing region;
   An image processing apparatus comprising:
2. The area designating unit designates, as the processing area, an area designated by the user in the plurality of images or an area selected by the user from candidate areas displayed in the plurality of images. Image processing apparatus.
3. The grouping process part which groups the pixel contained in the said several image for every area | region, The said area | region designation | designated part displays the said grouped area | region on the said display part as said candidate area | region. Image processing device.
4. The geometric region extraction unit determines a first geometric equation that is a geometric equation representing the geometric region, and extracts the geometric region based on the determined first geometric equation. The image processing apparatus according to item.
5. The image processing apparatus according to any one of claims 1 to 4, wherein the geometric region extraction unit extracts pixels whose distance from the geometric region is equal to or less than a threshold as pixels belonging to the geometric region.
6. The image processing apparatus according to claim 1, further comprising a first noise removing unit that removes noise from at least one of the calculated first parallax and the extracted geometric region.
7. The image processing apparatus according to any one of claims 1 to 6, further comprising a second noise removing unit that removes noise from the calculated second parallax.
8. The reduced image generation unit performs at least one of image processing for converting the plurality of images into a grayscale image and image processing for parallelizing the plurality of images, and performs the at least one image processing. The image processing apparatus according to claim 1, wherein the plurality of reduced images are generated for an image.
9. The image processing apparatus according to claim 1, further comprising an optical system that acquires the plurality of images, wherein the image input unit inputs an image acquired via the optical system.
10. The image processing according to any one of claims 1 to 9, further comprising a measurement unit that calculates two-dimensional information or three-dimensional information of a measurement target included in the processing region based on the calculated second parallax. apparatus.
11. The measurement unit calculates a second geometric equation that is a geometric equation representing the processing region based on the second parallax, and the two-dimensional based on the second geometric equation and the pixel position of the measurement target. The image processing apparatus according to claim 10, wherein the information or the three-dimensional information is calculated.
12. 12. The image processing apparatus according to claim 10, wherein the subject is a concrete structure, and the measurement target is damage to the concrete structure.
13. An image input step of inputting a plurality of images obtained by photographing one subject from a plurality of viewpoints;
    A reduced image generating step of generating a plurality of reduced images by reducing the plurality of images respectively;
    A first parallax calculating step of searching for a corresponding position between the plurality of reduced images and calculating a first parallax;
    A geometric region extraction step of extracting a geometric region in the plurality of reduced images based on the first parallax and the pixel positions of the plurality of reduced images;
    A display step of displaying the extracted geometric region in association with the plurality of images;
    A region designation step of detecting a user instruction input to the displayed plurality of images and designating a partial region of the plurality of images as a processing region;
    A second parallax calculating step of calculating a second parallax by searching for a corresponding position between the plurality of images for the designated processing region;
    An image processing method comprising:

Images