JP7319527B2 - Image processing system - Google Patents

Description

An image processing system that processes images of a target taken from multiple different viewpoints

In some cases, a building that has been built for several years or several tens of years has been remodeled, expanded, or updated. At this time, it is necessary to obtain a design drawing, but there is no design drawing at present, so it is often made from a drawing to create a design drawing. In the case of measuring a building, the cost is high because the measurement is performed by a professional engineer who spends a lot of manpower and time.

In order to save manpower and time in building measurement, for example, Non-Patent Document 1 (Luwei Yang et al., "Polarimetric Dense Monocular SLAM", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.) is described. It is conceivable to use such a SLAM (Simultaneously Localization and Mapping) method.

However, since there are objects such as metal members in buildings that specularly reflect light, if a method for acquiring distance information based on stereoscopic vision is used, specular reflections of these objects will cause errors in stereoscopic processing. It will be.

Image processing systems that process images of an object taken from different viewpoints, such as systems that use stereoscopic methods to obtain distance information, need to reduce errors due to specular reflections in processing the images of the object. There is a problem of

An image processing system according to a first aspect includes an image acquisition section, a recognition section, and a detection section. The image acquisition unit acquires images of a target photographed from a plurality of different viewpoints. The recognition unit recognizes specular reflection components in the first image and the second image that are the acquired images. The detection unit uses the recognition result of the recognition unit to detect the difference between the first image and the second image based on the difference in viewpoint.

The image processing system of the first aspect can detect the difference between the first image and the second image based on the difference in viewpoint in the detection unit while using the specular reflection component recognized by the recognition unit. As a result, the image processing system is enabled to reduce errors due to specular reflections in processing images of objects.

The image processing system of the second aspect is the system of the first aspect, wherein the detection unit corrects the image, masks the image, or removes part of the image based on the specular reflection component recognized by the recognition unit. and detect the difference between the first image and the second image based on the difference in viewpoint.

In the image processing system of the second aspect, the detection unit corrects the image, masks the image, or removes part of the image based on the specular reflection component, thereby producing the first image and the second image based on the difference in viewpoint. It is possible to reduce the effect of the specular reflection component on the error due to the specular reflection component when detecting the difference in .

The image processing system of the third aspect is the system of the first aspect or the second aspect, wherein the image acquisition unit acquires an image captured by a polarization camera having a polarizing plate at each viewpoint, and the recognition unit Based on the polarization angle and light intensity, the specular reflection component is recognized.

The image processing system of the third aspect can easily reduce the specular reflection component by capturing an image through a polarizing plate at each viewpoint and recognizing the specular reflection component from the polarization angle and light intensity at each viewpoint. can.

An image processing system according to a fourth aspect is the system according to any one of the first aspect to the third aspect, wherein the image acquisition unit acquires an image captured by a light field camera and the illuminance of a light source illuminating an object at each viewpoint. A plurality of images photographed by changing the shutter speed, a plurality of images photographed by changing the shutter speed, or a plurality of images photographed by changing the color of the light source illuminating the object are obtained.

The image processing system of the fourth aspect can make specular reflection components easier to recognize by using a light field camera, changing the illuminance of the light source, changing the shutter speed, or changing the color of the light source.

An image processing system according to a fifth aspect is the system according to any one of the first aspect to the fourth aspect, wherein the image acquisition unit is an image of an object captured by the same imaging device at different locations, or installed at a different location. to obtain images of the object photographed by the plurality of photographing devices.

In the image processing system of the fifth aspect, the number of photographing devices can be reduced when photographing with the same photographing device, and the positions of the photographing devices can be specified when photographing with a plurality of photographing devices installed at different locations. become easier.

An image processing system according to a sixth aspect is the system according to any one of the first aspect to the fifth aspect, and based on the detection result of the detection unit, the geometric information of the object, the optical information of the object, or , an information generator that generates information of the light source illuminating the object.

The image processing system of the sixth aspect enriches the information with respect to the geometrical information of the object, the optical information of the object, or the information of the light source that illuminates the object, by using the first and second images from different viewpoints. can do.

1 is a schematic diagram showing an example of the configuration of an image processing system according to an embodiment; FIG. FIG. 4 is a pictorial diagram for explaining a photographed image of a cup photographed from a first position; FIG. FIG. 11 is a pictorial diagram for explaining a photographed image of a cup photographed from a second position; FIG. FIG. 2B is a pictorial diagram for explaining a first modified image obtained by removing a specular reflection component from the photographed image of FIG. 2A; FIG. 2B is a pictorial diagram for explaining a second corrected image obtained by removing a specular reflection component from the photographed image of FIG. 2B; The figure which shows an example of the photography image of the ceiling space. The figure which shows the other example of the photography image of the space behind the ceiling. A graph showing the relationship between the polarization angle and the degree of polarization. Schematic diagram for explaining the mechanism of a polarization camera. Schematic diagram for explaining the mechanism of a polarization camera. Graph showing the relationship between the polarization angle φ and the light intensity _Iφ . FIG. 4 is a diagram showing an example of an image before performing specular reflection removal; FIG. 10 is a diagram showing an example of an image after performing specular reflection removal; FIG. 4 is a diagram showing an example of an image for explaining a detected specular reflection component; FIG. 10 is a diagram showing another example of an image before performing specular reflection removal; FIG. 10 is a diagram showing another example of an image after specular reflection removal; FIG. 10 is a diagram showing another example of an image for explaining the detected specular reflection component; FIG. 4 is a diagram for explaining feature point correspondence before specular reflection removal; FIG. 4 is a diagram for explaining feature point correspondence after specular reflection removal; 1 is a schematic diagram showing an example of the configuration of an image processing system according to an embodiment; FIG.

(1) Overall Configuration The image processing system 1 includes an image acquisition section 10 , a recognition section 20 , a detection section 30 and an information generation section 40 . The image processing system 1 is connected to a mobile camera 50 . The image acquisition unit 10 acquires target images photographed from a plurality of different viewpoints. The image processing system 1 acquires, from the mobile camera 50, a first image 201 and a second image 202 of the cup 100 shown in FIGS. 2A and 2B, for example. In this case, the object to be photographed is the cup 100 . Here, a case where the object to be photographed is the glass 100 will be described, but the object to be photographed is not limited to objects. The object to be photographed may be, for example, a scene (spectacle).

The mobile camera 50 is a camera mounted on a moving object that is operated by remote control. FIG. 1 shows a case where the mobile camera 50 runs on caterpillars, but the running means may be wheels or legs other than caterpillars. The mobile camera 50 changes the relative positional relationship with the glass 100 and takes an image. The mobile camera 50 can change its relative positional relationship with the glass 100 by, for example, self-propelled around the glass 100 . Here, a case where the photographing device is the mobile camera 50 is described, but the method of moving one camera is not limited to self-propelled. For example, it may be a portable camera that is held by a person and moved, or a camera that is attached to a robot and moves with the robot. As for the transmission and reception of data between the mobile camera 50 as a photographing device and the image processing system 1, FIG. may transmit and receive data by wire, or may transmit and receive data via a public line. Furthermore, although FIG. 1 shows the image processing system 1 being external to the mobile camera 50 , the image processing system 1 may be incorporated within the mobile camera 50 .

An image captured by the mobile camera 50 at the first position is the first image 201 , and an image captured at the second position is the second image 202 . The mobile camera 50 can adopt a configuration having a function of transmitting the coordinates (x1, y1) of the first position and the coordinates (x2, y2) of the second position to the image processing system 1 . On the other hand, the image processing system 1 receives the coordinates (x1, y1) of the first position and the coordinates (x2, y2) of the second position, and uses such position data for image processing. can be done. The image processing system 1 processes image data captured from a plurality of different viewpoints such as the first image 201 and the second image 202, the coordinates of the first position (x1, y1) and the coordinates of the second position (x2, y2 ) and the viewpoint position data such as ) to create a drawing of an object such as the cup 100 . As a method for the mobile camera 50 to measure its own position, GPS (Global Positioning System) is used here, but the method for measuring the photographing position is not limited to GPS. A programming method or a SLAM technique that performs self-localization (camera position) and target position simultaneously may be used.

The recognition unit 20 recognizes specular reflection components in the acquired first image 201 and second image 202 . For example, in the first image 201 and the second image 202 of FIGS. 2A and 2B, the specularly reflected portions are the first portion 211, the second portion 212, the third portion 213, and the fourth portion 214. FIG. The recognition unit 20 recognizes these first to fourth locations 211 to 214 .

The detection unit 30 uses the recognition result of the recognition unit 20 to detect the difference between the first image 201 and the second image 202 based on the difference in viewpoint. The difference between the first image 201 and the second image 202 based on the difference in viewpoint is that, for example, the first image 201 is captured with a part of the character "A" missing, while the second image 202 is captured with the character " A” is all captured. Another difference is, for example, that the orientation of the handle 110 of the cup 100 differs between the first image 201 and the second image 202 . The image processing system 1 can write, for example, the letter “A” and the handle 110 on the drawing of the cup 100 by image processing based on these differences.

The detection unit 30 uses the recognition result of the recognition unit 20, for example, to remove or reduce the specular reflection component. Alternatively, the detection unit 30 uses the recognition result of the recognition unit 20, for example, to emphasize components other than specular reflection components or extract only components other than specular reflection components. By removing the specular component, the image processing system 1 can obtain data representing, for example, the first modified image 203 and the second modified image 204 shown in FIGS. 3A and 3B.

Comparing the first point 211 and the third point 213, which are specularly reflected points, the first point 211 is positioned on the character "C", while the third point 213 is positioned on the character "B". do. In this way, the location of specular reflection changes as the positional relationship between the viewpoint, the light source, and the object changes. Similarly, the second location 212 and the fourth location 214 related to reflection on the handle 110 also move along with the movement of the viewpoint. If the position of specular reflection differs depending on the position of the viewpoint, for example, when a drawing of the cup 100 is created, an error will occur in the drawing. On the other hand, if a drawing of, for example, the cup 100 is created using the data representing the first corrected image 203 and the second corrected image 204 shown in FIGS. can be removed. In this way, the detection unit 30 uses the recognition result of the recognition unit 20 to detect the difference between the first image 201 and the second image 202 based on the difference in viewpoint, so that the image processing system 1 can process the target image. It is possible to reduce errors due to specular reflection in

The information generation unit 40 generates geometric information of the object, optical information of the object, or information of the light source that illuminates the object based on the detection result of the detection unit 30 . Object geometric information is, for example, information about the shape of an object or a 3D mapping of a scene. The optical information of the object is, for example, information about the reflective properties of the object. The information about the light source that illuminates the object is, for example, information about at least one of the direction and position of the light source.

The image processing system 1 is realized by, for example, a computer. The image processing system 1 is composed of, for example, a control arithmetic device (not shown) and a storage device (not shown). A processor, such as a CPU or a GPU, can be used for the control computing unit. The control arithmetic device reads a program stored in the storage device and performs predetermined image processing and arithmetic processing according to the program. Furthermore, the control arithmetic unit can write the arithmetic result to the storage device and read the information stored in the storage device according to the program. FIG. 1 shows various functional blocks implemented by the control arithmetic unit. A storage device can be used as a database.

(2) Detailed configuration (2-1) Regarding measurement in the ceiling When renovating, expanding, or updating equipment, the size of parts that can be brought in, and whether other equipment will get in the way during construction For the purpose of knowing things in advance, we sometimes make from drawings to make blueprints. The measurement of the building performed to make such a drawing will be explained.

In particular, in the space above the ceiling of a building, for example, a case of replacing a refrigerant pipe of an air conditioner can be considered. In many cases, it is necessary to measure the building during such construction work in the attic space. However, there is a problem that the measurement of the space above the ceiling is expensive because it is performed by a professional engineer, taking a lot of manpower and time.

As a method for solving such a problem, a method of performing measurement using a sensor using a robot or the like is conceivable, but it is difficult to perform measurement using a sensor in the space above the ceiling. For example, in the case of optical measurement, metals such as pipes 310 and 320 reflect light as shown in FIGS. Shape measurement is difficult. Moreover, in the case of measurement using radio waves, there is a problem that reinforcing bars and concrete block radio waves.

This disclosure presents an optical metrology technique that solves problems such as metal reflections and works in environments such as ceiling spaces. Specifically, we calculate the component of specular reflection using a polarization camera, generate an image with the specular reflection removed, and use it to perform feature point correspondence, aiming at matching objects from multiple viewpoints. .

(2-2) Measurement method using a polarizing camera A method using a polarizing camera to remove the specular reflection in the ceiling will be explained. In specular reflection, light is reflected immediately on the surface of an object, while in diffuse reflection, light is diffusely reflected because the refractive index of the object is non-uniform. Due to this difference, the intensity of the specularly reflected light varies depending on the polarization angle as shown in FIG. 6, but the diffusely reflected light is non-polarized with uniform intensity regardless of the polarization angle. Therefore, if polarization information such as the polarization angle and intensity of specular reflection can be calculated, the components of specular reflection and diffuse reflection can be separated, and the specular reflection can be removed. Since the polarization camera has polarizers 60 at several angles in front of the pixels, this can be used to calculate the polarization information of the specular reflection.

(2-2-1) Polarization Camera The polarization camera has a set of pixels with polarizing plates 61, 62, 63, and 64 stretched in the directions of 0, 45, 90, and 135 degrees as shown in FIG. 7A. Only light vibrating in a specific direction can be obtained. By using this, it is possible to calculate the polarization information and remove the specular reflection. FIG. 7B shows the relationship between a set of four pixels with polarizing plates 61, 62, 63, and 64 and RGB.

(2-2-2) Calculation of polarization information (2-2-2-1) Assumptions about polarization angle and light intensity Assuming that the image has only two specular reflection components, since 0 degrees and 180 degrees are the same state, the RGB values of each pixel can be expressed by the following equation (1). As shown in 8, the relationship between the polarization angle and the light intensity can be expressed.

Further, from this assumption, the RGB values _I0 , _I45 , I90, and I135 of four types of pixels of 0 degrees, 45 degrees, ₉₀ degrees, and ₁₃₅ degrees are obtained by the following equations (2), (3), and (4) and (5) can be defined.

The minimum value I _min in the graph corresponds to I _b −I _a , and the maximum value I _max corresponds to I _b +I _a . Also, the component I _max -I _min that varies with the polarization angle corresponds to the specular reflection component, and conversely, the component I _min that does not depend on the polarization angle corresponds to the diffuse reflection component.

(2-2-2-2) Process of deriving I _{a and} I _b using RGB values of pixels In order to calculate components excluding specular reflection, it is necessary to calculate I _b and I _a . First, equation (6) is derived from cos(2θ+180)=−cos(2θ).

Equation (7) is derived from cos(2θ+90)=−sin(2θ) and cos(2θ+270)=sin(2θ).

Therefore, _Ia can be obtained by the following equation (8).

Also, since the sum of cos (2θ), cos (2θ+90), cos (2θ+180), and cos (2θ+270) is 0 regardless of the value of θ, _Ib can be obtained by the following equation (9).

(2-2-2-3) Method for removing specular reflection using polarization information After obtaining I _a and I _b by the method shown in (2-2-2-2) above, specular reflection is used A component from which the specular reflection is removed is obtained by calculating the component I _min excluding . Since I _min is equal to I _b −I _a , if an image is generated with I _b −I _a being the RGB value of the pixel, as can be seen by comparing FIGS. 9A, 9B and 9C, or FIG. As can be seen by comparing 10B and 10C, an image with specular reflection removed is obtained. As can be seen by comparing FIGS. 11A and 11B, specular reflection can improve the accuracy of feature point correspondence.

(2-2-3) Measures against Saturation of RGB Values A camera has a limited luminance range and cannot express all the brightness in the real world. Therefore, if there is an out-of-range luminance in the image, the pixel value will always be the minimum or maximum. Therefore, when the pixel value is the maximum, there is a problem that the brightness cannot be associated with the pixel value in the real world, and specular reflection cannot be removed. One way to solve this problem is to use high dynamic range (HDR) images.

As a method of acquiring HDR images, for example, there is a method of acquiring a plurality of images at each viewpoint a plurality of times by changing the illuminance of the light source that illuminates the object. As another method of acquiring HDR images, for example, there is a method of acquiring multiple images at each viewpoint by changing the shutter speed of the mobile camera 50 multiple times at each viewpoint.

(2-3) Construction of Spatial Model By constructing the spatial model, for example, a drawing can be created. A drawing of the ceiling space can be created by constructing the space model of the ceiling space. The image processing system 1 can be configured so that a spatial model can be constructed using the concept of stereoscopic vision based on a plurality of images of an object photographed from a plurality of different viewpoints. For example, the image processing system 1 uses SLAM technology to estimate the movement position of the mobile camera 50 itself based on a plurality of images of an object taken from a plurality of different viewpoints, and constructs a space model. can be taken. For example, the mobile camera 50 self-runs in the ceiling space and transmits a plurality of images of the ceiling space to the image processing system 1, and the image processing system 1 receives images of the ceiling space from multiple viewpoints received from the mobile camera 50. receive. The image processing system 1 constructs a space model of the ceiling space using SLAM technology from a plurality of images of the ceiling space photographed from multiple viewpoints.

(3) Modification (3-1) Modification 1A
In the above embodiment, the case where the mobile camera 50, which is the same image capturing device, is used as the image capturing device for capturing images from a plurality of different viewpoints has been described. However, the photographing device for photographing from a plurality of different viewpoints is not limited to the mobile camera 50 . The imaging devices that capture images from a plurality of different viewpoints may be, for example, a plurality of imaging devices installed at different locations. As such a plurality of photographing devices, for example, a stereo camera composed of a first camera 51 and a second camera 52 as shown in FIG. 12 can be used.

(3-2) Modification 1B
In the above-described embodiment, a method has been described in which the detection unit 30 uses an image captured by a polarization camera to reduce the specular reflection component. A method of reducing the specular reflection component using a polarization camera is an example of a method of reducing the specular reflection component by image correction. Image correction includes, for example, a method of subtracting specular reflection components. However, the method of reducing errors due to specular reflection in processing an image of a target is not limited to image correction.

As a method other than correcting the image, for example, there is a method of masking a portion where specular reflection occurs in the image. Image masking is, for example, a method of ignoring locations where specular reflection occurs or reducing the weight (influence) of locations where specular reflection occurs in extracting characteristic portions, using a mask signal or the like. As another method, for example, there is a method of removing portions where specular reflection occurs (performing partial removal of the image) and using only portions where specular reflection does not occur for image processing.

(3-3) Modification 1C
In the above embodiment, the case where the image acquisition unit 10 acquires the images of the target photographed from a plurality of different viewpoints with the polarization camera has been described. However, the method of acquiring images is not limited to using a polarization camera. For example, the image acquisition unit 10 may acquire images captured by a light field camera as images of a target captured from a plurality of different viewpoints.

In addition, when acquiring an image, at least one of the conditions of the light striking the object and the light receiving conditions of the mobile camera 50 may be changed. The image acquisition unit 10 captures a plurality of images captured by changing the illuminance of the light source that illuminates the object, a plurality of images captured by changing the shutter speed, or a plurality of images captured by changing the color of the light source that illuminates the object at each viewpoint. It is also possible to adopt a configuration that acquires a plurality of images obtained by scanning. When changing the illuminance of the light source illuminating the object and the shutter speed, an HDR image can be captured as already described. In addition, when the color of the light source is changed, specular reflection and diffuse reflection have different properties with respect to color, so it becomes easier to separate the specular reflection.

(4) Features (4-1)
The image processing system 1 of the above embodiment can detect the difference between the first image 201 and the second image 202 based on the difference in viewpoint in the detection unit 30 while using the specular reflection component recognized by the recognition unit 20. can. As a result, the image processing system 1 is enabled to reduce errors due to specular reflection in the processing of images of objects.

(4-2)
As described in the above embodiment or modified example, the image processing system 1 allows the detection unit to perform image correction, image masking, or image partial removal based on the specular reflection component. With such measures, the image processing system 1 can reduce the effect of the specular reflection component on the error due to the specular reflection component when detecting the difference between the first image and the second image based on the difference in viewpoint. As a result, the image processing system 1 can reduce errors due to specular reflection in processing the image of the object.

(4-3)
As described in the above embodiment, the image processing system 1 captures an image through a polarizing plate at each viewpoint, recognizes the specular reflection component from the polarization angle and light intensity at each viewpoint, and recognizes the specular reflection component. can be easily reduced. By using a polarizing camera, it is possible to acquire only light vibrating in specific directions (four directions in the above embodiment), so that specular reflection components can be calculated from that information. By acquiring an image from which specular reflection components are removed, it is possible to realize feature point correspondence that is resistant to specular reflection. This improves the accuracy of correspondence of feature points of objects with specular reflections.

(4-4)
The image processing system 1 can easily recognize the specular reflection component by using a light field camera, changing the illuminance of the light source, changing the shutter speed, or changing the color of the light source.

(4-5)
In the image processing system 1, the number of photographing devices can be reduced when the same photographing device is used for photographing. For example, in the above-described embodiments, only one mobile camera 50 or one portable camera is required as an imaging device. When photographing with a plurality of photographing devices installed at different locations, it becomes easy to specify the positions of the photographing devices.

(4-6)
The information generating unit 40 of the image processing system 1 generates information about the geometric information of the object, the optical information of the object, or the information of the light source that illuminates the object by using the first image and the second image from different viewpoints. can be enriched. The information generator 40 can generate, for example, information about the shape of objects or 3D mapping of a scene. The information generator 40 can generate, for example, information about the reflection properties of the object. In addition, the information generator 40 can generate information about at least one of the direction and position of the light source, for example.

Although embodiments of the present disclosure have been described above, it will be appreciated that various changes in form and detail may be made without departing from the spirit and scope of the present disclosure as set forth in the appended claims. .

1 image processing system 10 image acquisition unit 20 recognition unit 30 detection unit 40 information generation unit 50 mobile camera (example of imaging device)
51 First camera (example of imaging device)
52 Second camera (example of imaging device)

Luwei Yang, Feitong Tan, Ao Li, Zhaopeng Cui, Yasutaka Furukawa, and Ping，“Polarimetric Dense Monocular SLAM”，IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

Claims (6)

an image acquisition unit (10) for acquiring images of an object photographed from a plurality of different viewpoints;
a recognition unit (20) for recognizing a specular reflection component in the first image and the second image, which are acquired images;
a detection unit (30) that detects a difference between the first image and the second image based on the difference in viewpoint using the recognition result of the recognition unit;
with
An image processing system (1) configured to construct a spatial model by a stereo method based on a plurality of images of an object photographed from a plurality of different viewpoints by detecting differences in images by the detection unit. The detection unit performs image correction, image masking, or partial removal of the image based on the specular reflection component recognized by the recognition unit to obtain the first image and the second image based on the difference in viewpoint. Detect differences with images,
Image processing system (1) according to claim 1. The image acquisition unit acquires an image captured by a polarization camera having polarizing plates (60 to 64) at each viewpoint,
the recognition unit recognizes the specular reflection component based on the polarization angle and the light intensity;
An image processing system (1) according to claim 1 or claim 2. The image acquisition unit obtains, at each viewpoint, an image captured by a light field camera, a plurality of images captured by changing the illuminance of the light source illuminating the target, a plurality of images captured by changing the shutter speed, or a target Obtaining multiple images, taken with different colors of the light source illuminating the
An image processing system (1) according to any one of claims 1-3. The image acquisition unit acquires images of a target captured by the same imaging device at different locations, or images of a target captured by a plurality of imaging devices installed at different locations.
An image processing system (1) according to any one of claims 1 to 4. an information generation unit (40) that generates geometric information of a target, optical information of a target, or information of a light source that illuminates the target based on the detection result of the detection unit;
further comprising
An image processing system (1) according to any one of claims 1 to 5.

Images