JP6166631B2 - 3D shape measurement system - Google Patents

Description

  The present invention relates to a three-dimensional shape measurement system that pastes color information of an image on shape information that is a measurement result of a surrounding three-dimensional shape.

  It is important to record the various situations at the site in the investigation of the current situation accompanying the aging of infrastructure equipment and the cause analysis of accidents at intersections and tunnels on general roads. Conventionally, mainly manual measurement has been widely performed. However, in the manual measurement, there is a problem in terms of reliability because there is a possibility that measurement accuracy is low, human error, and arbitrary intrusion of the measurer or the like may occur.

  On the other hand, there is a 3D scanner that can measure the surrounding three-dimensional shape in a non-destructive manner as a means for recording the shape of the site. The model measured by the 3D scanner is applied to the analysis of the situation of the site. it can. Specific advantages of 3D scanners include the ability to measure the shape of the entire space quickly and with high accuracy compared to manual measurement, and errors caused by human factors (such as recording errors and misunderstandings by the measurer, etc.) It can be prevented, non-destructive measurement can be performed from high points and dangerous places that cannot be reached by humans, and the situation can be reviewed from a free viewpoint once it is incorporated as a model.

  The 3D scanner measures the three-dimensional shape of the measurement target object by irradiating the measurement target object with laser. Specifically, the 3D scanner can calculate the distance from the 3D scanner to the position where the laser hits by measuring the time from when the laser beam is irradiated until the laser beam is reflected and returned. Based on the irradiation direction of the light and the calculated distance, the three-dimensional coordinates of the position where the laser light hit can be calculated. The 3D scanner can measure the surrounding shape as a set of measurement points (hereinafter referred to as a point group) by irradiating laser light in all directions around the 3D scanner and calculating the three-dimensional coordinates of the position where the laser light hits.

  Here, in the recording of the on-site situation, it is important not only to record the on-site shape but also to record on-site color information. However, although the 3D scanner can measure the shape of the object using laser light, it cannot measure the color information of the object. Therefore, a colored point cloud is acquired by combining the shape measured by the 3D scanner and the image captured by the camera. This specific method will be described. First, the recording person images the surrounding situation with the camera when measuring the shape of the 3D scanner. Based on the relative position and orientation of the 3D scanner and the camera, the color of each pixel of the captured image corresponding to each measurement point can be determined. Thereby, color information can be given to the point group measured by the 3D scanner. The relative position and orientation of the 3D scanner and the camera are calculated based on the shape information measured by the 3D scanner and the feature point and edge information from the image captured by the camera. It can be calculated by geometrically matching information and images so as to overlap (see, for example, Non-Patent Document 1).

  Since the 3D scanner acquires the shape of the measurement target as a point cloud, it cannot acquire information between points. However, since the shape of many measurement objects is smooth, even if the shape between points cannot be directly acquired, the shape can be reproduced by interpolating between the points. On the other hand, since the color information to be measured often changes abruptly between adjacent pixels, a situation in which color information of a shape existing between points is likely to occur is likely to occur. In Patent Document 1, a triangle patch is applied to a point group to interpolate the shape between the points, and the color information of the captured image is also given to the interpolation region, thereby allowing the points to be pointed. Describes a method that enables color reproduction between the two.

JP 2005-293021 A

Richard Hartley and Andrew Zisserman, "Multiple View Geometry in computer vision", 2004, Cambridge University Press

  When color information is added to the shape information, as described above, the relative posture between the 3D scanner and the camera is determined by geometrically matching the shape information measured by the 3D scanner and the image captured by the camera. It needs to be calculated. In order to appropriately calculate the relative posture between the 3D scanner and the camera, a camera having a wide angle of view of the lens is used so that a sufficient number of feature points or edge information can be shared between the shape information and the image. It is desirable. However, if the angle of view of the lens is widened without changing the number of pixels of the camera, the imaging area per pixel increases, so the image quality deteriorates and the camera cannot acquire a fine pattern to be imaged. There is. On the other hand, when the angle of view of the lens of the camera is narrow, the camera can acquire a fine pattern, but since the imaging region is small, a sufficient number of feature points or edge information used for matching shape information cannot be acquired. The accuracy of estimating the relative posture between the scanner and the camera is reduced.

  Accordingly, an object of the present invention is to provide a three-dimensional shape measurement system capable of reproducing a fine pattern while maintaining accuracy of matching between shape information and an image.

  If a typical example of the present invention is shown, imaging is performed by a shape information storage unit that stores shape information measured by a shape measurement unit capable of measuring a surrounding three-dimensional shape, and a first imaging unit that can capture color information. A second captured image stored by a first captured image storage unit that stores the captured first image and a second image captured by a second image capturing unit having a wider angle of view than the first image capturing unit and capable of capturing color information. An image storage unit, and a color information pasting unit that pastes color information of the first image and the second image on the shape information.

  The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows. That is, it is possible to provide a three-dimensional shape measurement system that can reproduce a fine pattern while maintaining the accuracy of matching between shape information and image information.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

1 is an overall configuration diagram of a three-dimensional shape measurement system according to a first embodiment of the present invention. It is explanatory drawing of arrangement | positioning of the shape measurement part of Example 1 of this invention, a 1st imaging part, and a 2nd imaging part. It is a hardware block diagram of the computer which has a color information addition part of Example 1 of this invention. It is a flowchart of a process after shape information of Example 1 of this invention is acquired until it memorize | stores in a shape memory | storage part. It is explanatory drawing of the outline of the process which pastes the color information of Example 1 of this invention on shape information. It is detailed explanatory drawing of the 2nd collation part of Example 1 of this invention. It is explanatory drawing when the user of Example 1 of this invention selects the corresponding point between shape information and a 2nd image. It is detailed explanatory drawing of the 1st collation part of Example 1 of this invention. It is explanatory drawing in case the user of Example 1 of this invention selects the corresponding point between a 1st image and a 2nd image. It is explanatory drawing of the process which the color information sticking part of Example 1 of this invention pastes color information on shape information. It is a whole block diagram of the three-dimensional shape measurement system of Example 2 of this invention. It is detailed explanatory drawing of the corresponding point extraction part of Example 2 of this invention. It is explanatory drawing of the process which the corresponding point extraction part of Example 2 of this invention extracts a corresponding point. It is detailed explanatory drawing of the multiple color information sticking part of Example 2 of this invention. It is explanatory drawing of the prior calculation process of the 1st coordinate transformation amount and the 2nd coordinate transformation amount in the three-dimensional shape measurement system of Example 3 of this invention. It is explanatory drawing of the actual measurement process by the three-dimensional shape measurement system of Example 3 of this invention. It is explanatory drawing of the prior calculation process of the 2nd coordinate transformation amount in the three-dimensional shape measurement system of Example 4 of this invention. It is explanatory drawing of the actual measurement process by the three-dimensional shape measurement system of Example 4 of this invention. It is detailed explanatory drawing of the three-dimensional shape measurement system of Example 5 of this invention. It is a flowchart of a process after the 1st image of Example 6 of this invention is image | photographed until it memorize | stores in a 1st captured image memory | storage part. It is explanatory drawing of the example of a display of the colored shape information to which the reliability of Example 6 of this invention was provided.

  The three-dimensional shape measurement system of the present invention is for adding color information to three-dimensional shape information. When pasting image information including color information to shape information, shape information, feature points of image information, edge information, and the like are required. Image information captured by a camera with a narrow angle of view has high image quality, but since the imaging range is narrow, there is a problem in that the accuracy of matching between shape information and image information is reduced. In addition, since the image information captured by a camera with a wide angle of view has a wide imaging range, the accuracy of matching between shape information and image information is improved. However, since the image quality is low, fine patterns cannot be reproduced. There was a problem.

  In the three-dimensional shape measurement system of the present invention, image information captured by a camera with a narrow angle of view is associated with image information captured by a camera with a wide angle of view and image information captured by a camera with a narrow angle of view. It is possible to improve the accuracy of attaching to the shape information. Accordingly, it is possible to provide a three-dimensional shape measurement system capable of reproducing a fine pattern while maintaining the accuracy of matching between shape information and image information.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  A first embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is an overall configuration diagram of a three-dimensional shape measurement system according to a first embodiment of the present invention.

  The three-dimensional shape measurement system includes a shape measurement unit 100, a shape storage unit 101, a first imaging unit 110, a first captured image storage unit 111, a second imaging unit 120, a second captured image storage unit 121, and a first verification unit 112. An image conversion amount storage unit 113, a second collation unit 122, a coordinate conversion amount storage unit 123, an image composition unit 130, a color information pasting unit 131, and a colored shape storage unit 132.

  The shape measuring unit 100 measures a surrounding three-dimensional shape. The shape measuring unit 100 is, for example, a laser distance sensor, a sonar sensor, a stereo camera, or the like, and measures a surrounding shape by measuring a distance to a measurement target. The shape measuring unit 100 only needs to be able to measure a three-dimensional shape, and is not limited to a laser distance sensor, a sonar sensor, a stereo camera, or the like. Further, for example, when the shape measuring unit 100 is a laser distance sensor, the shape measuring unit 100 can acquire the reflection intensity of the laser light to be measured. The stronger the reflection intensity, the whiter the whiteness, and the weaker the reflection intensity, the weaker the reflection intensity. Black and white color information can be acquired. Further, the shape measuring unit 100 can acquire black and white color information based on the distance even when the reflection intensity of the laser beam cannot be acquired. In this case, the color information is black as the distance is long and white as the distance is short.

  The shape storage unit 101 stores shape information measured by the shape measurement unit 100.

  The first imaging unit 110 captures an imaging target including color information, and stores a first image including the captured color information in the first captured image storage unit 111. The second imaging unit 120 includes color information, captures an imaging target with a wider angle of view than the first imaging unit 110, and stores a second image including the captured color information in the second captured image storage unit 121.

  The first imaging unit 110 and the second imaging unit 120 may be any device that can acquire color information of an imaging target, and examples of the first imaging unit 110 and the second imaging unit 120 include a camera. The angle of view of the second imaging unit 120 is wider than the angle of view of the first imaging unit 110. For example, the angle of view of the second imaging unit 120 is about 100 to 120 degrees capable of simultaneously imaging a ground building and a road surface. The angle of view of the first imaging unit 110 may be about 40 degrees to 60 degrees. Note that the second image and the shape information are collated (matched) based on at least one of the edge and the feature point, but if the area sufficient for the collation can be acquired, the angle of view of the second imaging unit 120 is 100 degrees. It is not limited to ~ 120 degrees. The first image and the second image are collated based on the color information. If color information such as a pattern sufficient for the collation can be acquired, the angle of view of the first imaging unit 110 is 40 degrees to 60 degrees. It is not limited to. In the present invention, since the first imaging unit 110 captures a fine pattern or the like that cannot be captured by the second imaging unit 120, the image quality of the first imaging unit 110 must be better than the image quality of the second imaging unit 120. I must. Since the angle of view of the second imaging unit 120 is wider than that of the first imaging unit 110, the number of pixels that can be imaged by the first imaging unit 110 and the number of pixels that can be imaged by the second imaging unit 120 are approximately the same. For example, the image quality of the first imaging unit 110 is better than the image quality of the second imaging unit 120.

  The first collation unit 112 geometrically collates the first image with the second image so that the color information of the first image and the color information of the second image overlap each other, thereby matching the first image with the second image. An image conversion amount that is a conversion amount necessary to correspond to the above is calculated. Specifically, the image conversion amount is a conversion amount for making the first image and the second image correspond to each other. The image conversion amount storage unit 113 stores the image conversion amount calculated by the first verification unit 112. The coordinates of the first image are converted into a coordinate system corresponding to the second image using Equation 1. H shown in Equation 1 represents the image conversion amount, and the coordinates (u, v, 1) of the first image are converted into coordinates (u ′, v ′, 1) using the image conversion amount. Details of the processing of the first verification unit 112 will be described with reference to FIGS.

  The second matching unit 122 uses the feature point and edge information common between the second image and the shape information, and uses the second image and the shape information so that the second image and the shape information overlap. By performing geometric matching, a coordinate transformation amount necessary to correspond to the three-dimensional shape information and the second image is calculated. The coordinate conversion amount is a conversion amount for making the coordinate system of the shape information correspond to the coordinate system of the second image. The coordinate conversion amount storage unit 123 stores the coordinate conversion amount calculated by the second verification unit 122. The shape information is converted into shape information corresponding to the second image picked up from the second image pickup unit 120 on the three-dimensional coordinates using Expression 2. R in Equation 2 represents a rotation matrix, T represents a translation vector, X represents coordinates before conversion, and Y represents coordinates after conversion.

  Then, the converted shape information is converted into two-dimensional coordinates at the same position as the second image using Expression 3. A shown in Expression 3 indicates the coordinate conversion amount, the coordinate Y indicates the coordinates of the shape information, and (u, v, 1) indicates the coordinates on the second image after the conversion of the shape information. Details of the processing of the second verification unit 122 will be described with reference to FIGS.

  The image composition unit 130 executes image composition processing for pasting the first image on the second image based on the image conversion amount. This is referred to as a composite image generated by the image composition processing.

  The color information pasting unit 131 pastes the color information of the composite image on the shape information based on the coordinate conversion amount, and stores the colored shape information on which the color information is pasted in the colored shape storage unit 132. Specifically, since the composite image is the coordinate system of the second image, the color information pasting unit 131 converts the shape information into the coordinate system of the second image based on the coordinate conversion amount, and the color of the angular coordinates of the composite image By associating the information with the same coordinates of the shape information, the color information of the composite image is pasted on the shape information. Since the shape information to which the color information is pasted is the coordinate system of the second image, the color information pasting unit 131 uses the equation 3 to convert the shape information that is the coordinate system of the second image into the shape information before conversion. The information is converted into a three-dimensional coordinate system that is a coordinate system, and the shape information to which the color information after conversion is pasted is stored in the colored shape storage unit 132 as colored shape information.

  In summary, the three-dimensional shape system of the present embodiment associates the first image with the second image based on the image conversion amount, and coordinates-converts the color information of the synthesized image obtained by synthesizing the first image and the second image. Paste to shape information based on quantity.

  FIG. 2 is an explanatory diagram of the arrangement of the shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120 according to the first embodiment of the present invention.

  As shown in FIG. 2, the shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120 do not need to mount each device in the same housing, and each device captures and captures shape information at different points. Measurement is possible. Each device may be mounted in the same housing.

  The first collation unit 112, the image conversion amount storage unit 113, the second collation unit 122, the coordinate conversion amount storage unit 123, the image composition unit 130, the color information pasting unit 131, and the colored shape storage unit 132 (hereinafter collectively referred to as these) The color information adding unit) may be implemented in any one of the shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120, or may be implemented in another computer. In FIG. 3, the case where these are mounted on a computer will be described. The shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120 can input the shape information, the first image, and the second image to a device that includes a color information adding unit.

  As described with reference to FIG. 1, the angle of view of the second imaging unit 120 is wider than the angle of view of the first imaging unit 110. In addition, the measurement range or the imaging range of the shape measurement unit 100, the first imaging unit 110, and the second imaging unit 120 is a range that is sufficient for the first verification unit 112 and the second verification unit 122 to perform verification. .

  FIG. 2 shows a case where the situation near the pedestrian crossing on the road is recorded using the shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120. Since the reflection intensity of laser light differs between the white line of the pedestrian crossing and the road asphalt, the shape information measured by the shape measuring unit 100 can distinguish between the pedestrian crossing and the road by the black and white color information.

  FIG. 3 is a hardware configuration diagram of the computer 300 having the color information adding unit according to the first embodiment of this invention.

  The computer 300 includes a CPU 310, a memory 320, a storage device 330, an input interface (IF) 340, and an output interface (IF) 350.

  The CPU 310 executes various programs stored in the memory 320 and includes a first verification unit 112, a second verification unit 122, an image composition unit 130, and a color information pasting unit 131. The memory 320 stores programs corresponding to the first collation unit 112, the second collation unit 122, the image composition unit 130, and the color information pasting unit 131. When the CPU 310 executes these programs, the CPU 310 The first collation unit 112, the second collation unit 122, the image composition unit 130, and the color information pasting unit 131 are included.

  The memory 320 includes a first captured image storage unit 111, a second captured image storage unit 121, an image conversion amount storage unit 113, a coordinate conversion amount storage unit 123, and a colored shape storage unit 132. Various information is stored in the storage device 330. The shape measurement unit 100, the first imaging unit 110, and the second imaging unit 120 are connected to the input IF 340. The shape information measured by the shape measurement unit 100, the first image captured by the first imaging unit 110, and the first image The second image picked up by the two image pickup unit 120 is taken into the computer 300. Note that a storage medium storing shape information, the first image, and the second image may be connected to the input IF 340, and these may be captured. An output device such as a display is connected to the output IF 350.

  In addition, although FIG. 3 demonstrated the case where a color information addition part was mounted in computers 300 other than the shape measurement part 100, the 1st imaging part 110, and the 2nd imaging part 120, the shape measurement part 100, the 1st imaging Even if the color information adding unit is mounted on any one of the unit 110 and the second imaging unit 120, the hardware configuration is the same as that shown in FIG.

  FIG. 4 is a flowchart of the processing from when the shape information according to the first embodiment of the present invention is acquired until it is stored in the shape storage unit 101.

  First, the shape measurement unit 100 executes shape measurement processing for measuring the surrounding shape, and acquires shape information (401). When the shape measuring unit 100 is a laser distance sensor, the shape information acquired by the shape measuring process is a set of points (point group), and therefore the shape measuring unit 100 cannot directly measure the shape between the points. . Therefore, the shape measuring unit 100 executes a shape interpolation process for interpolating the shape between the points (402). Specifically, the shape measuring unit 100 assumes that the target shape is a plane within a small region, and interpolates between the points using points in the vicinity of the target region. As the interpolation method, for example, a method of applying a curved surface to the shape information acquired by the shape measurement process using the least square method, a method of connecting points with a polygon mesh, or the like can be used. As a result, a fine pattern or the like of the first image and the second image can be pasted on the shape information.

  Next, the shape measuring unit 100 stores the shape information on which the shape interpolation processing has been executed in the shape storage unit 101 (403), and ends the processing.

  The shape measuring unit 100 stores the shape information as it is in the shape storage unit 101 without executing the shape interpolation process, and the computer 300 having the color information adding unit captures the shape information and then converts the shape information into the acquired shape information. On the other hand, shape interpolation processing may be executed.

  FIG. 5 is a schematic explanatory diagram of processing for pasting color information to shape information according to the first embodiment of the present invention.

  The shape information 500 is converted into shape information 501 corresponding to the second image 520 and having the same two-dimensional coordinates as the second image 520 based on the coordinate conversion amount stored in the coordinate conversion amount storage unit 123. Shape information 501 corresponding to the second image 520 is shape information from the imaging position of the second imaging unit 120.

  The coordinate conversion of the shape information 500 in this case will be described. First, the color information pasting unit 131 converts the shape information 500 into shape information corresponding to the second image 520 using Equation 2. Then, the color information pasting unit 131 converts the shape information after conversion into the same two-dimensional coordinate system as the second image 520 using Expression 3 based on the coordinate conversion amount, and acquires the shape information 501.

  The image composition unit 130 converts the first image 510 into an image corresponding to the second image 520 based on the image conversion amount, and pastes the converted first image 510 on the second image 520, thereby combining the first image 510 with the composite image. 521 is generated. A region 522 of the composite image 521 indicates a region where the first image 510 is pasted.

  As described above, since the coordinate systems of the shape information 500 and the composite image 521 become equal, the color information pasting unit 131 associates the color information of the composite image 521 with the corresponding coordinates between the shape information 500 and the composite image 521. Thus, the color information of the composite image 521 can be pasted on the shape information 500. That is, the first image 510 is generated based on the image conversion amount and the second image 520, so that the shape information and the first image are matched in the coordinate system of the second image 520. Based on this matched relationship, the color information of the first image is pasted on the shape information via the composite image. Note that the color information of the composite image 521 is the shape information 500 referred to as colored shape information 530.

  Next, the second verification unit 122 will be described with reference to FIGS. 6 and 7. FIG. 6 is a detailed explanatory diagram of the second verification unit 122 according to the first embodiment of this invention.

  The second verification unit 122 includes an initial conversion amount storage unit 601, a shape image storage unit 602 after initial coordinate conversion, a shape image display unit 603 after initial coordinate conversion, a second captured image display unit 621, a corresponding point instruction unit 630, a corresponding point. A storage unit 631 and a coordinate conversion amount calculation unit 632 are included.

  The second collating unit 122 superimposes the shape information measured by the shape measuring unit 100 and the second image captured by the second imaging unit 120, and geometrically collates them. When the color information is pasted on the shape information, the shape information is converted so as to be viewed from the position where the image including the color information is captured, thereby making the coordinate system of the shape information correspond to the coordinate system of the image. The second collating unit 122 calculates a coordinate conversion amount necessary for making the coordinate system of the shape information correspond to the coordinate system of the second image by geometric collation.

  The initial conversion amount storage unit 601 stores relative position information between the shape measuring unit 100 and the second imaging unit 120. As the relative position information, a specific numerical value may be input by the user. In addition, an interface for receiving input of the imaging position and imaging direction of the second imaging unit 120 from the user is displayed on the screen on which the shape information is displayed, and the relative position information is input by the user via the interface. Also good. When the second imaging unit 120 includes a GPS and an attitude sensor, information acquired by the GPS and the attitude sensor is input to the second verification unit 122, and relative position information is based on the input information. May be set.

  The second verification unit 122 converts the shape information stored in the shape storage unit 101 into shape information corresponding to the second image based on the initial conversion amount, and converts the converted shape information into a shape image storage unit after initial coordinate conversion. Store in 602. The shape image display unit 603 after the initial coordinate conversion displays the shape information stored in the shape image storage unit 602 after the initial coordinate conversion on a display device such as a display. The shape image display unit 603 after the initial coordinate conversion displays the shape information after conversion as an image, but the three-dimensional coordinates of the shape information corresponding to each pixel of the displayed image are stored separately. The second captured image display unit 621 displays the second image stored in the second captured image storage unit 121 on a display device such as a display.

  The corresponding point instruction unit 630 specifies a point (corresponding point) indicating the same portion between the shape information converted to correspond to the second image and the second image, and the corresponding corresponding point is stored in the corresponding point storage unit. 631 is stored. For example, the corresponding point instruction unit 630 is selected by the user based on the shape information displayed by the shape image display unit 603 after the initial coordinate conversion and the second image displayed by the second captured image display unit 621 and indicates the corresponding point. The corresponding point is specified by receiving the input. This process will be described in detail with reference to FIG.

  The corresponding point instruction unit 630 calculates at least one of the shape information, the feature amount of the second image, and the edge information, extracts the feature point based on these, and automatically specifies the extracted feature point as the corresponding point. It is also possible to do.

  The coordinate conversion amount calculation unit 632 calculates a coordinate conversion amount based on the corresponding points stored in the corresponding point storage unit 631, and stores the calculated coordinate conversion amount in the coordinate conversion amount storage unit 123. Specifically, the coordinate conversion amount calculation unit 632 calculates the coordinate conversion amount by calculating the coordinate conversion amount of the shape information for the corresponding point of the shape information and the corresponding point of the second image to be at the same position. calculate. The coordinate transformation amount calculation unit 632 may use, for example, a method such as Direct Linear Transformation Method for calculating the coordinate transformation amount. In order for the coordinate conversion amount calculation unit 632 to calculate the coordinate conversion amount, it is desirable that six or more corresponding points are specified.

  In the present embodiment, the corresponding points are specified, and the coordinate conversion amount is calculated based on the specified corresponding points. However, the coordinate conversion amount may be calculated without using the corresponding points. For example, the coordinate conversion amount of the shape information such that the shape information and the geometric shape of the second image overlap most may be calculated as the second coordinate conversion amount. Further, at least one of the feature point and the edge information is extracted from the shape information and the second image, and the coordinate conversion amount of the shape information in which at least one of the extracted feature point and the edge information matches between the shape information and the second image is It may be calculated as the second coordinate conversion amount.

  FIG. 7 is an explanatory diagram when the user of the first embodiment of the present invention selects a corresponding point between the shape information and the second image.

  In FIG. 7, a shape image display unit 603 after initial coordinate conversion displays an image 60 of shape information, and a second captured image display unit 621 displays a second image 61. The user views the displayed image 60 and the second image 61 and selects corresponding points indicating the same part between the image 60 and the second image 61.

  Here, the corresponding point 620A of the image 60 corresponds to the corresponding point 621A of the second image 61, the corresponding point 620B of the image 60 corresponds to the corresponding point 621B of the second image 61, and the corresponding point 620C of the image 60 The corresponding point 621C of the second image 61 corresponds, and the corresponding point 620D of the image 60 corresponds to the corresponding point 621D of the second image 61.

  Next, the first verification unit 112 will be described with reference to FIGS. 8 and 9. FIG. 8 is a detailed explanatory diagram of the first verification unit 112 according to the first embodiment of this invention.

  The first verification unit 112 includes a second captured image display unit 821, a first captured image display unit 810, a corresponding point instruction unit 830, a corresponding point storage unit 831, and an image conversion amount calculation unit 832.

  The second captured image display unit 621 displays the second image stored in the second captured image storage unit 121 in the same manner as the second captured image display unit 621 of the second collation unit 122 shown in FIG. The first captured image display unit 810 displays the first image stored in the first captured image storage unit 111.

  The corresponding point instruction unit 830 specifies a point (corresponding point) indicating the same portion between the first image and the second image, and stores the specified corresponding point in the corresponding point storage unit 831. The corresponding point specifying method of the corresponding point specifying unit 830 is the same as the corresponding point specifying unit 630 of the second collating unit 122, and the corresponding point is specified by inputting the instruction of the corresponding point selected by the user. Alternatively, the corresponding points may be specified by calculating feature amounts and edge information of the first image and the second image.

  The image conversion amount calculation unit 832 calculates the image conversion amount based on the corresponding points stored in the corresponding point storage unit 831, and stores the calculated image conversion amount in the image conversion amount storage unit 113. Specifically, the image conversion amount calculation unit 832 calculates the image conversion amount by calculating the conversion amount of the coordinates of the first image so that the corresponding point of the first image and the corresponding point of the second image are at the same position. Calculate the quantity. The image conversion amount calculation unit 832 may use, for example, a method such as Direct Linear Transformation Method for calculating the coordinate conversion amount. For the image conversion amount calculation unit 832 to calculate the image conversion amount, it is desirable that four or more corresponding points are specified.

  In the present embodiment, the corresponding points are specified, and the coordinate conversion amount is calculated based on the specified corresponding points. However, the coordinate conversion amount may be calculated without using the corresponding points. For example, the conversion amount of the first image up to the position where the color information of the first image and the second image most overlap may be calculated as the image conversion amount. Also, feature points may be extracted from the first image and the second image, and the conversion amount of the first image up to the position where the extracted feature points match may be calculated as the image conversion amount.

  FIG. 9 is an explanatory diagram when the user of the first embodiment of the present invention selects a corresponding point between the first image and the second image.

  In FIG. 9, the first captured image display unit 810 displays the first image 91, and the second captured image display unit 821 displays the second image 90. The user views the displayed first image 91 and second image 90 and selects corresponding points indicating the same part between the first image 91 and the second image 90.

  Here, the corresponding point 921A of the first image 91 and the corresponding point 920A of the second image 90 correspond, the corresponding point 921B of the first image 91 and the corresponding point 920B of the second image 90 correspond, and the first image A corresponding point 921C of 91 corresponds to a corresponding point 920C of the second image 90, and a corresponding point 921D of the first image 91 corresponds to a corresponding point 920D of the second image 90.

  FIG. 10 is an explanatory diagram of a process in which the color information pasting unit 131 of Example 1 of the present invention pastes color information on shape information.

  In FIG. 10, the color information pasting unit 131 performs coordinate conversion of the shape information 1000 based on the coordinate conversion amount, and an image corresponding to the second image is indicated as an image 1001. An image obtained by combining the first image and the second image by the image combining unit 130 is referred to as a combined image 1010.

  Since the image 1001 and the composite image 1010 correspond to the coordinate system of the second image, the coordinates of the composite image 1010 and the coordinates of the image 1001 correspond to each other. Therefore, the color information pasting unit 131 can paste the color information of the composite image 1010 to the shape information 1000 by adding the color information at each coordinate of the composite image 1010 to the corresponding coordinates of the image 1001.

  When color information has already been assigned to a certain coordinate of the image 1001, the color information pasting unit 131 obtains a color obtained by linearly interpolating the color information already given and the RGB component of the color information given this time. Information may be given, or color information of an image with better image quality may be given with priority.

  As described above, since the color information of the first image and the second image can be pasted on the shape information, the color information of the first image having a narrow angle of view is pasted on the shape information, so that a fine pattern can be reproduced. . Further, by preparing a second image having a wider angle of view, when the first image is pasted on the shape information, the second image having sufficient information used for matching the shape information can be used. The accuracy of matching between images and images can also be maintained.

  Furthermore, the three-dimensional shape measurement system according to the present embodiment calculates an image conversion amount that associates the position of the first image with the second image based on the color information of the first image and the second image, and calculates the calculated image conversion amount. Based on the above, the position of the first image is made to correspond to the second image, a synthesized image obtained by synthesizing the first image and the second image is generated, and the synthesized image is pasted on the shape information. Therefore, the color information of the first image can be pasted on the shape information based on the correspondence between the first image and the shape information obtained using the information of the second image, and the shape information and the image The accuracy of matching can be maintained.

  A second embodiment of the present invention will be described below with reference to FIGS.

  In the first embodiment, a composite image obtained by combining the first image and the second image is pasted on the shape information. However, in this embodiment, the first image is made to correspond to the second image without generating the composite image. By associating the first image corresponding to the second image with the shape information, the color information of the first image and the color information of the second image are respectively pasted to the shape information.

  FIG. 11 is an overall configuration diagram of the three-dimensional shape measurement system according to the second embodiment of the present invention. Among the configurations shown in FIG. 11, the same configurations as those of the three-dimensional shape measurement system shown in FIG.

  The three-dimensional shape measurement system of the present embodiment includes a shape measurement unit 100, a shape storage unit 101, a first imaging unit 110, a first captured image storage unit 111, a second imaging unit 120, a second captured image storage unit 121, a first 1 collation unit 112, image conversion amount storage unit 113, second collation unit 122, second coordinate conversion amount storage unit 1110, corresponding point extraction unit 1100, first coordinate conversion amount storage unit 1101, multiple color information pasting unit (color information A pasting unit) 1102 and a colored shape storage unit 132.

  In the present embodiment, the coordinate conversion amount of the first embodiment will be described as the second coordinate conversion amount. The second coordinate conversion amount storage unit 1110 is the same as the coordinate conversion amount storage unit 123 of the first embodiment.

  The corresponding point extraction unit 1100 converts the first image converted so as to correspond to the second image based on the image conversion amount and the shape information into the second image based on the second coordinate conversion amount. Corresponding points between the converted shape information image converted so as to correspond to each other are extracted, and a first image for matching the coordinate system of the first image and the coordinate system of the shape information based on the extracted corresponding points. The coordinate conversion amount is calculated, and the first coordinate conversion amount is stored in the first coordinate conversion amount storage unit 1101.

  The multiple color information pasting unit 1102 is shape information in which the color information is pasted by pasting the color information of the first image and the second image on the shape information based on the first coordinate transformation amount and the second coordinate transformation amount. Colored shape information is stored in the colored shape storage unit 132.

  FIG. 12 is a detailed explanatory diagram of the corresponding point extraction unit 1100 according to the second embodiment of this invention.

  The corresponding point extracting unit 1100 includes a shape coordinate converting unit 1200, an image coordinate converting unit 1210, a corresponding selecting unit 1211, and a first coordinate conversion amount calculating unit 1212.

  The shape coordinate conversion unit 1200 converts the shape information stored in the shape storage unit 101 into an image in the coordinate system of the second image based on the second coordinate conversion amount stored in the second coordinate conversion amount storage unit 1110. .

  The image coordinate conversion unit 1210 converts the first image stored in the first captured image storage unit 111 into an image in the coordinate system of the second image based on the image conversion amount stored in the image conversion amount storage unit 113. .

  As a result, the first image and the shape information are converted into the coordinate system of the second image.

  Next, the correspondence selection unit 1211 selects a plurality of points at predetermined intervals in a random or grid pattern from the first image converted by the image coordinate conversion unit 1210. Then, the correspondence selection unit 1211 selects a point having the same coordinates as the point selected by the shape coordinate conversion unit 1200 from the image of the shape information converted by the shape coordinate conversion unit 1200, and specifies the three-dimensional coordinates of the selected point. . The correspondence selection unit 1211 then selects the two-dimensional coordinates of the point selected from the first image converted by the image coordinate conversion unit 1210 and the three-dimensional coordinates of the point specified from the shape information image converted by the shape coordinate conversion unit 1200. Is associated with. As a result, the two-dimensional coordinates of the first image are associated with the three-dimensional coordinates of the shape information, and the first coordinate conversion amount calculation unit 1212 calculates the first coordinate conversion amount based on these correspondences.

  FIG. 13 is an explanatory diagram of processing for extracting corresponding points by the corresponding point extracting unit 1100 according to the second embodiment of this invention.

  The first image 1310 becomes an image 1311 converted into coordinates on the second image by the image coordinate conversion unit 1210. In addition, the shape information 1300 is converted into an image viewed from the imaging position from the second imaging unit 120 by the shape coordinate conversion unit 1200, thereby becoming the shape information 1301 after coordinate conversion. As a result, the same coordinates correspond to each other in the image 1311 and the shape information 1301 after coordinate conversion. Moreover, since the post-coordinate conversion shape information 1301 includes three-dimensional coordinates corresponding to the two-dimensional coordinates, the three-dimensional coordinates of the post-coordinate conversion shape information 1301 corresponding to each point of the first image 1310 are uniquely determined. Accordingly, the correspondence selection unit 1211 selects a plurality of points at predetermined intervals in a random or grid pattern from the region 1312 where the second image and the first image 1310 appropriately overlap, and shape information after coordinate conversion corresponding to the selected points. The two-dimensional coordinates 1301 are specified, and the three-dimensional coordinates corresponding to the specified two-dimensional coordinates are specified. Then, the correspondence selection unit 1211 associates the two-dimensional coordinates of the point selected from the region 1312 with the three-dimensional coordinates of the specified shape information.

  FIG. 14 is a detailed explanatory diagram of the multiple color information pasting unit 1102 according to the second embodiment of this invention.

  The multiple color information pasting unit 1102 pastes the color information of the first image and the color information of the second image on the shape information. In the pasting of the color information of the first image and the color information of the second image to the shape information, a coordinate conversion process is executed on the shape information based on the imaging position of the second imaging unit. The color information on the first image and the second image corresponding to each point of the shape information is pasted on the image.

  The multiple color information pasting unit 1102 includes a first coordinate converted shape calculation unit 1410, a first coordinate correspondence calculation unit 1411, a first color information setting unit 1412, a second coordinate conversion shape calculation unit 1420, and a second coordinate correspondence calculation unit. 1421 and a second color information setting unit 1422.

  Based on the first coordinate conversion amount stored in the first coordinate conversion amount storage unit 1101, the first coordinate conversion shape calculation unit 1410 converts the shape information stored in the shape storage unit 101 into the coordinate system of the first image. Convert. The first coordinate correspondence calculation unit 1411 calculates a coordinate correspondence between the first image stored in the first captured image storage unit 111 and the shape information converted by the first coordinate-converted shape calculation unit 1410. . Then, the first color information setting unit 1412 gives the color information of each point of the first image to the corresponding point in the shape information based on the coordinate correspondence.

  Based on the second coordinate conversion amount stored in the second coordinate conversion amount storage unit 1110, the second coordinate conversion shape calculation unit 1420 converts the shape information stored in the shape storage unit 101 into the coordinate system of the second image. Convert. The second coordinate correspondence calculation unit 1421 calculates a coordinate correspondence between the second image stored in the second captured image storage unit 121 and the shape information converted by the second coordinate-converted shape calculation unit 1420. . Then, the second color information setting unit 1422 gives the color information of each point of the second image to the corresponding point of the shape information based on the coordinate correspondence.

  Note that the multiple color information pasting unit 1102 linearly interpolates the RGB components of the color information of both images for the region where the first image and the second image overlap, or the color information of the first image with high image quality. Is given priority.

  As described above, the three-dimensional shape measurement system of this embodiment is converted to correspond to the second image based on the shape information converted into the coordinate system of the second image based on the second coordinate conversion amount and the image conversion amount. A corresponding point is selected from the first image, and the corresponding point of both images is matched to calculate a first coordinate conversion amount for making the coordinate system of the first image correspond to the shape information. The three-dimensional shape measurement system pastes the color information of the first image on the shape information based on the first coordinate conversion amount, and pastes the color information on the second image on the shape information based on the second coordinate conversion amount. Accordingly, based on the correspondence between the first image and the shape information obtained using the information of the second image, the color information of the first image can be pasted on the shape information. The accuracy of matching can be maintained.

  Hereinafter, Example 3 of the present invention will be described with reference to FIGS. 15 and 16.

  In the present embodiment, the first imaging unit 110 and the second imaging unit 120 are mounted on the shape measuring unit 100, and the relative postures of the shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120 are fixed. The case where this is done will be described. In the present embodiment, the shape measuring unit 100 measures the shape of the site, and the first coordinate conversion amount and the second coordinate conversion amount are calculated in advance before the first imaging unit 110 and the second imaging unit 120 image the site. In advance, measurement by the shape measurement unit 100 and imaging by the first imaging unit 110 and the second imaging unit 120 are performed in advance, and the first coordinate conversion amount and the second coordinate conversion amount at the time of actual measurement and imaging. The color information can be pasted on the shape information without calculating.

  FIG. 15 is an explanatory diagram of pre-calculation processing of the first coordinate conversion amount and the second coordinate conversion amount in the three-dimensional shape measurement system according to the third embodiment of the present invention. In addition, the same structure as the structure shown in FIG. 11 of Example 2 among the structures shown in FIG. 15 is provided with the same code | symbol, and description is abbreviate | omitted.

  In FIG. 11, the color information of the first image and the second image is actually pasted to the shape information by the multi-color information pasting unit 1102, but the processing by the multi-color information pasting unit 1102 is executed in the pre-calculation process of this embodiment. Not. The other points are the same as in FIG.

  FIG. 16 is an explanatory diagram of actual measurement processing by the three-dimensional shape measurement system according to the third embodiment of the present invention.

  In the pre-calculation process in FIG. 15, the first coordinate conversion amount is stored in the first coordinate conversion amount storage unit 1101, and the second coordinate conversion amount is stored in the second coordinate conversion amount storage unit 1110. In the present embodiment, the relative postures of the shape measuring unit 100, the first imaging unit 110, and the second imaging unit 120 are fixed. For this reason, in this embodiment, the first coordinate calculated by the pre-calculation process is calculated without calculating the first coordinate conversion amount and the second coordinate conversion amount every time the color information is pasted on the shape information as in the second embodiment. The coordinate conversion amount and the second coordinate conversion amount are read, and the color information is pasted on the shape information.

  First, the shape measuring unit 100 stores the measured shape information in the shape storage unit 101, the first imaging unit 110 stores the captured first image in the first captured image storage unit 111, and the second imaging unit 120 captures the image. The second image is stored in the second captured image storage unit 121.

  Next, the first coordinate conversion amount reading unit 1610 reads the first coordinate conversion amount stored in the first coordinate conversion amount storage unit 1101. The second coordinate conversion amount reading unit 1620 reads the second coordinate conversion amount stored in the second coordinate conversion amount storage unit 1110.

  The multi-color information pasting unit 1102 is based on the first coordinate conversion amount read by the first coordinate conversion amount reading unit 1610 and the second coordinate conversion amount read by the second coordinate conversion amount reading unit 1620. The color information of the image and the second image is pasted on the shape information, and the colored shape information that is the shape information on which the color information is pasted is stored in the colored shape storage unit 132. Note that the processing of the multi-color information pasting unit 1102 is the same as the processing described in FIG.

  As described above, the first coordinate conversion amount and the second coordinate conversion amount are calculated in advance before actually starting measurement and imaging in the field, and when the color information is pasted on the shape information, the first coordinate conversion amount and the second coordinate conversion amount are added. Since it is not necessary to calculate the coordinate conversion amount, it is possible to reduce the processing load of the color information pasting process on the shape information.

  A fourth embodiment of the present invention will be described below with reference to FIGS.

  In the present embodiment, a case where the second imaging unit 120 is mounted on the shape measuring unit 100 and the relative postures of the shape measuring unit 100 and the second imaging unit 120 are fixed will be described. In the present embodiment, the shape measuring unit 100 measures the shape of the site, and before the first imaging unit 110 and the second imaging unit 120 image the site, the second coordinate conversion amount is calculated in advance. The color measurement information can be pasted on the shape information without performing the measurement by the shape measurement unit 100 and the image pickup by the second imaging unit 120 and calculating the second coordinate conversion amount at the time of actual measurement.

  FIG. 17 is an explanatory diagram of a second coordinate conversion amount pre-calculation process in the three-dimensional shape measurement system according to the fourth embodiment of the present invention. Note that, in the configuration illustrated in FIG. 17, the same configuration as the configuration illustrated in FIG. 15 is denoted by the same reference numeral, and description thereof is omitted.

  In the third embodiment, since the relative orientation of the first imaging unit 110 is also fixed, the first coordinate conversion amount is calculated in advance, but in this embodiment, the relative orientation of the first imaging unit 110 is fixed. Therefore, the first coordinate conversion amount is not calculated in advance, and only the second coordinate conversion amount is calculated in advance.

  For this reason, the second collation unit 122 collates the shape information stored in the shape storage unit 101 with the second image stored in the second captured image storage unit 121 to calculate the second coordinate conversion amount, The two coordinate conversion amount is stored in the second coordinate conversion amount storage unit 1110.

  FIG. 18 is an explanatory diagram of actual measurement processing by the three-dimensional shape measurement system according to the fourth embodiment of the present invention.

  In the pre-calculation process in FIG. 17, the second coordinate conversion amount is stored in the second coordinate conversion amount storage unit 1110, and the relative postures of the shape measurement unit 100 and the second imaging unit 120 are fixed. For this reason, in this embodiment, when the color information is pasted on the shape information, the relative orientation of the first imaging unit 110 is not fixed, so the first coordinate conversion amount is calculated and the second coordinate conversion amount is calculated. The second coordinate conversion amount calculated in the pre-calculation process is read without calculating and color information is pasted on the shape information.

  The first collation unit 112 collates the first image stored in the first captured image storage unit 111 and the second image stored in the second captured image storage unit 121, calculates an image conversion amount, and performs image conversion. The amount is stored in the image conversion amount storage unit 113.

  The corresponding point extraction unit 1100 converts the shape information stored in the shape storage unit 101 into the coordinate system of the second image based on the second coordinate conversion amount read by the second coordinate conversion amount reading unit 1620, and performs the first imaging. The first image stored in the image storage unit 111 is converted into the coordinate system of the second image based on the image conversion amount. Then, the corresponding point extraction unit 1100 extracts corresponding points between the converted shape information and the converted first image, calculates a first coordinate conversion amount, and calculates the calculated first coordinate conversion amount to the first coordinate conversion. It is stored in the quantity storage unit 1101.

  The multiple color information pasting unit 1102 pastes the color information of the first image and the second image on the shape information based on the first coordinate conversion amount and the second coordinate conversion amount, and the shape information on which the color information is pasted. The colored shape information is stored in the colored shape storage unit 132.

  As described above, the second coordinate conversion amount is calculated in advance before actually starting measurement and imaging on the site, and it is not necessary to calculate the second coordinate conversion amount when pasting the color information on the shape information. It is possible to reduce the processing load of the process of attaching color information to shape information. In addition, since the first imaging unit 110 is not mounted on the shape measurement unit 100, the first imaging unit 110 can capture an area where detailed color information needs to be stored.

  The present embodiment can also be applied to the first embodiment for calculating the second coordinate transformation amount and the fifth embodiment described later.

  The fifth embodiment of the present invention will be described below with reference to FIG.

  In this embodiment, the color information of the second image is pasted on the shape information, the second colored shape information, which is the shape information on which the color information of the second image is pasted, and the first image are collated. The color information of one image is pasted on the second colored shape information.

  FIG. 19 is a detailed explanatory diagram of the three-dimensional shape measurement system according to the fifth embodiment of the present invention. Of the configurations shown in FIG. 19, the same configurations as those shown in FIG. 1 of the first embodiment and the configurations shown in FIG. 11 of the second embodiment are given the same reference numerals, and the description thereof is omitted.

  The three-dimensional shape measurement system includes a shape measurement unit 100, a shape storage unit 101, a first imaging unit 110, a first captured image storage unit 111, a second imaging unit 120, a second captured image storage unit 121, and a second verification unit 122. , Second coordinate conversion amount storage unit 1110, second image color information pasting unit 1901, second colored shape storage unit 132a, colored shape image matching unit (first coordinate conversion amount calculation unit) 1902, first coordinate conversion amount storage. Unit 1101, first image color information pasting unit 1903, and colored shape storage unit 132b.

  The second image color information pasting unit 1901 pastes the color information of the second image on the shape information based on the second coordinate transformation amount stored in the second coordinate transformation amount storage unit 1110, and the color information of the second image. Is stored in the second color-shaped shape storage unit 132a.

  The colored shape image matching unit 1902 collates the first image stored in the first captured image storage unit 111 with the second colored shape information stored in the second colored shape storage unit 132a, and performs first coordinate conversion. The amount is calculated, and the first coordinate conversion amount is stored in the first coordinate conversion amount storage unit 1101. The calculation method of the first coordinate conversion amount of the colored shape image matching unit 1902 may be the same as that of the second matching unit 122 shown in FIG. 6 of the first embodiment, but in this embodiment, the second image color is used. Since the color information of the second image is pasted to the shape information by the information pasting unit 1901, the colored shape image inter-collation unit 1902 is based on the color information of the first image and the color information of the second colored shape information. By directly collating the first image and the second colored shape information, the first coordinate conversion amount for associating the coordinate system of the first image with the coordinate system of the second colored shape information can be calculated.

  The first image color information pasting unit 1903 associates the first image with the second colored shape information based on the first coordinate transformation amount stored in the first coordinate transformation amount storage unit 1101, and converts the first image into the second colored shape information. The color information of the first image is pasted, and the second colored shape information to which the color information of the first image is pasted is stored in the colored shape storage unit 132b as colored shape information. Note that the first image color information pasting unit 1903 may add color information obtained by linearly interpolating the RGB components of both color information for an area where the color information of the first image and the color information of the second image overlap. Alternatively, the color information of the first image with high image quality may be given with priority.

  As described above, the three-dimensional shape measurement system of the present embodiment converts the coordinate system of the first image into the coordinate system of the shape information based on the shape information to which the color information of the second image is pasted and the color information of the first image. The first coordinate conversion amount corresponding to is calculated. The three-dimensional shape measurement system pastes the color information of the first image on the shape information to which the color information of the second image is pasted based on the first coordinate conversion amount. Accordingly, based on the correspondence between the first image and the shape information obtained using the information of the second image, the color information of the first image can be pasted on the shape information. The accuracy of matching can be maintained.

  In this embodiment, since it is not necessary to calculate the image conversion amount, the processing load of the three-dimensional shape measurement system is reduced as compared with the second embodiment.

  Embodiment 6 of the present invention will be described below with reference to FIGS.

  When the shape measuring unit 100 is an infrared laser distance sensor, the measurement accuracy of the measured shape information depends on the brightness of the surrounding environment and the color of the measurement target. Specifically, the measurement accuracy of shape information decreases as the surrounding environment becomes brighter and decreases as the measurement target color becomes darker. Therefore, in this embodiment, color information indicating the luminance of the first image is also pasted to the shape information as the reliability of the measurement accuracy of the shape information.

  The illuminance in the direction in which the imaging unit images the imaging target is determined by the amount of light reflected in the image captured by the imaging unit with a fixed exposure time. For this reason, the brightness of the surrounding environment and the brightness of the color to be measured are proportional to the luminance of the image captured by the imaging unit with a fixed exposure time. The angle of view of the second imaging unit 120 is as wide as 100 degrees to 120 degrees, and an image may not be clearly captured outdoors due to the influence of backlight, so the exposure time of the second imaging unit 120 needs to be variable. is there. For this reason, the 2nd image imaged by the 2nd imaging part 120 cannot be used for the reliability of the measurement accuracy of shape information. On the other hand, since the first imaging unit 110 has a narrow angle of view and is not easily affected by backlight, the exposure time can be fixed. For this reason, the brightness of the first image captured by the first imaging unit 110 is used as the reliability of the measurement accuracy of the shape information.

  FIG. 20 is a flowchart of processing from when the first image is captured in Example 6 of the present invention until it is stored in the first captured image storage unit 111.

  First, the first imaging unit 110 executes an imaging process for imaging the surrounding environment (2001) and acquires a first image. The first imaging unit 110 executes a reliability image calculation process for calculating the luminance value of the first image as the reliability (2002). Next, the first imaging unit 110 assigns the luminance value calculated in the reliability image calculation process to the first image as the reliability (2003). Specifically, the first imaging unit 110 stores the coordinates of the first image and the luminance value of the coordinates in association with each other. Then, the first imaging unit 110 stores the first image to which the reliability is given in the first captured image storage unit 111 (2004), and ends the process.

  Accordingly, the color information pasting unit 131 of the first embodiment, the multiple color information pasting portion 1102 of the second to fourth embodiments, or the first image color information pasting portion 1903 of the fifth embodiment has the color information of the first image. When pasted on the shape information, the reliability indicating the measurement accuracy of the shape information can be pasted together, and the reliability can be given to the colored shape information.

  The first imaging unit 110 stores the first image as it is in the first captured image storage unit 111 without executing the reliability image calculation process and the reliability provision process, and the computer 300 having the color information addition unit After the first image is captured, the reliability image calculation process and the reliability provision process may be executed on the captured first image.

  FIG. 21 is an explanatory diagram of a display example of colored shape information to which the reliability according to the sixth embodiment of the present invention is given.

  The region where the color information of the first image of the shape information is pasted becomes a reliability providing region 2100 to which reliability is added. A portion with low luminance in the first image indicates that the reliability is low and is displayed in a dark color, and a portion with high luminance in the first image indicates that the reliability is high and is displayed in a light color.

  Since the brightness of the first image is higher in the pedestrian crossing portion than the asphalt portion, the pedestrian crossing portion in FIG. 21 becomes a high reliability region 2102 indicating that the reliability is high, and the asphalt portion in FIG. 21 indicates that the reliability is low. The low reliability region 2101 is obtained.

  As described above, the user of the three-dimensional shape measurement system can grasp the measurement accuracy of the shape information by looking at the colored shape information.

  In addition, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files that realize each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

DESCRIPTION OF SYMBOLS 100 ... Shape measurement part 101 ... Shape memory | storage part 110 ... 1st imaging part 111 ... 1st captured image memory | storage part 112 ... 1st collation part 113 ... Image conversion amount memory | storage part 120 ... second imaging unit 121 ... second captured image storage unit 122 ... second verification unit 123 ... coordinate conversion amount storage unit 130 ... image composition unit 131 ... color information pasting unit 132 ... Colored shape storage unit 601 ... Initial conversion amount storage unit 602 ... Shape image storage unit after initial coordinate conversion 603 ... Shape image display unit after initial coordinate conversion 621 ... Second captured image display unit 630 ... corresponding point instruction unit 631 ... corresponding point storage unit 632 ... coordinate conversion amount calculation unit 810 ... first captured image display unit 830 ... corresponding point instruction unit 831 ... corresponding point storage Unit 832 ... image conversion amount calculation unit 1100 ... corresponding point extraction 1101... First coordinate conversion amount storage unit 1102. Multiple color information pasting unit 1110... Second coordinate conversion amount storage unit 1200... Shape coordinate conversion unit 1210. Corresponding selection unit 1212 ... first coordinate conversion amount calculation unit 1410 ... first coordinate converted shape calculation unit 1411 ... first coordinate correspondence calculation unit 1412 ... first color information setting unit 1420 ... Second coordinate converted shape calculation unit 1421 ... second coordinate correspondence calculation unit 1422 ... second color information setting unit 1610 ... first coordinate conversion amount reading unit 1620 ... second coordinate conversion amount reading unit 1901: Second image color information pasting unit 1902: Colored shape image matching unit 1903 ... First image color information pasting unit 2100 ... Reliability imparting region 2101 ... Low reliability region 2102 ..High reliability area

Claims (12)

A shape information storage unit for storing shape information measured by a shape measurement unit capable of measuring a surrounding three-dimensional shape;
A first captured image storage unit that stores a first image captured by a first imaging unit capable of capturing color information;
A second captured image storage unit for storing a second image captured by a second imaging unit having a wider angle of view than the first imaging unit and capable of capturing color information;
The color information of the first image and the second image have a, and color information attachment unit affixed to said shape information,
The color information pasting unit pastes color information of the first image on the shape information based on a correspondence relationship between the first image obtained using the second image and the shape information. A three-dimensional shape measurement system.   A shape information storage unit for storing shape information measured by a shape measurement unit capable of measuring a surrounding three-dimensional shape;
  A first captured image storage unit that stores a first image captured by a first imaging unit capable of capturing color information;
  A second captured image storage unit for storing a second image captured by a second imaging unit having a wider angle of view than the first imaging unit and capable of capturing color information;
  A color information pasting unit that pastes color information of the first image and the second image on the shape information;
  The image of the first image for matching the first image and the second image by matching the first image and the second image based on the color information of the first image and the second image A first verification unit for calculating a conversion amount;
  Based on the geometric information of the second image and the shape information, a second verification unit that calculates a second coordinate transformation amount for associating the coordinate system of the shape information with the coordinate system of the second image When,
  An image combining unit that generates a combined image by combining the first image and the second image by causing the first image to correspond to the second image based on the image conversion amount;
  The color information pasting unit pastes the color information of the synthesized image on the shape information based on the second coordinate conversion amount, whereby the color information on the first image and the second image is added to the shape information. A three-dimensional shape measurement system characterized by being attached.   A shape information storage unit for storing shape information measured by a shape measurement unit capable of measuring a surrounding three-dimensional shape;
  A first captured image storage unit that stores a first image captured by a first imaging unit capable of capturing color information;
  A second captured image storage unit for storing a second image captured by a second imaging unit having a wider angle of view than the first imaging unit and capable of capturing color information;
  A color information pasting unit that pastes color information of the first image and the second image on the shape information;
  Based on the geometric information of the second image and the shape information, a second verification unit that calculates a second coordinate transformation amount for associating the coordinate system of the shape information with the coordinate system of the second image And having
  The color information pasting unit pastes color information of the second image on the shape information based on the second coordinate conversion amount,
  Further, based on the shape information to which color information is pasted by the color information pasting unit and the color information of the first image, the coordinate system of the first image and the coordinate system of the shape information are made to correspond to each other. A first coordinate conversion amount calculation unit for calculating the first coordinate conversion amount;
  The color information pasting unit pastes the color information of the first image on the shape information to which the color information is pasted by the color information pasting unit based on the first coordinate conversion amount. Dimensional shape measurement system.   A shape information storage unit for storing shape information measured by a shape measurement unit capable of measuring a surrounding three-dimensional shape;
  A first captured image storage unit that stores a first image captured by a first imaging unit capable of capturing color information;
  A second captured image storage unit for storing a second image captured by a second imaging unit having a wider angle of view than the first imaging unit and capable of capturing color information;
  A color information pasting unit that pastes color information of the first image and the second image on the shape information;
  Based on the color information of the first image and the second image, the first image and the second image are collated to calculate an image conversion amount for associating the first image with the second image. A first verification unit;
  Based on the geometric information of the second image and the shape information, a second verification unit that calculates a second coordinate transformation amount for associating the coordinate system of the shape information with the coordinate system of the second image When,
  Between the shape information converted to the coordinate system of the second image based on the second coordinate conversion amount and the first image converted to correspond to the second image based on the image conversion amount, A corresponding point selection unit for selecting corresponding points;
  By matching the corresponding point of the first image selected by the corresponding point selecting unit with the corresponding point of the shape information, the first image for matching the coordinate system of the first image with the coordinate system of the shape information. A first coordinate conversion amount calculation unit for calculating one coordinate conversion amount;
  The color information pasting unit pastes the color information of the first image on the shape information based on the first coordinate conversion amount, and the color information of the second image on the shape based on the second coordinate conversion amount. A three-dimensional shape measurement system, wherein the color information of the first image and the second image is pasted on the shape information by pasting on the information.   The three-dimensional shape measurement system according to claim 4,
  The second verification unit
  Pre-calculating the second coordinate transformation amount;
  Storing the calculated second coordinate transformation amount;
  The first matching unit calculates the image conversion amount in advance,
  The corresponding point selection unit adds the shape information converted into the coordinate system of the second image based on the pre-calculated second coordinate conversion amount and the second image based on the pre-calculated image conversion amount. Select corresponding points with the first image converted to correspond,
  The first coordinate conversion amount calculation unit includes:
  Based on the corresponding points of the shape information, the first coordinate conversion amount is calculated in advance based on the corresponding points of the first image selected by the corresponding point selection unit,
  Storing the calculated first coordinate transformation amount;
  The color information pasting unit reads the previously stored first coordinate conversion amount, pastes the color information of the first image on the shape information based on the read first coordinate conversion amount, By reading the stored second coordinate conversion amount and pasting the color information of the second image on the shape information based on the read second coordinate conversion amount, the first image and the second image A three-dimensional shape measurement system characterized by pasting color information on the shape information.   The three-dimensional shape measurement system according to claim 4,
  The corresponding point selection unit is configured to convert the shape information converted into the coordinate system of the second image based on the second coordinate conversion amount and the second image converted to correspond to the second image based on the image conversion amount. A three-dimensional shape measurement system, wherein one image is displayed and the corresponding point is selected by receiving an instruction of the corresponding point from a user.   The three-dimensional shape measurement system according to claim 4,
  The color information pasting part is
  A first coordinate-converted shape calculation unit that converts the shape information into the coordinate system of the first image based on the first coordinate conversion amount;
  A first coordinate correspondence calculation unit that calculates a correspondence relationship between the coordinates of the shape information converted by the shape calculation unit after the first coordinate conversion and the coordinates of the first image;
  A first color information setting unit that adds color information of the first image to the shape information based on the correspondence calculated by the first coordinate correspondence calculation unit;
  A second coordinate-converted shape calculation unit that converts the shape information into the coordinate system of the second image based on the second coordinate conversion amount;
  A second coordinate correspondence calculation unit that calculates a correspondence relationship between the coordinates of the shape information converted by the shape calculation unit after the second coordinate conversion and the coordinates of the second image;
  A three-dimensional shape measurement system, comprising: a second color information setting unit that adds color information of the second image to the shape information based on the correspondence calculated by the second coordinate correspondence calculation unit. .   The three-dimensional shape measurement system according to any one of claims 2 to 4,
  The second verification unit
  Pre-calculating the second coordinate transformation amount;
  Storing the calculated second coordinate transformation amount;
  The color information pasting unit reads the stored second coordinate conversion amount and pastes the color information of the second image on the shape information.   The three-dimensional shape measurement system according to any one of claims 2 to 4,
  The second verification unit calculates the second coordinate conversion amount using any one of the first method, the second method, and the third method,
  In the first method,
  The second matching unit calculates, as the second coordinate conversion amount, a coordinate conversion amount of the shape information such that the shape information and the geometric shape of the second image overlap most,
  In the second method,
  The second verification unit
  Extracting at least one of feature points and edge information from the shape information and the second image;
  Calculating at least one of the extracted feature points and edge information as the second coordinate conversion amount, the coordinate conversion amount of the shape information that matches between the shape information and the second image;
  In the third method,
  The second verification unit
  Converting the shape information based on an initial conversion amount that is preset and roughly matches the shape information to the coordinate system of the second image;
  Displaying the transformed shape information and the second image;
  Accepting an indication of the displayed shape information and corresponding points of the second image from the user,
  A three-dimensional shape measurement system characterized in that a coordinate conversion amount of the shape information such that corresponding points having received the instructions match is calculated as the second coordinate conversion amount.   The three-dimensional shape measurement system according to claim 2 or 4,
  The first matching unit calculates the image conversion amount using any one of a fourth method, a fifth method, and a sixth method,
  In the fourth method,
  The first collation unit calculates a conversion amount of the first image up to a position where color information of the first image and the second image overlaps most as the image conversion amount.
  In the fifth method,
  The first matching unit extracts feature points from the first image and the second image, calculates a conversion amount of the first image up to a position where the extracted feature points match as the image conversion amount,
  In the sixth method,
  The first verification unit
  Displaying the first image and the second image;
  An instruction of corresponding points of the displayed first image and second image is received from a user;
  A three-dimensional shape measurement system characterized in that a conversion amount of the first image up to a position where corresponding points having received the instructions coincide is calculated as the image conversion amount.   The three-dimensional shape measurement system according to any one of claims 1 to 8,
  The measurement result of the shape measuring unit is a set of points,
  3. A shape interpolation unit that interpolates by connecting a region between the points with a mesh or interpolates by approximating a region between the points with a curved surface 3 Dimensional shape measurement system.   A shape information storage unit for storing shape information measured by a shape measurement unit capable of measuring a surrounding three-dimensional shape;
  A first captured image storage unit that stores a first image captured by a first imaging unit capable of capturing color information;
  A second captured image storage unit for storing a second image captured by a second imaging unit having a wider angle of view than the first imaging unit and capable of capturing color information;
  A color information pasting unit that pastes color information of the first image and the second image on the shape information;
  A reliability calculation unit that calculates a luminance value of the first image as a reliability of measurement accuracy of the shape measurement unit;
  A reliability adding unit that applies the reliability calculated by the reliability calculating unit to the first image;
  The color information pasting unit pastes color information indicating the reliability of the first image on the shape information.