WO2015159835A1

WO2015159835A1 - Image processing device, image processing method, and program

Info

Publication number: WO2015159835A1
Application number: PCT/JP2015/061306
Authority: WO
Inventors: 大輔村山; 徳井　圭
Original assignee: シャープ株式会社
Priority date: 2014-04-17
Filing date: 2015-04-13
Publication date: 2015-10-22

Abstract

An image processing device for calculating three-dimensional position information for a measurement position on a subject to be measured on the basis of a plurality of information items for images of the subject and a marker photographed from a plurality of photograph positions, wherein the image processing device is characterized in that the marker has a plurality of pointers and the positional relationships of the plurality of pointers express a three-dimensional coordinate system, and is characterized by having a three-dimensional position information calculation unit for calculating three-dimensional position information expressed by the three-dimensional coordinate systems of each of the plurality of photograph positions on the basis of the plurality of photograph information items and a three-dimensional position information conversion unit for converting the coordinate systems expressing the three-dimensional position information calculated by the three-dimensional position information calculation unit from the three-dimensional coordinate systems for each photograph position to the three-dimensional coordinate system expressed by the marker.

Description

Image processing apparatus, image processing method, and program

The present invention relates to an image processing apparatus, an image processing method, and the like that calculate the length and area of a subject based on image information.

For example, in home renovations and cleaning services, there is a demand for measuring the length and area of parts such as floors, walls, and windows in order to estimate the construction fee according to the size of the construction area.

Conventionally, the area was measured by manually measuring the length of the measurement object manually by a user using a measure or laser measuring instrument. If measurement is performed manually as described above, there is a possibility that an error may be caused by the measurer or the measurement may be forgotten. In view of this, a method has been developed in which an object to be measured is imaged using an imaging device such as a digital camera, and the three-dimensional position information of the object is estimated from the acquired image information and measured.

As a technique using an imaging device, for example, there is a technique using one sign having a known shape or size (Patent Document 1). In this method, a subject to be measured and a marker are imaged by an imaging device, and a plurality of images are acquired by performing this multiple times from different imaging positions. Then, the user specifies the vertex position of the measurement range on the acquired image. Estimate the 3D position information of the subject at the position specified by the user based on the actual shape and dimensions of the sign and the area and inclination information of the sign appearing in each of the multiple images. Is calculated.

JP 2010-286450 A

In the technique disclosed in Patent Document 1, a subject to be measured and a marker are imaged by an imaging device from a plurality of different imaging positions. At this time, it is necessary that the sign is reflected in all of the plurality of images to be captured, and it is necessary to capture the entire measurement target with all of the plurality of images. That is, the measurable range is a range in which the sign and the measurement target can be imaged within the imaging range of the imaging apparatus.

Therefore, when measuring a large subject with an imaging device with a narrow angle of view, it is necessary to take an image at a distance from the subject in order to keep the subject within the imaging range of the imaging device. However, in a scene where a sufficient distance from the subject cannot be taken, such as when measuring indoors, it is impossible to obtain the area with a single measurement.

In the above-described scene, for example, in a large room or a long corridor floor, when the area is to be measured by the technique of Patent Document 1, first, the measurement range is set to a plurality of ranges that fall within the imaging range of the imaging device. To divide. And it can measure in each divided range, and can acquire the area of the whole floor by integrating each measurement result.

However, this method requires the user to accurately grasp the positional relationship in each measurement, repeat the measurement, and integrate the results.

As described above, when using the technique of Patent Document 1, there is a problem that the user himself / herself must grasp the measurement position accurately and take a countermeasure, and the burden is extremely large. In addition, there is a problem that an error in the integrated final measurement result becomes large, for example, when the accuracy of specifying the measurement range by the user is low or when there is a measurement failure.

In addition, even when measuring a subject having a size that falls within the imaging range of the imaging apparatus, there may be a case where all the vertex positions in the measurement range cannot be captured at any time from any imaging position due to the shielding object. For example, the floor area of a corridor including a corner or the floor area of a non-rectangular room that includes a recessed portion on the wall surface side. In these cases as well, since it is necessary to measure the entire floor by dividing it into a plurality of ranges as described above, the same problem as described above arises.

The present invention has been made in view of the above points, and an image processing apparatus capable of measuring with high accuracy by a method with less burden on the user even in a measurement range that does not fall within the imaging range of the imaging device. An object of the present invention is to provide an image processing method.

According to one aspect of the present invention, an image processing device that calculates three-dimensional position information of a measurement position on a subject based on a plurality of pieces of image information obtained by imaging a subject to be measured and a marker at a plurality of imaging positions. The marker has a plurality of pointers, and the positional relationship between the plurality of pointers represents a three-dimensional coordinate system, and the three-dimensional coordinate system for each of the plurality of imaging positions is based on the plurality of image information. A three-dimensional position information calculation unit that calculates the three-dimensional position information represented, and a coordinate system that represents the three-dimensional position information calculated by the three-dimensional position information calculation unit. There is provided an image processing apparatus comprising: a three-dimensional position information converting unit that converts the information into a three-dimensional coordinate system represented by the marker.

According to the present invention, by imaging a marker and a part of the measurement range for each imaging position, the user designates the measurement position from each of the captured images, and the calculation is performed based on the three-dimensional coordinate system of the marker. Even in a situation where a subject cannot be captured with a single image, the length and area of the subject can be accurately measured.

According to the present invention, the marker includes a first marker representing a first three-dimensional coordinate system, a second marker representing a second three-dimensional coordinate system different from the first three-dimensional coordinate system, and , And the three-dimensional position information calculation unit is based on image information in which the first marker and the second marker are simultaneously imaged among the plurality of image information. A first positional relationship between the first three-dimensional coordinate system and the second three-dimensional coordinate system is calculated, and the three-dimensional positional information conversion unit is configured to determine whether the second marker among the plurality of image information is the second marker. A coordinate system representing the three-dimensional position information calculated by the three-dimensional position information calculation unit based on the image information being imaged is changed from the three-dimensional coordinate system based on the image processing device to the second cubic. Convert to the original coordinate system, and further, the first positional relationship Based on an image processing apparatus characterized by converting the coordinate system to the first three-dimensional coordinate system from said second three-dimensional coordinate system.

According to the present invention, all the calculated three-dimensional position information is converted into a three-dimensional coordinate system of one reference marker, and the length of the subject is based on the three-dimensional position information in which the coordinate system is integrated. The sheath area can be accurately measured.

This specification includes the contents described in the specification and / or drawings of Japanese Patent Application No. 2014-085659 which is the basis of the priority of the present application.

According to the present invention, even in a measurement range that does not fall within the imaging range of the imaging device, it is possible to measure with high accuracy by a method with less burden on the user.

1 is a schematic block diagram illustrating a configuration example of an image processing apparatus according to a first embodiment of the present invention. It is a functional block diagram which shows one structural example of an image process part. It is a schematic block diagram which shows the example of 1 structure of the image processing apparatus using the measuring method by two markers which concerns on the 2nd Embodiment of this invention. It is a figure which shows an example of the measurement scene which has one marker. It is a figure which shows the image imaged with the image processing apparatus. It is a flowchart figure which shows the flow of the measurement process of the measuring method by one marker which concerns on the 1st Embodiment of this invention. It is a figure which shows the relationship between parallax and distance. It is an external view which shows one structural example of a marker. It is a figure which shows the coordinate position relationship of an image processing apparatus, a marker, and a to-be-photographed object. It is a figure which shows the vector to the marker in a three-dimensional coordinate system. It is an overhead view of the measurement range which concerns on the 1st Embodiment of this invention. It is an overhead view of the measurement range which concerns on the 1st Embodiment of this invention. It is an overhead view of the measurement range which concerns on the 1st Embodiment of this invention. It is an overhead view of the measurement position which concerns on the 1st Embodiment of this invention. It is a bird's-eye view connected in order of specification of a measurement position concerning a 1st embodiment of the present invention. It is an overhead view of the measurement range which concerns on the 1st Embodiment of this invention. It is an overhead view which divided | segmented the measurement range which concerns on the 1st Embodiment of this invention into the triangular area | region. It is a flowchart figure which shows the flow of the measurement process of the measuring method by the two markers which concern on the 2nd Embodiment of this invention. It is an overhead view of the measurement range which concerns on the 2nd Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the expressions in the drawings are exaggerated for easy understanding, and may differ from actual ones.

(First embodiment)
FIG. 1A is a functional block diagram illustrating a schematic configuration example of the image processing apparatus 1 according to the first embodiment of the present invention.

The image processing apparatus 1 according to the present embodiment captures a subject to acquire image information, calculates distance information corresponding to the image information, and outputs information from the acquisition unit 10 and the input unit 14. An image processing unit 11 that performs image processing based on input information and calculates a measurement value, a storage unit 12 that stores output information of the acquisition unit 10 and the image processing unit 11, and an acquisition unit 10 and an image processing unit 11 The display unit 13 displays output information and information stored in the storage unit 12, and the input unit 14 receives a user operation and outputs input information to the image processing unit 11.

The acquisition unit 10 includes two

imaging units

100 and 101, and a distance calculation unit 102 that calculates distance information based on the two pieces of image information acquired by the two

imaging units

100 and 101.

The

imaging units

100 and 101 are arranged side by side or vertically so that their optical axes are substantially parallel. The

imaging units

100 and 101 include an optical system such as a lens module (not shown) and an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). In addition, an analog signal processing unit, an A / D (Analog / Digital) conversion unit, and the like are provided, and a signal from the image sensor is output as image information. In the present embodiment, the two

imaging units

100 and 101 having the same configuration are used as an example. However, if the two imaging units can capture the same area and take correspondence between the pixels, the resolution and the image can be obtained. Imaging units having different configurations such as corners may be used.

The distance calculation unit 102 calculates subject distance information by a stereo method based on the image information acquired by the

imaging units

100 and 101.

The stereo system is arranged so that the optical axes of the two imaging units are substantially parallel, and the two imaging units capture substantially the same region, obtain the parallax of the corresponding pixels between the two obtained images, and calculate the parallax. Based on this, the distance is calculated. In the stereo method, obtaining corresponding pixels between two images is called stereo matching.

In stereo matching, one of the two images is set as a standard image, and the other is set as a reference image. For a certain pixel in the standard image when the imaging units are arranged on the left and right, the pixel is matched by scanning the reference image in the horizontal direction. Pixel matching is performed in units of blocks centered on the pixel of interest, and SAD (Sum of Absolute Difference) that calculates the sum of absolute value differences of the pixels in the block is calculated to determine a block that minimizes the SAD value. Thus, the pixel on the reference image corresponding to the target pixel of the standard image is obtained. In addition to SAD calculation methods, there are calculation methods such as SSD (Sum of Squared Difference), graph cut, and DP (Dynamic Programming) matching. The parallax value can be calculated even when the two imaging units are arranged in the vertical direction instead of the horizontal direction. In this case, the reference image may be scanned in the vertical direction instead of the horizontal direction.

When the corresponding pixel is obtained by stereo matching, the parallax value of the pixel is known, and if this is performed for all the pixels, the parallax value corresponding to each pixel of the reference image is obtained. However, since the parallax is obtained by stereo matching, it is an area that is imaged in common by two images.

Based on the calculated parallax value, the distance between the image processing apparatus 1 and the subject can be calculated by the equation Z = (f × B) / (p × d). Here, Z is a distance value, f is a focal length of the two imaging units, B is a baseline length representing a distance between the two imaging units, p is a pixel pitch of the image sensor of the imaging unit, and d is a parallax value. is there.

Further, there is a correlation between the distance and the parallax, and the relationship is as shown in FIG. In the graph shown in FIG. 5, the horizontal axis represents distance, and the vertical axis represents parallax. As shown in FIG. 5, the distance and the parallax are in an inversely proportional relationship, not a linear relationship. That is, the closer the distance, the larger the parallax, the farther the distance, the smaller the parallax, and the closer the distance, the higher the resolution of the distance. Therefore, when the distance of the subject is obtained by the stereo method, the distance to the subject can be obtained with higher resolution as the image is taken closer to the subject.

In the present embodiment, the reference of the three-dimensional coordinate system at the imaging position is the imaging unit 100. This changes depending on which imaging unit the above reference image is used as. That is, if the reference image is the image of the imaging unit 101, the reference of the three-dimensional coordinate system is the imaging unit 101, and the rear principal point position of the optical system of the imaging unit 101 is the origin of the three-dimensional coordinate system. The reference may be either as long as it is fixed to either one.

In the above description, the method for calculating the parallax value for all the pixels of the reference image has been described. However, in order to calculate the area and the length, it is only necessary to calculate the three-dimensional coordinates of the measurement position and the marker. The amount of calculation can be reduced by calculating the parallax value.

The image processing unit 11 includes, for example, a processor such as a CPU (Central Processing Unit) and a main storage device such as a RAM (Random Access Memory), and executes a program stored in the storage device to perform processing. It is.

The storage unit 12 is a storage device configured by, for example, a flash memory or a hard disk.

The display unit 13 is a display that uses, for example, a liquid crystal element or an organic EL (Electro Luminescence) as a pixel.

The input unit 14 is, for example, an input device such as a button or switch provided in the image processing apparatus 1 or a touch panel. In the case of a touch panel, the display unit 13 and the input unit 14 can be integrated into one device by using a general touch panel such as a resistive film type or a capacitance type as the display unit 13. As a result, an operation such as touching the image displayed on the screen and designating the measurement position of the subject becomes possible.

In this embodiment, the image processing apparatus 1 including the acquisition unit 10, the image processing unit 11, the storage unit 12, the display unit 13, and the input unit 14 will be described as an example. However, each processing unit is a separate device. It doesn't matter. For example, image information and distance information acquired by a separately prepared device are transferred to an image processing device having the function of the image processing unit 11 via a network or a storage device such as a flash memory. May be. The display unit 13 may be an external display device such as a liquid crystal monitor, and the input unit 14 may be an external input device such as a mouse or a keyboard, and the same effects as those of the image processing apparatus 1 of the present embodiment can be obtained.

In the following, first, as a first embodiment, a method of measuring by performing imaging a plurality of times using one marker as an integration reference of three-dimensional position information will be described.

<Measurement method with one marker>
FIG. 2 is an overview diagram showing how the area of the subject is measured using one marker. In FIG. 2, the marker 2 and the subject 3 are imaged by the image processing apparatus 1 having the acquisition unit 10. The situation is shown.

A three-dimensional coordinate system Ma shown in FIG. 2 is a three-dimensional coordinate system based on the image processing apparatus 1 when the image processing apparatus 1 performs imaging at the imaging position a.

The three-dimensional coordinate system Mb is a three-dimensional coordinate system based on the image processing apparatus 1 when the image processing apparatus 1 performs imaging at the imaging position b.

The origins of the three-dimensional coordinate systems Ma and Mb coincide with the rear principal point position of the optical system of the imaging unit 100, and the optical axis of the imaging unit 100 coincides with the Z axis of each three-dimensional coordinate system. In the present embodiment, the reference of the three-dimensional coordinate system at each imaging position is the imaging unit 100, but the imaging unit 101 may be used. As described above, the reference of the three-dimensional coordinate system changes depending on which image of the

imaging unit

100 or 101 is used as the reference image when calculating the distance by the stereo method.

Further, the three-dimensional coordinate system Om is a three-dimensional coordinate system based on the marker 2. The origin of the three-dimensional coordinate system Om is a feature point (hereinafter also referred to as “pointer”) located at the center of the marker 2. Details of the marker 2 will be described later.

In the example of FIG. 2, the measurement range is indicated by a thick line on the subject 3, and in order to measure the area of the rectangular area that is the measurement range, four measurement positions a 1, a 2, b 1 indicated by x marks are measured. , B2 three-dimensional position information is acquired. Measurement positions a1, a2, b1, and b2 are the vertices of the measurement range. The measurement range on the subject 3 is a size that does not fit within the imaging range of the

imaging units

100 and 101, and measurement is impossible with one imaging. Therefore, the image processing apparatus 1 picks up images from the two positions of the imaging position a and the imaging position b, acquires the three-dimensional position information of the measurement positions a1, a2, b1, and b2, and measures the area of the rectangular region a1a2b2b1.

3A and 3B show images captured by the image processing apparatus 1 from the positions of the imaging positions a and b. 3A and 3B show images captured by the imaging unit 100, respectively. Note that a total of four images are acquired because imaging is performed by both of the

imaging units

100 and 101 at each of the imaging positions a and b.

3A and 3B show that the marker 2 and a part of the subject 3 are reflected. 3A shows the vertices a1 and a2 of the subject 3, and FIG. 3B shows the vertices b1 and b2 of the subject 3. The positions of the imaging positions a and b need only be set so that the marker 2 and the four vertices a1, a2, b1, and b2 of the measurement range can be seen, and the imaging position and the position of the marker 2 can be freely set. Can be set. However, in the case of the example of the present embodiment, the marker is fixed without being moved during the measurement of one measurement range. This is because coordinate conversion described later is performed based on the marker position.

As described above, images are captured by the

imaging units

100 and 101 at the imaging positions a and b, respectively. Based on the two pieces of image information, the distance calculation unit 102 calculates distance information to the subject by a stereo method. This distance information is distance information corresponding to each pixel of the image of the imaging unit 100. Image information of the imaging unit 100 and distance information of the distance calculation unit 102 are output to the image processing unit 11.

The image processing unit 11 uses the three-dimensional position information of the subject shown in FIG. 3A as the three-dimensional coordinate value in the three-dimensional coordinate system Ma based on the image information captured at the imaging position a and the calculated distance information. calculate. Further, based on the image information captured at the imaging position b and the calculated distance information, the three-dimensional position information of the subject shown in FIG. 3B is calculated as a three-dimensional coordinate value in the three-dimensional coordinate system Mb. At this time, the calculated three-dimensional position information of the vertices a1 and a2 and the three-dimensional position information of the vertices b1 and b2 are different from each other in the reference three-dimensional coordinate system. .

Therefore, the image processing unit 11 according to the present embodiment uses the three-dimensional position information of the vertices a1 and a2 and the vertices b1 and b2 calculated in different coordinate systems as the third order of the three-dimensional coordinate system Om with the marker 2 as a reference. Convert to original location information. The coordinate system conversion method will be described later. The converted three-dimensional position information of the vertices a1, a2, b1, and b2 is in the same three-dimensional coordinate system Om, and represents the positional relationship in the actual space, so that the area can be calculated.

The image processing unit 11 calculates the area of the rectangular region connecting the vertices based on the three-dimensional position information obtained by converting the coordinate system. The method of calculating the area of a region formed by known vertices with a three-dimensional position is a well-known technique and will not be described in detail. For example, a triangular region connecting vertices a1, a2, and b1, and vertices a2, b1, If the areas of two regions of the triangular region connecting b2 are calculated and added, the areas of the rectangular regions a1a2b2b1 can be obtained.

In the above example, the area of the rectangular area surrounded by the four vertices is obtained. However, if three or more measurement positions are specified, the area of the triangular area surrounded by the three vertices can be measured. If two or more points are specified, the distance between the two points, that is, the length can be calculated. Therefore, the image processing unit 11 can output the calculated length or area value as measurement information. It is also possible to calculate and output information such as straight lines passing through the vertices and angles formed by line segments.

The area value calculated by the image processing unit 11 as described above is output to and stored in the storage unit 12. Moreover, a measurement result can be notified to a user by outputting to the display part 13 and displaying on a display. As a result, the measurement result can be confirmed later or displayed on the display unit 13 together with other measurement results for confirmation. In addition to the measurement information, the storage unit 12 may store image information, distance information, and the like of the acquisition unit 10. As a result, it is possible not to perform measurement on the spot but to perform measurement based on image information and distance information later.

Further, by displaying the image information of the imaging unit 100 in addition to the measurement result on the display unit 13, the user can confirm the captured image. As a result, the user can determine the measurement position by specifying the point on the subject that appears in the image by the input unit 14 while confirming the display on the display unit 13. For example, in the above example, the user designates the two-dimensional coordinate positions of the vertices a1 and a2 and the vertices b1 and b2 on the images of FIGS. 3A and 3B displayed on the display unit 13. The information on the designated position input by the input unit 14 is output to the image processing unit 11, and the image processing unit 11 acquires the three-dimensional position information on the designated position, and further performs coordinate conversion processing.

As described above, the image processing apparatus 1 according to the present embodiment calculates the three-dimensional position information of the measurement position on the subject from the acquired image information and distance information, and uses a plurality of different three-dimensional coordinate systems as a reference. The three-dimensional position information is converted into a three-dimensional coordinate system of the marker 2 which is one integrated reference. Then, the area of the measurement range is calculated based on the three-dimensional position information represented by one three-dimensional coordinate system.

In the above example, the measurement is performed by imaging from two imaging positions. However, the number of images to be captured may be increased by adding imaging positions, and the measurement positions specified by the user may be increased. In this case as well, measurement can be performed by converting the three-dimensional position information of the subject calculated at each imaging position into a three-dimensional coordinate system based on one marker by the same method as described above.

Note that the measurement position on the subject and the position of the marker appearing on the image are specified by the user using the input unit 14 while confirming the image displayed on the display unit 13. At this time, by correcting the position specified by the user to the most characteristic position in the vicinity of the specified position, or the position at the corner or end of the subject, the position specifying accuracy of the user Even when is low, the measurement position and marker position are specified with high accuracy. This is because the measurement position is likely to be a characteristic part such as a corner of a room or an edge of a wall. The specified position is corrected by the image processing unit 11 based on image information and distance information. However, since the shape and color of the marker are known, the marker may be automatically detected using a known image recognition technique such as a template matching method.

(Measurement method processing procedure with one marker)
Hereinafter, the processing procedure of the measurement method using one marker will be described with reference to the functional block diagram of FIG. 1B and the flowchart of FIG. FIG. 1B is a functional block diagram showing a configuration example of the image processing unit 11, and such a function can be realized by either a hardware configuration or a software configuration. The image processing unit 11 includes a measurement position designation input receiving unit 11-1, a three-dimensional position information calculation unit 11-2, a three-dimensional position information conversion unit 11-3, an area calculation unit 11-4, and a display control unit. 11-5.

First, the user installs the marker 2 near the center front of the measurement range as shown in FIG. 2 (step S101). In the present embodiment, the marker installation position is appropriately set according to the measurement range of the subject, and is fixed until the measurement is completed once it is installed.

Next, the user moves the image processing apparatus 1 to the imaging position a (step S102), and acquires a part of the subject 3 including the vertices a1 and a2 of the rectangular area that is the measurement range and the marker 2 of the acquisition unit 10. Imaging is performed by the two imaging units 100 and 101 (step S103). At this time, distance information is calculated by the distance calculation unit 102 based on the two pieces of captured image information.

The image processing apparatus 1 displays a reference image (in this embodiment, the image of the imaging unit 100) on the display unit 13 among the two images captured by the

imaging units

100 and 101, and the user confirms the displayed image. Meanwhile, the apexes a1 and a2 of the measurement range are designated by the input unit 14 (step S104). The input information is received by the measurement position designation input receiving unit 11-1.

Next, it is confirmed whether or not all the measurement positions, that is, all the vertex positions in the measurement range have been specified (step S105). If the measurement positions have been completed, the process proceeds to step S106. Return to step S102. If all measurement positions have not been specified, step S102 to step S105 are repeated until all measurement positions are specified. Whether or not all the measurement positions have been designated is displayed on the display unit 13 by, for example, displaying a question as to whether or not the measurement positions are to be designated.

In the case of the example in FIG. 2, after imaging the subject at the imaging position a and selecting the vertices a1 and a2, the process once returns to step S102. Then, the user moves the image processing apparatus 1 to the imaging position b (step S102), and images a part of the subject 3 including the vertices b1 and b2 and the marker 2 with the image processing apparatus 1 (step S103).

The image processing apparatus 1 displays the image of the imaging unit 100 on the display unit 13, and the user designates the vertices b1 and b2 of the measurement range on the displayed image by the input unit 14 (step S104). Then, it is confirmed again whether or not all the measurement positions have been designated (step S105). Since the designation has been completed, the process proceeds to step S106.

Next, the three-dimensional position information calculation unit 11-2 of the image processing unit 11 determines the measurement positions a1 and a2 and the measurement position b1 designated by the user based on the image information and the distance information output from the acquisition unit 10. The three-dimensional position information with b2 is calculated (step S106).

Then, the 3D position information conversion unit 11-3 uses the 3D position information of the measurement positions a1 and a2 and the measurement positions b1 and b2 calculated in different 3D coordinate systems as a 3D coordinate system with the marker 2 as a reference. Is converted into the three-dimensional position information represented by (step S107).

Based on the three-dimensional position information of the converted measurement positions a1, a2, b1, and b2, the area calculation unit 11-4 of the image processing unit 11 calculates an area (step S108), and the calculation result is sent to the display unit 13. Then, the display control unit 11-5 performs display on the display unit 13 (step S109).

By the above procedure, the image processing apparatus 1 can measure the area of the subject using one marker.

Note that output information in each of the above steps, that is, imaged image information, measurement position information designated by the user, and output information of the acquisition unit 10 and the image processing unit 11 are stored in the storage unit 12 and read out appropriately. Is called.

In the above description, after step S104, it is confirmed whether or not all the measurement positions have been specified in step S105. If the measurement positions have been completed, the process proceeds to step S106. However, the present invention is not limited to this. is not. For example, the processing of step S106 and step S107 may be performed when step S104 is completed, and then the confirmation of step S105 may be performed. That is, when the user designates the measurement position on the captured image, the three-dimensional position information of the measurement position may be calculated, and it may be confirmed whether all the measurement positions have been designated thereafter. In this case, the calculated three-dimensional position information is stored in the storage unit 12, and after the three-dimensional position information of all measurement positions is calculated, the process proceeds to step S <b> 108. Information may be read to calculate the area.

Furthermore, for example, when three or more measurement positions are designated, the processing up to step S108 may be executed to calculate the area of the region formed at the designated measurement positions. Every time a measurement position is designated, if the area of the region formed at all the measurement positions designated so far is calculated and displayed on the display unit 13, the procedure for performing the confirmation in step S105 every time is performed. It can be omitted.

(About markers)
As described above, the image processing apparatus 1 according to the present embodiment converts the three-dimensional position information of the subject calculated based on the three-dimensional coordinate system of the image processing apparatus 1 into a three-dimensional coordinate system based on the marker. At this time, the image processing unit 11 detects a marker from the images captured by the

imaging units

100 and 101, and detects a pointer that is a feature point included in the marker, whereby a three-dimensional coordinate system based on the marker is used. Is calculated. Therefore, the marker of this embodiment has a shape for representing a three-dimensional coordinate system and a feature point that can be easily detected from an image. The coordinate system conversion method will be described later.

FIG. 6 shows an example of an external view of the marker 2 in the present embodiment. The marker 2 shown in FIG. 6 has three axes orthogonal to each other, and a sphere as a pointer is arranged at the intersection of the three axes and the tip of each axis. The pointer at the intersection of the three axes represents the origin P ₀ of the three-dimensional coordinate system, and the line segment connecting the center of gravity of the pointer at the origin and the center of gravity of the other pointers Px, Py, Pz is the three-dimensional coordinate system. Represents three axes: X-axis, Y-axis, and Z-axis. By making the three line segments connecting the pointers orthogonal to each other in this way, the three-dimensional coordinate system can be represented by the markers. At this time, since the axes of the marker 2 are orthogonal to each other, the three-dimensional coordinate system represented by the marker 2 is a three-dimensional orthogonal coordinate system. The line segments connecting the pointers are synonymous even if they are straight lines passing through the pointers. If the marker 2 is installed on the floor surface, the XZ plane becomes parallel to the floor surface.

In the example of FIG. 6, the pointer has a sphere shape so that the pointer has the same shape when viewed from any angle and direction. This facilitates marker detection from the image, that is, detection of each pointer. Furthermore, if the hue and saturation of each pointer are made different, each pointer can be easily distinguished and detected in the image, and the direction of the coordinate axis can be determined from any angle and direction.

In the example of FIG. 6 as described above, the pointers are arranged so that the respective axes are orthogonal so that the markers themselves represent a three-dimensional orthogonal coordinate system. However, the arrangement of the pointers is limited to this. is not. Even if the axes of the marker are not orthogonal, if the shape of the marker, that is, the positional relationship of the pointers is known, and the position of the origin and the angle between the axes are known, the axes based on the prior information The direction can be corrected and handled as a three-dimensional orthogonal coordinate system. However, in that case, the number of processes for correcting the axial direction increases, and an error may occur during correction. Therefore, the marker itself may have a shape that can represent a three-dimensional orthogonal coordinate system as in the example of FIG. desirable.

In the example shown in FIG. 6, the pointer spheres are connected by an axis. However, the marker may have any configuration as long as the positional relationship between the pointers does not change. For example, a configuration may be adopted in which a pointer sphere is supported from below by a pedestal, or a cube having a known size may be used as a marker, and each vertex position of the cube may be set as a pointer. However, when a cube is used as a marker, the vertices appearing in the image may differ depending on the angle and direction of imaging. Therefore, it is necessary to match the origin position serving as the reference of the three-dimensional coordinate system with the position and direction of the axis based on the dimension information of the cube, that is, the positional relationship information of the vertices. At this time, if the coloration of each surface of the cube is different, or a different mark is written for each surface, the vertex position can be determined from the information of the surface shown in the image, and the identification of the three-dimensional coordinate system is easy. Become.

It should be noted that, regardless of the marker configuration, it is desirable that all the pointers are easily captured in the image when the marker is imaged from different imaging positions.

(Conversion of 3D coordinate system)
As described above, the image processing apparatus 1 according to the present embodiment converts the three-dimensional position information of the subject calculated in the three-dimensional coordinate system of the image processing apparatus 1 into a three-dimensional coordinate system based on the marker. Hereinafter, the coordinate conversion method will be described with reference to the example of FIG.

FIG. 7 is an external view showing a coordinate position relationship among the image processing apparatus 1, the marker 2, and the subject 4. 7 shows a three-dimensional coordinate system M based on the image processing apparatus 1 at a certain imaging position, a two-dimensional coordinate system m representing the image coordinate system of the imaging unit 100 included in the image processing apparatus 1, and a marker 2. A three-dimensional coordinate system O and a point P on the subject 4 are shown. In the three-dimensional coordinate system O with the marker 2 as a reference, i, j, and k representing unit vectors in the respective axis directions are shown. The two-dimensional coordinate system m is an image projection plane, and shows a state in which the subject 4 and the marker 2 are projected.

Here, the Z axis of the three-dimensional coordinate system M coincides with the optical axis of the imaging unit 100 which is the reference in the image processing apparatus 1. The origin of the two-dimensional coordinate system m is the position of the upper left corner on the image projection plane, the horizontal axis direction of the two-dimensional coordinate system m coincides with the X-axis direction of the three-dimensional coordinate system M, and the two-dimensional coordinate system m The vertical axis direction coincides with the Y-axis direction of the three-dimensional coordinate system M.

In the example of FIG. 7, the three-dimensional coordinate system is all represented by the right-handed system. For the sake of explanation, the relationship between the three-dimensional coordinate system M and the two-dimensional coordinate system m is shown as a perspective projection model. Since the perspective projection model is a known technique, a detailed description thereof is omitted.

The three-dimensional position information (X, Y, Z) of a point on the subject in the three-dimensional coordinate system M is the same as the two-dimensional position information (u, v) of the point projected on the image projection plane in the two-dimensional coordinate system m. It can be calculated based on the subject distance information Z obtained by the stereo method described above. The calculation formula is shown in Formula (1). In addition, the calculation formula of Z shown in Formula (1) is the same as what was mentioned above about the stereo system.

Here, f in Expression (1) is the focal length of the imaging unit, p is the pixel pitch in the image sensor of the imaging unit, w is the horizontal resolution of the image captured by the imaging unit, h is the vertical resolution, and B is the two imagings. The base line length between the parts, d, represents the parallax value obtained by the stereo method. X is an operator representing multiplication. Note that, as shown in Expression (1), X and Y may be calculated by an expression that substitutes the base line length B and the parallax value d.

The image processing unit 11 calculates the three-dimensional position information in the three-dimensional coordinate system M from the two-dimensional position information in the two-dimensional coordinate system m of the point P on the subject and the marker 2 by Expression (1).

Next, the calculated 3D position information in the 3D coordinate system M is converted into 3D position information in the 3D coordinate system O by the following method.

The conversion from the three-dimensional coordinate system M to the three-dimensional coordinate system O can be expressed by a matrix operation of rotation and translation. The vector to the point P viewed from the three-dimensional coordinate system M is P _M , the vector to the pointer at the origin position of the marker 2 is O _M, and the vector to the point P viewed from the three-dimensional coordinate system O is P _O. . At this time, each vector ^{_{_{_{t P M = (P MX,}}}} P MY, P MZ), t O M = (O MX, O MY, O MZ), t P O = (P OX, P OY, P OZ) It is represented by ^t represents transposition.

The unit vectors i, j, and k are represented by ^ti = (ix, iy, iz), ^tj = (jx, jy, jz), and ^tk = (kx, ky, kz). The three-dimensional position information representing the vectors P _M and O _M is calculated by the above equation (1). Further, the three-dimensional position information of the point P in the three-dimensional coordinate system O, that is, _PO is calculated by the equation (2).

Here, Equation (2) is expressed using a homogeneous coordinate system. Further, · is an operator representing an inner product operation.

The matrix included in the right side of Expression (2) is a transformation matrix, and the three-dimensional position information in the three-dimensional coordinate system M is converted into the three-dimensional position information in the three-dimensional coordinate system O by the transformation matrix. The transformation matrix V is shown in Formula (3).

The transformation matrix V can be calculated from the position of the marker 2 in the image, and the unit vectors i, j, and k are calculated from the three-dimensional position information of each pointer of the marker 2. A method for calculating unit vectors i, j, and k will be described below.

The two-dimensional position information of each pointer of the marker 2 in the two-dimensional coordinate system m is acquired by detecting each pointer from the image, and the three-dimensional position information of each pointer in the three-dimensional coordinate system M is obtained by the above equation (1). Is calculated. Here, FIG. 8 shows an enlarged view of the area of the three-dimensional coordinate system M and the marker 2 of FIG. As shown in FIG. 8, the vector to the origin pointer of the marker 2 viewed from the three-dimensional coordinate system M is g0, the vector to the pointer in the X-axis direction is g1, the vector to the pointer in the Y-axis direction is g2, and the Z-axis When the vector to the direction pointer is g3, the unit vectors i, j, and k can be calculated by the equation (4).

Here, || represents an absolute value, and the denominator portion in each expression of Expression (4) represents the distance between the pointers. Incidentally, the vector g0 is identical to the vector _{O M.} O _M a i, j, based on the k, it is possible to calculate a transformation matrix V.

By the above method, the three-dimensional position information of the three-dimensional coordinate system based on the image processing apparatus can be calculated from the two-dimensional position information of the subject in the image. Furthermore, by calculating the conversion matrix, it is possible to convert the three-dimensional position information into a three-dimensional coordinate system based on the marker.

(Second Embodiment)
Next, a method according to the second embodiment in which a measurement range that does not fall within the imaging range of the imaging device is measured using two markers will be described.

<Measurement method using two markers>
In the present embodiment, the image processing apparatus 1 in FIG. 1A is used as in the first embodiment. However, the functional block diagram of the image processing unit 11 illustrated in FIG. 1B is replaced with a functional block diagram illustrating a configuration example of the image processing unit 11a illustrated in FIG. 1C. Details of the functional block will be described later.

The measurement range that does not fall within the imaging range of the imaging apparatus described in this embodiment is a measurement range that requires multiple measurements in the measurement method using one marker. For example, it is a measurement range such as a large room such as a hall or a floor such as a corridor where a shielding area is generated in one shooting including a corner.

In the following, a measurement method using two markers will be described by taking as an example the case of a corridor including a corner shown in FIG. 9A, 9B and 9C are overhead views of a corridor including a corner, each showing the same region. FIG. 9A shows two markers 20 and 21, FIG. 9B shows two markers 21 and 22, and FIG. 9C shows two markers 22 and 23. The marker 22 in FIG. 9B is the same as the marker 20 in FIG. 9A, and indicates that the marker 20 has been moved from the position in FIG. 9A in the direction of the one-dot chain line L1 shown in FIG. 9B. Similarly, the marker 23 in FIG. 9C is the same as the marker 21 in FIGS. 9A and B, which indicates that the marker 21 has been moved from the position in FIGS. 9A and B in the direction of the one-dot chain line L2 shown in FIG. 9C. Further, the marker 20 (or marker 22) and the marker 21 (or marker 23) are shown as different figures for easy understanding, but the actual shape is the same as the marker 2 shown in FIG. belongs to. However, in order to easily detect that two markers are different in an image to be captured, for example, the color scheme of the markers may be different. This makes it possible to perform marker discrimination at the time of marker association described later. Note that the present invention is not limited to the method of changing the color scheme, and any method may be used as long as the two markers can be distinguished, such as marking the marker.

FIG. 9A shows imaging positions 90, 91, and 92, which indicate that the image processing apparatus 1 captures images from the respective imaging positions in the direction of the dashed arrows. In addition, the measurement positions 900, 901, and 902 specified by the user in the image captured at the imaging position 90, the measurement position 910 specified by the user in the image captured at the imaging position 91, and the image captured at the imaging position 92 are displayed. , Measurement positions 920 and 921 designated by the user are shown.

Note that FIG. 9A is merely an example, and when 900 is shown in both images at the imaging positions 90 and 91, any image may be selected. In this example, since 900 has already been selected in the image at the imaging position 90, the 900 position is not selected in the captured image at the imaging position 91. Therefore, in the image of the

imaging position

90, 900 may be selected without selecting 900, or both images 900 may be selected.

FIG. 9B shows an image pickup position 93, which shows that the image processing apparatus 1 picks up an image in the direction of the broken line arrow. In addition,

measurement positions

930 and 931 designated by the user in the image captured at the imaging position 93 are shown.

FIG. 9C shows

imaging positions

94 and 95, which indicate that the image processing apparatus 1 captures images from the respective positions in the direction of the broken-line arrows. In addition,

measurement positions

940 and 941 specified by the user in the image captured at the imaging position 94 and

measurement positions

950 and 951 specified by the user in the image captured at the imaging position 95 are shown.

The number of each imaging position described above represents the order of imaging, and imaging is performed in the order of

imaging positions

90, 91, 92, 93, 94, and 95.

In the case of measurement using two markers, similarly to the measurement method using one marker, the marker and a part of the measurement range are imaged for each imaging position, and the user designates the measurement position from each of the captured images.

First, as shown in FIG. 9A, imaging is performed at

imaging positions

90, 91, and 92 so that the marker 20 and the vertex position of the measurement range are shown. Then, the vertex position is specified in each image to determine the measurement position, and the three-dimensional position information is converted into the three-dimensional coordinate system of the marker 20 in the same manner as described above. That is, the measurement positions 900, 901, 902, 910, 920, and 921 are converted into three-dimensional position information represented by the three-dimensional coordinate system of the marker 20.

Here, both the marker 20 and the marker 21 are shown in the image at the imaging position 92 in FIG. 9A, and each three-dimensional position information is based on the three-dimensional coordinate system based on the image processing apparatus 1 at the imaging position 92. It is represented by Therefore, the positional relationship between the three-dimensional coordinate system of the marker 20 and the marker 21 can be obtained, and the three-dimensional position information represented by the three-dimensional coordinate system of one marker is used as the three-dimensional coordinate of the other marker. It is possible to convert it into three-dimensional position information represented by a system. In addition, the coordinate conversion method between markers is mentioned later.

Next, the marker 20 in FIG. 9A is moved to the position of the marker 22 in FIG. 9B, and the marker 21, the marker 22, and the measurement position that has not yet been specified are imaged at the imaging position 93 in FIG. 9B. Then, the three-dimensional position information is calculated by designating the measurement positions 930 and 931 in the image of the imaging position 93 and converted into the three-dimensional coordinate system of the marker 21. Furthermore, since the three-dimensional coordinate system can be converted from the marker 21 to the marker 20 as described above, the measurement positions 930 and 931 specified by the image of the imaging position 93 are represented by the three-dimensional coordinate system of the marker 20. Into three-dimensional position information.

Here, both the marker 21 and the marker 22 are shown in the image at the imaging position 93 in FIG. 9B, and the correspondence between the three-dimensional coordinate systems of the marker 21 and the marker 22 can be obtained in the same manner as described above. That is, the three-dimensional position information represented by the three-dimensional coordinate system of the marker 22 can be converted into the three-dimensional position information represented by the three-dimensional coordinate system of the marker 21.

Next, the marker 21 in FIG. 9B is moved to the position of the marker 23 in FIG. 9C, and the marker 22, the marker 23, and the measurement position that has not yet been specified are imaged at the imaging position 94 in FIG. 9C. Then, the measurement positions 940 and 941 are specified in the image of the imaging position 94 to calculate the three-dimensional position information, and the marker 22 is converted into the three-dimensional coordinate system. Further, the three-dimensional position information is calculated by designating the imaging positions 950 and 951 in the image captured at the imaging position 95 and converted into the three-dimensional coordinate system of the marker 23.

Here, both the marker 22 and the marker 23 are shown in the image at the imaging position 94 in FIG. 9C, and the correspondence between the three-dimensional coordinate systems of the marker 22 and the marker 23 can be obtained in the same manner as described above. That is, the three-dimensional position information represented by the three-dimensional coordinate system of the marker 23 can be converted into the three-dimensional position information represented by the three-dimensional coordinate system of the marker 22.

As described above, the three-dimensional coordinate systems indicated by the markers 20, 21, 22, and 23 are associated with each other based on the images at the imaging positions 92, 93, and 94, and can be converted into each other's coordinate system.

By the above method, the plurality of measurement positions designated by the images captured at the plurality of imaging positions are converted into the three-dimensional position information represented by the three-dimensional coordinate system of one marker serving as a reference for integration. be able to. That is, when the marker 20 is set as the reference marker in the above example, all measurement positions shown in FIGS. 9A, 9B, and 9C are calculated as three-dimensional position information represented by the three-dimensional coordinate system of the reference marker 20. Can do. The reference marker may be a marker other than the marker 20. By setting the reference marker as the marker to be set first, it is possible to adjust the setting position of another marker (non-reference marker) that is not the reference in accordance with the position of the reference marker.

Note that the marker installation position and the imaging position shown in FIG. 9 are not limited to this, and the marker appears in each image, and all measurement positions can be selected from images captured at any imaging position. Each may be set as follows. Further, when coordinate conversion between markers is performed, settings are made so that two markers are captured at one imaging position, such as imaging positions 92, 93, and 94 in FIG.

Next, create an overhead view of the measurement range in order to determine the range for calculating the area. Although the three-dimensional position information of all measurement positions is calculated by the above method, the connection between them is unknown, and it is not known what range the area for calculating the area is. Therefore, the three-dimensional position information of all measurement positions selected by the user is projected onto a surface where the measurement range exists or a surface parallel to the surface where the measurement range exists, and the measurement positions as shown in FIG. Create an overhead view. In this case, since the measurement range is the floor surface, if the reference marker 20 is installed on the floor surface, the XZ plane in the three-dimensional coordinate system of the reference marker 20 becomes parallel to the floor surface of the measurement range. Therefore, the three-dimensional position information of all measurement positions is projected onto the XZ plane, the two-dimensional position information represented by the two-dimensional coordinate system of the X-axis and the Z-axis is calculated, and this two-dimensional coordinate system is represented in the overhead view. By using the image coordinate system, it is possible to create an overhead view as if the floor surface was viewed from vertically above.

Note that, for example, when measuring the area of the wall surface standing perpendicular to the floor, the overhead view may be created by projecting the three-dimensional position information of all measurement positions onto a plane parallel to the wall surface. At this time, when the marker is placed on the floor so that the X axis of the three-dimensional coordinate system of the marker is parallel to the wall surface of the measurement range, the XY plane of the marker may be used as the projection plane. The bird's-eye view in this case is not a bird's-eye view looking down from above, but a diagram seen in a direction perpendicular to the wall surface, and is a bird's-eye view for convenience of explanation. In addition, when the surface of the measurement range and any surface represented by any two axes of the marker are not parallel, the position information of the surface to be measured is calculated, and the 3D coordinate system of the marker What is necessary is just to obtain | require positional relationship and to project three-dimensional positional information on the surface of a measurement range based on positional relationship. Here, since the surface information can be calculated if the three-dimensional position information of three or more points existing on the surface is known, the user designates three or more points on the surface of the measurement range in the image. By doing so, it is possible to calculate information on the surface of the measurement range.

In addition, since the connection of each measurement position is not known in the state of FIG. 10A, the measurement range is unknown. Therefore, the image processing apparatus 1 connects the measurement positions shown in FIG. 10A in the order specified by the user, thereby creating an overhead view as shown in FIG. 10B and displaying it on the display unit 13. However, since the connection between the specified order and the measurement position does not necessarily match, the user makes corrections. For example, while confirming the display on the display unit 13, the user specifies and corrects an erroneously connected part or an unconnected part using the input unit 14, and the measurement position is correctly connected as shown in FIG. 10C. Create an overhead view. In this way, it is preferable that the measurement points are connected in the order designated by the user and only the erroneous part is corrected so that the work of the user is reduced. Note that the connection method of the measurement position is not limited to the above, and the positional relationship between the measurement position and surrounding objects may be estimated based on image information or three-dimensional position information. For example, if there is a wall between two measurement positions, the connection order of the measurement positions can be estimated such that the measurement positions are not connected through the wall.

Then, in order to calculate the area, the image processing apparatus 1 divides the measurement range as shown in FIG. 10D based on the three-dimensional position information of each measurement position and the connection information of the measurement position in the overhead view of FIG. 10C. To do. By dividing into triangular areas as shown in FIG. 10D, the area can be calculated from the three-dimensional position information of the three vertices for each divided area, and the total area can be obtained by adding them. Any method may be used for dividing the area, but it is necessary that the entire range is included in any divided area and that the divided areas do not overlap each other. If the overhead view of FIG. 10D is displayed on the display unit 13, even if the area division of the image processing apparatus 1 is incorrect and the same range is included in a plurality of divided areas, the user can input the portion 14. Can be specified and corrected.

As described above, if the measurement position is automatically connected and the region is divided by the image processing apparatus 1, the user can manually connect the measurement points and save the trouble of dividing the region of the measurement range. it can.

Finally, the image processing apparatus 1 calculates the area of each divided triangular region based on the three-dimensional position information of the measurement position, and calculates the area of the entire measurement range by adding all the calculation results. Note that the area calculation method is not limited to the method of dividing the area as described above, and any method can be used as long as the area of the entire measurement range can be calculated based on the three-dimensional position information of the measurement position. I do not care. In addition, when measuring the length of the line segment connecting the measurement positions, that is, the length of each side, the interval between sides, etc., there is no need to divide the area as described above, and the tertiary of the corresponding measurement position. What is necessary is just to calculate based on original position information.

As described above, by using two markers, the entire area can be calculated in a single measurement even if the measurement range requires multiple measurements. In addition, since all 3D position information is expressed in one 3D coordinate system with the reference marker as a reference, the user accurately grasps the positional relationship of the measurement positions as in the case of performing multiple measurements. Thus, the trouble of integrating a plurality of results can be eliminated.

In addition, as described in the first embodiment, if the measurement position designated by the user is automatically corrected by the image processing unit 11 based on the image information and the distance information, even when the user's designation accuracy is low. Measurement is possible by specifying the position with high accuracy.

Furthermore, if the imaging and the movement of the marker are repeated as described above, it is possible to measure even a wider range or a complicated shape range than the example of FIG. At this time, it is possible to repeat the coordinate system conversion from the marker to the marker by imaging while alternately exchanging the marker to be moved and the fixed marker. If you set the first marker as the reference marker and take an image, then move the reference marker and the other markers (non-reference markers) while moving them alternately. The three-dimensional position information of the measurement position can be converted into the three-dimensional coordinate system of the reference marker before movement. And based on the three-dimensional position information with which the coordinate system was integrated, an overhead view can be produced and measured.

Note that when the first set reference marker is moved, the moved reference marker is handled in the same manner as other non-reference markers. That is, since the three-dimensional coordinate system of the reference marker after movement exists at a different position from the three-dimensional coordinate system of the reference marker before movement, it can be handled in the same manner as the three-dimensional coordinate system of the non-reference marker. Therefore, it is possible to convert the coordinate system from the three-dimensional coordinate system of the reference marker after movement to the three-dimensional coordinate system of the non-reference marker, and further to the three-dimensional coordinate system of the reference marker before movement. .

In the above description, the method using two markers is described, but the number of markers is not limited to this. In the above example, the marker 22 is the same as the marker 20, but the marker 22 may be a marker different from the marker 20 or 21. That is, three markers may be used. Also in that case, coordinate conversion is possible from the positional relationship between the markers appearing in the image, and conversion into a three-dimensional coordinate system of one marker as a reference is possible. Even if more markers are used, if coordinate conversion is performed in the same manner as described above, all measurement positions can be converted into three-dimensional position information of one three-dimensional coordinate system, and measurement can be performed. However, it is desirable to use a minimum number of markers because the number of markers increases the cost and takes time and effort.

In the above example, the measurement is performed by performing the imaging while moving the two markers and moving in a certain direction in the order of the imaging positions 90 to 95, but the imaging order is not limited to this. . For example, the imaging order of the imaging positions 90 and 91 may be reversed, or imaging may be performed in the order of the imaging positions 95 to 90 while moving the marker in the reverse order to the above, and any imaging order It does not matter.

However, in the above example, the coordinate system is converted based on the positional relationship between the two markers that appear in the images at the imaging positions 92, 93, and 94. However, once the marker is moved, it is completely in the same position and orientation as before the movement. It is difficult to return. Therefore, for example, if the marker 20 is moved before the imaging at the imaging position 92, the correspondence between the coordinate system of the marker 20 and the marker 21 cannot be achieved. For this reason, it is necessary to determine the movement timing and movement position of the marker, the imaging order, the imaging position, and the like so that the coordinate systems of all the markers can be associated with each other.

When the moving direction of the marker and the imaging position is constant as in the above example, the image processing apparatus 1 can convert the coordinate system by associating the markers appearing in the image in the order of imaging. It is. In addition, when the imaging order and the marker movement timing are not constant as in the above example, the user informs the image processing apparatus 1 of information such as in which image the marker in the image is the same that has not moved. By inputting, all the markers are associated with each other and the coordinate system is converted.

In the above description, all the three-dimensional position information is converted into the three-dimensional coordinate system of the reference marker. However, the information may be converted into a three-dimensional coordinate system based on the image processing apparatus 1. That is, the reference three-dimensional coordinate system may be the three-dimensional coordinate system of the image processing apparatus 1. In this case, in the same manner as described above, all the three-dimensional position information is converted through the marker three-dimensional coordinate system, and further converted into the coordinate system of the image processing apparatus 1 when the reference marker is first imaged. That's fine. Conversion from the coordinate system of the reference marker shown in the image to the coordinate system of the image processing apparatus 1 can be performed based on the camera parameters. If all the three-dimensional position information is converted into the coordinate system of the image processing apparatus 1, measurement can be performed in the same manner as described above. Further, the axial direction of the three-dimensional coordinate system of the image processing apparatus 1 changes depending on the orientation of the imaging unit 100 as a reference. For example, when the projection plane when creating the overhead view is a horizontal plane such as a floor surface, the attitude information of the image processing apparatus 1 can be determined by detecting the attitude information of the imaging unit 100 using a triaxial acceleration sensor or the like, The positional relationship with the projection plane can be grasped. Thereby, it is possible to save the trouble of calculating the information of the surface in the measurement range from the image and estimating the positional relationship between the surface and the reference three-dimensional coordinate system. Examples of the three-axis acceleration sensor include a capacitance type, a piezoresistive type, a heat detection type sensor, and the like to which MEMS (Micro Electro Electro Mechanical Systems: a micro electro mechanical element and its creation technology) is applied. The capacitance type is a method for detecting a change in capacitance between the movable part of the sensor element and the fixed part, and the piezoresistive type is a spring that is accelerated by a piezoresistive element placed in the spring part that connects the movable part of the sensor element and the fixed part. This is a method for detecting distortion generated in a portion. Further, the heat detection type is a method in which a hot air current is generated in a housing by a heater and a convection change due to acceleration is detected by a thermal resistance or the like.

(Measurement procedure using two markers)
Hereinafter, the processing procedure of the measurement method using the two markers described above will be described with reference to the flowchart in FIG. 11 and the functional block diagram of the image processing unit 11a in FIG. 1C. The image processing unit 11a includes a measurement position designation input receiving unit 11a-1, a three-dimensional position information calculating unit 11a-2, an inter-marker association processing unit 11a-3, a three-dimensional position information converting unit 11a-4, A marker movement instruction receiving unit 11a-5, an overhead view creation / display control unit 11a-6, an overhead view correction control unit 11a-7, a measurement range region division processing unit 11a-8, an area calculation unit 11a-9, Display control unit 11a-10.

First, the user installs the two markers 20 and 21 at, for example, the position shown in FIG. 9A (step S201). At this time, the marker 20 is installed in the vicinity of the center of the lower region of FIG. 9A, and the marker 21 is installed at a position where it can be imaged simultaneously with the marker 20 and in the vicinity of the center of the upper region of FIG.

Next, the user moves the image processing apparatus 1 to the imaging position 90 (step S202), and the image processing apparatus 1 images the marker 20 and a part of the measurement range (step 203). The image information captured by the imaging unit 100 and the distance information calculated by the distance calculation unit 102 based on the image information of the imaging unit 100 and the imaging unit 101 are output to the image processing unit 11a. Further, the image information of the acquisition unit 100 is displayed on the display unit 13. While confirming the image displayed on the display unit 13, the user designates the measurement positions 900 and 901 with the input unit 14 (step S204), and the measurement position designation input reception unit 11a-1 receives the input. The three-dimensional position information calculation unit 11a-2 of 11a calculates the three-dimensional position information of each measurement position by the method described above (step S205). Further, the marker-to-marker association processing unit 11a-3 of the image processing unit 11a detects a marker in the image, and associates the marker (step S206). Details of step S206 will be described later. Next, the three-dimensional position information conversion unit 11a-4 of the image processing unit 11a converts the three-dimensional position information of the measurement position calculated in step S205 into the three-dimensional coordinate system of the reference marker associated in step S206. (Step S207).

Next, it is confirmed whether or not all measurement positions in the measurement range have been designated (step S208). If the designation has not been completed, the process proceeds to step S209. If the designation has been completed, the process proceeds to step S211. In step S208, the user is allowed to select the display unit 13 by displaying a question as to whether or not to continue specifying the measurement position.

At this point, since all the measurement positions have not yet been specified, the process proceeds to step S209. In step S209, the marker movement instruction receiving unit 11a-5 confirms whether or not to move the marker (step S209). If the marker is not moved, the process returns to step S202. If the marker is moved, the user moves the marker (step S209). S210), the process returns to step 202. In step S209, the question is displayed on the display unit 13 and the user is allowed to select.

Next, since imaging at the imaging positions 91 and 92 has not been completed at this point, the process returns to step S202 without moving the marker 20. Then, the user moves the image processing apparatus 1 to the imaging position 91 (step S202), and executes steps S203 to S208, whereby the tertiary of the measurement positions 910 and 911 converted into the three-dimensional coordinate system of the marker 20 is obtained. Get original location information. At this point in time, since all the measurement positions have not been specified yet, the process proceeds to step S209. Since the imaging at the imaging position 92 is not completed, the process returns to step S202. Similarly to the above, the user moves the image processing apparatus 1 to the imaging position 92 (Step S202), and executes Steps S203 to S208, whereby the measurement position 920 converted to the three-dimensional coordinate system of the marker 20, The three-dimensional position information 921 and 922 is acquired.

Next, since all the measurement positions have not yet been specified at this point, the process proceeds to step S209. Since the imaging of the lower region in FIG. 9A and the designation of the measurement position have been completed, the user moves the marker 20 to the position of the marker 22 in FIG. 9B (step S210). At this time, by specifying the marker 20 to be moved on the image of the imaging position 92 displayed on the display unit 13, information indicating which marker is the moving marker is input to the image processing unit 11. .

In actual processing, it is not always necessary to confirm step S209. If it is determined in step S208 that the designation of the measurement position has not been completed, and the marker needs to be moved, the imaging is performed before step 203. The user may move the marker. At this time, it is necessary to input information indicating which marker has been moved to the image processing unit 11a by selecting it on the image.

Next, the user moves the image processing apparatus 1 to the imaging position 93 (Step S202), and executes Steps S203 to S208, whereby the measurement positions 930, 931, converted to the three-dimensional coordinate system of the marker 20, are obtained. 932 three-dimensional position information is acquired. Since all the measurement positions have not been specified even at this time, the process proceeds to step S209 again. Since the marker 20 has already been moved, the process returns to step S202, and the user moves the image processing apparatus 1 to the imaging position 94 (step S202). Then, Steps S203 to S208 are executed, and the three-dimensional position information of the measurement positions 940 and 941 converted into the three-dimensional coordinate system of the marker 20 is acquired. Since all the measurement positions have been designated by the above processing, the process proceeds to step S211.

In step S211, the overhead view creation / display control unit 11a-6 projects the three-dimensional position information of all measurement positions onto the XZ plane of the marker 20, and represents the two-dimensional position represented by the two-dimensional coordinate system of the XZ axis. Calculate the information and create an overhead view. Furthermore, the overhead view creation / display control unit 11a-6 connects the measurement positions and displays the overhead view as shown in FIG. 10B on the display unit 13 (step S211).

While confirming the bird's-eye view image displayed on the display unit 13, the user designates an erroneously connected portion or a non-connected portion by the input unit 14, and the overhead view correction control unit 11a-7 The overhead view is corrected based on the designation information (step S212). The corrected overhead view is as shown in FIG. 10C.

Based on the connection information of the measurement position in the corrected overhead view, the measurement range region division processing unit 11a-8 divides the measurement range into triangular regions (step S213), and creates an overhead view as shown in FIG. 10D. Is displayed on the display unit 13. If the region division is incorrect, the user designates the erroneous portion by the input unit 14, and the measurement range region division processing unit 11a-8 performs correction based on the designation information.

Then, based on the region division in step S213, the area calculation unit 11a-9 calculates an area for each divided region, and adds all the calculation results to calculate the area of the entire measurement range (step S214). Then, the display control unit 11a-10 displays the calculated result on the display unit 13 (step S215).

According to the above processing procedure, the image processing apparatus 1 according to the present embodiment can perform measurement using two markers even in a measurement range where there is a shielding area such as a corridor including a corner.

In the above processing procedure, the bird's-eye view is created in step S211 after all the measurement positions are specified, but the present invention is not limited to such processing. For example, each time a measurement position is designated, an overhead view is created only from the designated measurement position and displayed on the display unit 13. By doing in this way, it is possible to perform measurement while the user confirms which range of the measurement position has been specified and which range has not been specified. Therefore, since the user can determine the measurement position and measurement range while checking the overhead view, it is possible to eliminate the omission of the measurement position.

(Step S206: Marker association)
In step S206, a marker is detected from the captured image and two markers are associated with each other. The image processing unit 11a distinguishes the first set reference marker from the other non-reference markers. Then, association is performed by calculating a transformation matrix of a three-dimensional coordinate system from the non-reference marker to the reference marker.

In the following, marker association will be described using the example of FIG. 9 described above.

First, the image processing unit 11a sets, as a reference marker, the marker 20 that appears in the image at the imaging position 90 that is first imaged. However, the reference marker may be specified and set on the image by the user. For example, when a plurality of markers are shown in one image, the user selects the reference marker and designates it using the input unit 14. It is good to do. Alternatively, a priority order may be given as the first reference marker, and it may be set so as to be discriminated by color or the like. Since only the marker 20 is shown in the image at the imaging position 91, it is associated with the same as the reference marker.

Next, since the markers 20 and 21 are shown in the image at the imaging position 92, the image processing unit 11a detects the two markers, and which is the reference marker due to a difference in coloration of the two markers or the like. Is estimated. Then, a conversion matrix for converting from the three-dimensional coordinate system of the marker 21 to the three-dimensional coordinate system of the marker 20 as the reference marker is calculated by a method described later.

Next, in the image at the imaging position 93, the marker 21 and the marker 22 that has moved the marker 20 are shown. The marker 22 is the same as the marker 20, but since it is known which marker the user has moved in step S209 and step S210, the two markers detected from the image at the imaging position 93 are both standards. It can be seen that it is not a marker (non-reference marker). It can be seen that the marker 21 detected in the image at the imaging position 93 has not moved since the time when the image was captured at the imaging position 92, and the marker 22 has been moved from the time when the image was captured at the imaging position 92. Yes.

The image processing unit 11a calculates a conversion matrix for converting the three-dimensional coordinate system from the marker 22 to the marker 21 by a method described later. At this time, since the transformation matrix of the three-dimensional coordinate system from the marker 21 to the marker 20 has already been calculated as described above, the three-dimensional coordinate system from the marker 22 to the marker 20 that is the reference marker based on the two transformation matrices. Can be calculated.

Finally, the marker 22 is shown in the image at the imaging position 94, and the marker 22 is obtained by moving the marker 20, and it is known that the marker 22 has not moved since the image was captured at the imaging position 93. Therefore, by applying the three-dimensional coordinate system conversion matrix from the marker 22 to the marker 20, the conversion to the reference marker is possible.

By the above procedure, a conversion matrix for converting from the marker shown in each image to the three-dimensional coordinate system of the reference marker is calculated. That is, the marker is associated. Based on the calculated transformation matrix, coordinate system transformation of the three-dimensional position information is performed in step S207.

As described above, if the image processing unit 11a recognizes the reference marker and the non-reference marker and distinguishes the moved marker and the non-moving marker, the imaging position and the measurement position can be set to measure a wider range. Even when the number is increased, conversion to a reference marker is possible. The same conversion is possible when the number of markers is two or more. However, since it is necessary to calculate a conversion matrix to a reference marker via a marker that has not moved, it is not possible to move all the markers simultaneously. Therefore, imaging and marker movement may be repeated while alternately exchanging the marker to be moved and the marker not to be moved.

In the above description, the reference marker is selected from the markers that appear in the first captured image. However, the present invention is not limited to this, and it is only necessary to calculate a conversion matrix from each marker to the reference marker. For example, the conversion matrix of the three-dimensional coordinate system from the marker 20 and the marker 22 to the marker 21 may be calculated using the marker 21 imaged at the imaging position 92 as a reference marker. In this case, the area can be measured if the above-described overhead view and the like are prepared based on the marker 21.

(About coordinate conversion between markers)
A method for converting the three-dimensional position information from the three-dimensional coordinate system of one marker to the three-dimensional coordinate system of the other marker when imaging by measuring a plurality of markers will be described below.

For example, if two images of the marker K and the marker L are captured in the image captured at the imaging position c, the three-dimensional coordinate system with the image processing apparatus 1 at the imaging position c as a reference is represented by _Mc and the marker K. the three-dimensional coordinate system based O _K, the three-dimensional coordinate system based on the marker L and O _L. Further, when a certain point on the object to be imaged is Q, vectors to the point Q viewed from each of the three three-dimensional coordinate systems are Q _Mc , Q _OK , and Q _OL . At this time, the transformation matrices V _K and V _L into the three-dimensional coordinate system represented by each of the two markers can be calculated by using the above-described method using the equation (3). Therefore, the relationship between Q _Mc and Q _OK and Q _OL is expressed by equation (5).

Further, when the reference marker is the marker K, the transformation of the three-dimensional coordinate system from the marker L to the marker K is expressed by Expression (6).

V _KL is a transformation matrix. Note that ^-1 represents an inverse matrix.

Furthermore, after imaging at the imaging position c, when the marker L and a new marker J are imaged from other imaging positions without moving the marker L, a transformation matrix of the three-dimensional coordinate system from the marker J to the marker L V _LJ can be calculated by equation (6). At this time, _assuming that the vector from the three-dimensional coordinate system of the marker J to the point Q is Q _OJ , the conversion from the three-dimensional coordinate system of the marker J to the three-dimensional coordinate system of the marker K, which is the reference marker, is expressed by Equation (7). It is represented by

V _KJ is a transformation matrix of the three-dimensional coordinate system from the marker J to the reference marker.

As described above, a conversion matrix between two markers appearing in an image can be calculated. Furthermore, it is possible to calculate a conversion matrix for converting a three-dimensional coordinate system of a marker imaged at a different imaging position into a three-dimensional coordinate system of a reference marker via a marker that has not moved.

As described above, according to the image processing apparatus 1 of the first embodiment and the second embodiment, the coordinate system between a plurality of markers can be converted based on the image information, and all the three-dimensional position information can be converted to the reference markers. Can be integrated into the coordinate system. Therefore, it is possible to integrate and associate the positional relationship of the subject appearing in a plurality of images captured at different positions into one coordinate system. Thereby, since a user can grasp | ascertain correctly positional relationships, such as a measurement range, and can eliminate the effort which integrates a result himself, a user's burden can be eased.

In addition, since the measurement position specified by the user is corrected to a characteristic position such as the corner or edge of the subject by image recognition technology, even if the user's specified accuracy is low, measurement position deviation is eliminated and measurement is performed with high accuracy. be able to.

In addition, by creating a bird's-eye view based on the three-dimensional position information of the measurement position specified by the user, it is possible to measure while confirming the specified position by the user, thereby eliminating the omission of specification of the measurement position. it can.

(Third embodiment)
In the example described in the first embodiment and the second embodiment, measurement positions at different positions are all designated in images captured at different imaging positions. For example, when the measurement is actually performed by the measurement method described in the first embodiment, the user may specify a previously designated measurement position again while repeating the imaging and measurement position designation. It is also assumed that the same measurement position is photographed from a plurality of imaging positions, three-dimensional position information of the measurement position is calculated for each, and measurement is performed using the information calculated with the highest accuracy.

Therefore, when the user designates the same measurement position, the image processing apparatus 5 in the present embodiment uses the measurement position with the shorter distance from the imaging position for measurement. This is because, as described above with respect to the stereo system, when distance information is calculated using the stereo system, distance information is calculated with higher resolution for a subject with a shorter distance. That is, the closer the distance is, the higher the accuracy of calculating the three-dimensional position information and the accuracy of the measurement result is improved. The image processing apparatus 5 of the present embodiment has the same configuration as the image processing apparatus 1 of the first embodiment, and the processing described below is added instead of the image processing unit 11 of FIG.

Hereinafter, the image processing apparatus 5 of the present embodiment will be described with reference to the example of FIG. In the case of FIG. 9A, for example, the measurement position 921 is shown in both images of the imaging position 90 and the imaging position 92, and therefore can be specified in both images. When the user has selected the measurement position 921 in both images, the specified measurement is performed by converting the three-dimensional position information of the measurement position 921 specified in each image into the three-dimensional coordinate system of the marker 20. It can be seen that the positions are the same. The image processing unit 51 determines whether or not the measurement position has been designated when the transformation of the three-dimensional coordinate system is performed in the processing step S207. However, strictly speaking, there is a possibility that the coordinate-converted three-dimensional position information does not match due to the influence of the designation accuracy of the user. Therefore, a threshold value is set, and when the distance between the two measurement positions is equal to or less than the threshold value, it is determined that the measurement positions are the same. When the same position has been designated, the distance information at each imaging position is compared, and the measurement position with the shorter distance from each imaging position to the measurement position is prioritized.

In the case of the example in FIG. 9, the distance from the imaging position 90 to the measurement position 921 is compared with the distance from the imaging position 92 to the measurement position 921, and the distance from the imaging position 90 is short. The three-dimensional position information of the designated measurement position 921 is used for measurement.

Since the distance acquisition method is not limited to the stereo method, it is preferable that the measurement position with the highest distance calculation accuracy is selected in the image processing unit 51 in accordance with the distance acquisition method. For example, the reliability is calculated at the measurement position of each image, and the value with the highest reliability is used. The reliability includes the evaluation value of stereo matching and the difference from the peripheral distance value.

As described above, according to the image processing apparatus 5 of the present embodiment, even when the user designates the same position in three-dimensional space on different images, the calculation accuracy of the three-dimensional position information is automatically higher. Since the measurement position is selected, measurement can be performed with high accuracy based on more accurate three-dimensional position information.

Also, it is not necessary for the user to memorize all designated measurement positions, and the measurement position can be designated in each image without being conscious of it, so the burden on the user can be reduced.

(Fourth embodiment)
In the measurement method using the two markers of the second embodiment, measurement is possible even in a measurement range that does not fit within the imaging range by alternately moving the reference marker and the other marker and imaging. Said that.

On the other hand, the image processing apparatus 7 of the present embodiment performs imaging by moving at least one of two or more markers as an absolute reference marker that is not fixed and moved, and moving other markers. Then, the three-dimensional position information of all designated measurement positions is converted into a three-dimensional coordinate system based on the absolute reference marker. This reduces the amount of processing when re-execution is necessary, such as when the imaging position or measurement position is specified incorrectly during measurement, thereby reducing the burden on the user. Note that the image processing device 7 of the present embodiment can be realized by the same configuration as the image processing device 1 of the first and second embodiments or the image processing device 5 of the third embodiment. In the following description, it is assumed that the image processing device 7 has the same configuration as the image processing device 1.

Hereinafter, as an example, a method for measuring the area using the absolute reference marker and the other two movement markers by the image processing apparatus 7 of the present embodiment will be described.

First, the user places a marker in the measurement range and takes an image with the acquisition unit 10 of the image processing device 7, and the image processing unit 11 of the image processing device 7 detects an absolute reference marker that appears in the first imaged image. The user repeats imaging while alternately moving markers (non-reference markers) other than the absolute reference marker as in the first embodiment, and designates all vertex positions in the measurement range. The image processing unit 11 calculates the three-dimensional position information of all the designated measurement positions, and further converts all the three-dimensional position information into a three-dimensional coordinate system based on the absolute reference marker. Based on the three-dimensional position information obtained by converting the coordinate system, the area is calculated in the same manner as the method described in the first embodiment. For calculating the area, the image processing unit 11 creates an overhead view based on the three-dimensional position information, and the user confirms the information displayed on the display unit 13 of the image processing device 7 while checking the information displayed on the input unit of the image processing device 7. The area to be measured is designated and corrected in step 14, and the image processing unit calculates the area.

When imaging is performed while moving the marker in the above processing, the image processing unit 11 detects whether or not the absolute reference marker is included in the captured image, and if detected, the timing is stored in the image processing device 7. Store in the unit 12.

The first detection timing is when the absolute reference marker is first detected, and the second detection timing is when the absolute reference marker is detected next. Thereafter, the detection timing is stored in the storage unit every time it is detected.

For example, in the measurement with a total of six times of imaging, if an absolute reference marker is detected in each of the images acquired in the first, third, and fifth imaging, the absolute reference in the first captured image Detection of the marker is the first detection timing, detection at the third captured image is the second detection timing, and detection at the fifth captured image is the third detection timing.

At this time, the measurement position designated in the first to third captured images between the first and second detection timings is coordinate-converted via a marker appearing in the first to third captured images. The three-dimensional position information of these measurement positions is converted into a three-dimensional coordinate system based on the absolute reference marker at the first detection timing.

Similarly, the measurement position designated in the 3rd to 5th captured images between the second and third detection timings is coordinate-converted via a marker appearing in the 3rd to 5th captured images. The three-dimensional position information of these measurement positions is converted into a three-dimensional coordinate system based on the absolute reference marker at the second detection timing.

Then, the measurement positions specified in the fifth and sixth captured images are coordinate-converted via markers appearing in the respective captured images, and converted into a three-dimensional coordinate system based on the absolute reference marker at the third detection timing. Is done.

If there is a fourth or later detection timing, the three-dimensional position information of the measurement position designated between the two detection timings is reflected in the first detected image in the same manner as described above. Convert to a coordinate system based on an absolute reference marker.

As described above, the image processing apparatus 7 according to the present embodiment varies the image used for coordinate system conversion, that is, the marker used during coordinate conversion, according to the detection timing of the absolute reference marker, and uses the absolute reference marker as a reference. Convert to a 3D coordinate system.

Since the position of the absolute reference marker is always fixed even if the captured images are different, all converted three-dimensional position information is information based on the same coordinate system. Thereby, the image processing unit can calculate the area by associating all the three-dimensional position information.

Therefore, for example, when it is desired to correct the measurement position designated in the first to third imaging, it is possible to re-designate the measurement position by imaging only the necessary part without redoing the fourth to sixth imaging. Become.

As described above, by performing coordinate conversion by dividing at the detection timing of the overall reference marker, even if there is an error in the designation of the imaging position or measurement position, it is possible to correct only the detection timing section. .

For example, even when the installation position of the marker moved at a certain timing is wrong, the marker installation and imaging can be performed again only in the detection timing section including the image obtained by imaging the erroneously moved marker.

In addition, for example, if it is found that there is an imaging omission in a part of the measurement range after all the imaging is completed, the imaging omission portion should be additionally imaged while copying the absolute reference marker and the non-reference marker. That's fine. If the absolute reference marker has not moved, the three-dimensional position information of the measurement position specified by the additional captured image can also be converted into a coordinate system based on the same absolute reference marker. As described above, even when there is a measurement omission, it is possible to easily specify additional imaging and a measurement position.

Hereinafter, a specific processing procedure will be described using the example of FIG.

FIG. 12 shows an overhead view of a long passage with a corner such that two passages are connected. FIG. 12 shows an absolute reference marker 60 and movable non-reference markers 61 and 62. In addition, the imaging positions 70, 71, 72, 73, 74, 75, 76, 77, 78, and 79 are indicated by broken-line circle shapes, and broken-line arrows extending from the respective imaging positions indicate the imaging directions. In addition, alternate long and short dash line arrows L11, 12, 13, and 14 indicate the directions in which the markers 61 and 62 are moved, and the positions of the moved markers 61 and 62 are indicated by broken line figures at the ends of the alternate long and short dash arrows, respectively. ing. The markers 60, 61, 62 all have the same shape as the marker 2 shown in FIGS. 2 and 6 of the first embodiment. However, in order to make it easier to detect that the markers are different in the image to be captured, the color scheme of the markers is different. Further, a cross indicates a measurement position, and one of them is a measurement position 80.

In the case of the example in FIG. 12, the user first places the absolute reference marker 60 near the center of the measurement range. Also, the markers 61 and 62 are installed at the positions shown in the figure. The absolute reference marker 60 is not moved until it is confirmed that there is no correction such as re-imaging after completion of the measurement.

Next, the absolute reference marker 60 and the marker 61 are sequentially imaged at the imaging position 70, the markers 61 and 62 at the imaging position 71, and the marker 62 at the imaging position 72.

Subsequently, the marker 62 is moved in the direction of the one-dot chain line arrow L11, and the markers 61 and 62 are sequentially imaged at the imaging position 73, and the marker 62 is sequentially imaged at the imaging position 74.

In the image captured at the imaging positions 70 to 74, the measurement position of the right region of the overhead view of FIG.

Next, the markers 61 and 62 are moved from the right side region to the left side region in the overhead view in the direction of dashed-dotted arrows L12 and L13. The absolute reference marker 60 and the marker 61 are sequentially imaged at the imaging position 75, the markers 61 and 62 are sequentially imaged at the imaging position 76, and the marker 62 is sequentially imaged at the imaging position 77.

Subsequently, the marker 62 is moved from the lower left portion of the overhead view to the upper left portion in the direction of a one-dot chain line arrow L14, and the markers 61 and 62 are sequentially imaged at the imaging position 78 and the marker 62 is sequentially imaged at the imaging position 79.

In the image captured at the imaging positions 75 to 79, the measurement position of the left region of the overhead view of FIG. 12 is designated.

Here, the absolute reference marker 60 is reflected in the images taken at the imaging positions 70 and 75, and the image processing unit 11 detects the absolute reference marker 60 and stores it in the storage unit 12 as the first and second detection timings. Remember.

The marker is moved and imaged by the above procedure, and the user designates the measurement position in the captured image. Marker detection and measurement position designation are performed in the same manner as described in the first embodiment.

The image processing unit 71 converts the measurement position designated in each image into a three-dimensional coordinate system with the absolute reference marker 60 as a reference. At this time, the image captured between the first detection timing and the second detection timing, that is, the measurement positions designated in the images from the imaging positions 70 to 75 are the markers shown in the images from the imaging positions 70 to 75. The coordinate conversion is performed via. In addition, the measurement positions designated in the images captured after the second detection timing, that is, in the images captured from the imaging positions 75 to 79, are coordinated via the markers shown in the images from the imaging positions 75 to 79. Conversion is performed.

That is, the image used for coordinate conversion, that is, the marker used during coordinate conversion differs between the right region and the left region in the overhead view of FIG.

The image processing unit 11 creates an overhead view based on the coordinate-converted three-dimensional position information and displays it on the display unit 13. The user specifies and corrects the measurement range and the like with the input unit 14 while confirming the display, and the image processing unit 11 calculates the area. Each processing such as coordinate conversion, overhead view creation, user designation, and area calculation is performed in the same manner as the method described in the first embodiment.

Here, for example, when the designated position of the measurement position 80 is to be corrected, or when the measurement position 80 is forgotten to be designated, the markers 61 and 62 are set again at the positions indicated by the solid line figure in FIG. Then, imaging is performed at the imaging positions 70, 71, and 72, and the measurement position is designated again. Then, the three-dimensional position information of the measurement position is coordinate-converted to the coordinates of the absolute reference marker 60 and integrated with other designated measurement positions to calculate the area. As described above, since the absolute reference marker 60 is not moved, only necessary portions can be corrected or added.

As described above, according to the image processing device 7 of the present embodiment, the three-dimensional position information is converted into the three-dimensional coordinate system of the absolute reference marker that does not move, so that the measurement position and the like can be corrected. This eliminates the need to redo all measurements, improving the convenience for the user. Furthermore, even if there is a measurement omission, it is not necessary to redo all measurements from the beginning, and additional imaging and measurement positions can be specified, reducing the burden on the user.

In the above example, the case where one absolute reference marker and two movable non-reference markers are used has been described. However, the present invention is not limited to this. For example, measurement may be performed by imaging while moving one non-reference marker around one absolute reference marker, or three or more non-reference markers may be used. Furthermore, in the case of a large measurement range, measurement is performed with one absolute reference marker and a plurality of moving markers for each area, and finally coordinate conversion between the plurality of absolute reference markers is performed to integrate the three-dimensional position information. You may make it measure.

In each of the embodiments described above, the configuration and the like illustrated in the accompanying drawings are not limited to these, and can be appropriately changed within a range in which the effects of the present invention are exhibited. In addition, various modifications can be made without departing from the scope of the object of the present invention.

Each component of the present invention can be arbitrarily selected, and an invention having a selected configuration is also included in the present invention.

The

image processing apparatus

1, 5 or 7 described in each of the above embodiments includes, for example, a processor such as a DSP (Digital Signal Processor) and a CPU (Central Processing Unit), a main storage device such as a RAM (Random Access Memory), and the like. The processing of each processing unit described above can be realized by executing a program stored in the storage device. Alternatively, the above processing may be realized by hardware by providing a programmable integrated circuit such as an FPGA (Field （Programmable Gate Array) or an integrated circuit dedicated to the above processing.

The acquisition unit in each of the above embodiments acquires image information and distance information by the two imaging units and the distance calculation unit. However, the distance between the image information and the subject corresponding to each pixel of the image information. Any device may be used as long as information is input to the image processing unit. For example, image information and distance information may be acquired by an imaging device and a distance measuring device. As the distance measuring device, for example, a technique using infrared rays represented by TOF (Time of Flight) method may be used. The TOF method irradiates light such as infrared light that cannot be visually recognized by humans from a light source such as an LED (Light Emitting Diode), and measures the elapsed time from when the light is irradiated until it is reflected by the surface of the subject and returned. . The distance to the subject is measured based on this elapsed time. By measuring the measurement for each finely divided area, it is possible to measure not only one point but also various parts of the subject. As a method of measuring the above elapsed time, a method of directly measuring the time until the laser beam is pulsed and reflected by the surface of the object and returned, or the irradiation light (for example, infrared) is modulated and irradiated There is a method of calculating based on the phase difference between the phase of the light and the phase of the light reflected and returned.

In the

image processing apparatuses

1, 5, or 7 described in the above embodiments, the image is acquired by the acquisition unit, the distance information is calculated by the distance calculation unit, and then the distance information is input to the image processing unit. However, after the image is acquired, the user may specify a measurement position on the image, and the distance calculation unit may calculate distance information for only the specified position. As a result, it is possible to measure without calculating distance information of a subject in a range not used for measurement, and thus it is possible to reduce the processing amount.

1, 5, 7... Image processing device,
DESCRIPTION OF SYMBOLS 10 ... Acquisition part, 100, 101 ... Imaging part, 102 ... Distance calculation part,
11, 11a, 51 ... image processing unit, 11-1, 11a-1, ... measurement position designation input receiving unit,
11-2, 11a-2... 3D position information calculation unit,
11-3, 11a-4... 3D position information conversion unit,
11-4, 11a-9... Area calculation unit,
11-5, 11a-10 ... display control unit 11a-3 ... inter-marker association processing unit,
11a-5 ... Marker movement instruction reception part,
11a-6 ... overhead view creation display control unit,
11a-7 ... overhead view correction control unit,
11a-8... Measurement range area division processing unit,
12 ... storage unit,
13 ... display part,
14 ... input part,
2, 20, 21, 22, K, L, 61, 62 ... markers,
60 ...

Absolute reference markers

3, 4 ... Subject,
_{_{M, M c, O, O}} K, O L ... three-dimensional coordinate system,
m ... 2D coordinate system,
P, Q: Points on the subject,
i, j, k ... unit vector,
P _M , O _M , P _O , g0, g1, g2, g3, Q _Mc , Q _OK , Q _OL ...

vector

70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 90, 91 , 92, 93, 94 ... imaging positions a1, a2, b1, b3, 80, 900, 901, 910, 911, 920, 921, 922, 930, 931, 932, 940, 941 ... measurement positions

All publications, patents and patent applications cited in this specification shall be incorporated into the present specification as they are.

Claims

An image processing apparatus that calculates three-dimensional position information of measurement positions on a subject based on a plurality of pieces of image information obtained by imaging a subject to be measured and a marker at a plurality of imaging positions.
The marker has a plurality of pointers, and the positional relationship between the plurality of pointers represents a three-dimensional coordinate system,
A three-dimensional position information calculation unit that calculates the three-dimensional position information represented in a three-dimensional coordinate system for each of the plurality of imaging positions based on the plurality of image information;
A three-dimensional position information conversion unit that converts the coordinate system representing the three-dimensional position information calculated by the three-dimensional position information calculation unit from the three-dimensional coordinate system for each imaging position to the three-dimensional coordinate system represented by the marker When,
An image processing apparatus comprising:
The marker includes a plurality of markers including a first marker representing a first three-dimensional coordinate system and a second marker representing a second three-dimensional coordinate system different from the first three-dimensional coordinate system. There is
The three-dimensional position information calculation unit includes the first three-dimensional coordinate system based on image information in which the first marker and the second marker are simultaneously imaged among the plurality of image information. Calculating a first positional relationship with the second three-dimensional coordinate system;
The three-dimensional position information conversion unit represents the three-dimensional position information calculated by the three-dimensional position information calculation unit based on image information in which the second marker is captured among the plurality of pieces of image information. A coordinate system is converted into the second three-dimensional coordinate system, and the coordinate system is further changed from the second three-dimensional coordinate system to the first three-dimensional coordinate system based on the first positional relationship. The image processing apparatus according to claim 1, wherein conversion is performed.
The plurality of markers includes a third marker representing a third three-dimensional coordinate system different from the first three-dimensional coordinate system and the second three-dimensional coordinate system,
The three-dimensional position information calculation unit includes the second three-dimensional coordinate system based on image information in which the second marker and the third marker are simultaneously imaged among the plurality of image information. Calculating a second positional relationship with the third three-dimensional coordinate system;
The three-dimensional position information conversion unit represents the three-dimensional position information calculated by the three-dimensional position information calculation unit based on image information in which the three markers are imaged among the plurality of pieces of image information. A coordinate system is converted into the third three-dimensional coordinate system, and the coordinate system is further changed from the third three-dimensional coordinate system to the second three-dimensional coordinate system based on the second positional relationship. The coordinate system is converted, and further, the coordinate system is converted from the second three-dimensional coordinate system to the first three-dimensional coordinate system based on the first positional relationship. Image processing device.
The image processing apparatus according to claim 3, wherein the third marker is the first marker whose installation position has been moved.
The three-dimensional position information calculation unit calculates a third positional relationship between a three-dimensional coordinate system represented by the marker and a two-dimensional coordinate system representing a surface on the subject including the measurement position;
The three-dimensional position information whose coordinate system has been converted by the three-dimensional position information conversion unit is projected onto a surface on the subject or a plane parallel to the surface on the subject based on the third positional relationship. 5. The image according to claim 1, further comprising an overhead view creation display control unit that calculates two-dimensional position information and creates an overhead view based on the two-dimensional position information. Processing equipment.
The overhead view creation display control unit
The three-dimensional position information calculation unit calculates the three-dimensional position information of each of a plurality of different measurement positions on the subject,
Each time the coordinate system representing the three-dimensional position information of each of the plurality of measurement positions is converted by the three-dimensional position information conversion unit,
Two-dimensional position information obtained by projecting the three-dimensional position information whose coordinate system has been converted to the surface on the subject or a surface parallel to the surface on the subject is calculated, and the two-dimensional position information is The image processing apparatus according to claim 5, wherein the overhead view is updated by creating an overhead view based on the image.
The three-dimensional position information calculation unit is configured to determine the three-dimensional position based on each of the two or more pieces of image information for the same measurement position on the subject that is captured in two or more pieces of image information of the plurality of pieces of image information. Information is calculated, and among the calculated three-dimensional position information of the same measurement position, the one having the highest distance resolution is calculated as the three-dimensional position information of the same measurement position. Item 7. The image processing apparatus according to any one of Items 1 to 6.
An image processing method for calculating three-dimensional position information of a measurement position on a subject based on a plurality of pieces of image information obtained by imaging a subject to be measured and a marker at a plurality of imaging positions,
The marker has a plurality of pointers, and the positional relationship between the plurality of pointers represents a three-dimensional coordinate system,
A three-dimensional position information calculating step for calculating the three-dimensional position information represented in a three-dimensional coordinate system for each of the plurality of imaging positions based on the plurality of image information;
A three-dimensional position information conversion step for converting the coordinate system representing the three-dimensional position information calculated in the three-dimensional position information calculation step from the three-dimensional coordinate system for each imaging position to the three-dimensional coordinate system represented by the marker. When,
An image processing method comprising:
The marker includes a plurality of markers including a first marker representing a first three-dimensional coordinate system and a second marker representing a second three-dimensional coordinate system different from the first three-dimensional coordinate system. There is
The three-dimensional position information calculation step includes the first three-dimensional coordinate system based on image information in which the first marker and the second marker are simultaneously imaged among the plurality of image information. Calculating a first positional relationship with the second three-dimensional coordinate system;
The three-dimensional position information conversion step represents the three-dimensional position information calculated in the three-dimensional position information calculation step based on image information in which the second marker is captured among the plurality of image information. A coordinate system is converted into the second three-dimensional coordinate system, and the coordinate system is further changed from the second three-dimensional coordinate system to the first three-dimensional coordinate system based on the first positional relationship. 9. The image processing method according to claim 8, wherein conversion is performed.
A program for causing a computer to execute the image processing method according to claim 8 or 9.