JP6087224B2 - Designated point display image generator - Google Patents

Description

  The present invention relates to a technique for calculating three-dimensional coordinates of a designated point from a photographed image and displaying the designated point on another photographed image.

  Since the Great East Japan Earthquake, attention has been paid to seismic repair of the ceiling. When performing earthquake-proof repairs on the ceiling, the back of the ceiling is photographed in advance to investigate damage and damage status (see Patent Document 1).

JP 2007-1003003 A

  The present inventors are developing a technique for displaying a virtual reinforcing material or the like on a photographed image of the back of the ceiling and presenting the state after the seismic retrofit. Here, in order to accurately display the virtual reinforcement material, it is necessary to specify the start point and end point of the reinforcement material, but since there are various equipment and beams on the back of the ceiling, one of the start point and the end point must be specified. It may not be possible to specify. In this case, it is difficult to accurately calculate the three-dimensional coordinates and the three-dimensional distance of the reinforcing material.

  The present invention has been made in view of the above circumstances, and a designated point display image capable of suitably displaying a designated point in a photographed image on another photographed image in an environment where obstacles in photographing exist. It is an object to provide a generation device.

In order to solve the above-mentioned problem, the designated point display image generation device of the present invention is photographed by the first image data photographed by the first photographing device and the second photographing device separated from the first photographing device by a predetermined distance. Based on the first image data, the second image data and the third image data captured by the third image capturing device that is separated from the first image capturing device by a predetermined distance are stored in association with each other. A panorama image data generation unit that generates panorama image data, an image data output unit that outputs image data to a display device, and each of the second image data and the third image data according to an operation of the input device by a user A designated point determining unit for determining a designated point for display, and a designated point for display and an imaging device corresponding to the designated point for display based on the designated point for display A three-dimensional coordinate calculation unit that calculates a three-dimensional coordinate of a specified point that is an intersection of two straight lines, and the image data storage unit includes one of the first imaging device and one of the second imaging device. And one of the first image data, one of the second image data and one of the third image data captured by one of the third imaging devices is stored in association with each other, and the other of the first image data is stored. One imaging device, the other second imaging device, and the other third imaging device, the other first image data, the other second image data, and the other third image data are associated with each other. The three-dimensional coordinate calculation unit is based on one of the second image data determined by the specified point determination unit and the first display specified point in the third image data, First Calculates the three-dimensional coordinates of a specified point of the image data output unit, based on the three-dimensional coordinates of the first designated point, displaying a first display for the specified point in the other of said first image data The panoramic image data based on the other first image data is processed and output, and the second image data of the other determined by the designated point determination unit and the second image data of the other third image data are output. Based on the display designated point, the three-dimensional coordinate of the second designated point is calculated, and the three-dimensional coordinate of the first designated point and the three-dimensional coordinate of the second designated point are input to the image data output unit. And processing and outputting the panoramic image data based on the first image data so as to display the first display designated point and the second display designated point in the one first image data. about It is characterized by.

  According to this configuration, the three-dimensional coordinates of the designated point are calculated based on the two photographed image data, and the other photographed image data is processed so as to display the designated point based on the calculated three-dimensional coordinates. The designated point can be suitably displayed even in an environment such as the back of a ceiling where an obstacle in photographing exists.

  The designated point display image generating device is configured to obtain a cubic of the first designated point and the second designated point based on the three-dimensional coordinate of the first designated point and the three-dimensional coordinate of the second designated point. It is desirable to further include a three-dimensional distance calculation unit that calculates the original distance.

  According to such a configuration, since compound eye photography is performed from two viewpoints, a three-dimensional distance between two designated points can be suitably calculated even in an environment such as a ceiling where an obstacle exists in photography. For example, a virtual member constructed between two designated points can be suitably displayed.

The image data output unit, one of the first to display a virtual member that is bridged to the a first display for the specified point and the second display specified point in one of the first image data It is desirable to process and output panoramic image data based on the image data.

  According to such a configuration, since compound-eye photography is performed from two viewpoints, a virtual member installed between two designated points can be suitably displayed even in an environment such as a ceiling behind an obstacle in photographing. .

  According to the present invention, it is possible to suitably calculate a three-dimensional coordinate or a three-dimensional distance from a photographed image in an environment where an obstacle in photographing exists.

1 is a block diagram illustrating a panoramic image display system according to an embodiment of the present invention. It is a schematic diagram for demonstrating arrangement | positioning of a board model. (A) is a figure for demonstrating the calculation principle of a 1st three-dimensional coordinate, (b) is an xz top view of (a). (A) is a figure for demonstrating the rotation around an X-axis, (b) is a figure for demonstrating the rotation around a Y-axis. It is a figure for demonstrating the operation example of a panoramic image display system, and is a schematic diagram which shows the state which has arrange | positioned each compound-eye imaging device on the back of a ceiling. (A) is the schematic diagram which expands and shows the 1st inspection port side of FIG. 5, (b) is the schematic diagram which expands and shows the 2nd inspection port side of FIG. It is a figure for demonstrating the operation example of a panoramic image display system, and is a figure which shows the example of a monocular panoramic image from the 1st inspection port side. (A) (b) is a figure for demonstrating the operation example of a panoramic image display system, and is a figure which shows the example of a screen of the compound-eye image from the 1st inspection port side. It is a figure for demonstrating the operation example of a panoramic image display system, and is a figure which shows the example of a monocular panoramic image from the 2nd inspection port side. (A) (b) is a figure for demonstrating the operation example of a panoramic image display system, and is a figure which shows the example of a screen of the compound-eye image from the 2nd inspection port side. It is a figure for demonstrating the operation example of a panoramic image display system, and is a figure which shows the example of a monocular panoramic image from the 1st inspection port side.

  Hereinafter, an example in which the designated point display image generation apparatus of the present invention is applied to a panoramic image display of a ceiling of a building will be described with reference to the drawings as appropriate. Similar parts are denoted by the same reference numerals, and redundant description is omitted.

  As shown in FIG. 1, a panoramic image display system 1 according to an embodiment of the present invention is a system that generates and displays an omnidirectional panoramic image inside a building, for example, one compound eye photographing unit 10A and the other. The compound eye photographing unit 10 </ b> B, the input device 20, the designated point display image generating device 30, and the display device 40 are provided.

<Composite photographing unit 10A>
One compound eye photographing unit 10A is a unit that photographs the back of the ceiling from the first inspection port. The first photographing device 11a, the second photographing device 11b, the third photographing device 11c, and the photographing devices 11a, 11b, The drive part 12 for changing the imaging | photography direction of 11c, and the control part 13 which drive-controls each imaging device 11a, 11b, 11c and the drive part 12 are provided.
The first photographing device 11a, the second photographing device 11b, and the third photographing device 11c are monocular digital cameras, respectively, and the focal lengths of the second photographing device 11b and the third photographing device 11c are f, and each photographing device 11a, The second imaging device 11b, the first imaging device 11a, the first imaging device 11a from the left so that the optical axes of 11b and 11c are parallel to each other and the distance between the optical axes of the adjacent imaging devices 11a to 11c is the distance d / 2. They are arranged in the order of the three photographing devices 11c, and are unitized while maintaining this arrangement. In addition, the focal distance of the 1st imaging device 11a may differ from the focal distance of the other 2nd imaging device 11b and the 3rd imaging device 11c, and may be the same. Here, the 1st picked-up image data by the 1st imaging device 11a is for producing | generating a monocular panoramic image, the 2nd picked-up image data by the 2nd imaging device 11b, and the 3rd picked-up image by the 3rd imaging device 11c. The data is for calculating the three-dimensional coordinates of the designated point.

The control unit 13 rotates the driving unit 12 to direct the image capturing apparatuses 11a, 11b, and 11c in a desired image capturing direction, and controls the shutters of the image capturing apparatuses 11a, 11b, and 11c in this state to perform image capturing. The captured image data relating to the image is output to the designated point display image generation device 30. In the present embodiment, the control unit 13 rotates the compound eye photographing unit 10A by a predetermined angle with the first photographing device 11a as the rotation center, and causes the photographing devices 11a, 11b, and 11c to photograph.
In addition, the control unit 13 assigns an ID based on the shooting order (that is, the shooting direction) to the shot image data, and outputs the shot image data to which the ID is given to the designated point display image generation device 30. Here, the 1st picked-up image data regarding the 1st picked-up image image | photographed by the 1st imaging device 11a, the 2nd picked-up image data regarding the 2nd picked-up image image | photographed by the 2nd image pick-up device 11b, and the 3rd image pick-up device 11c image | photographed. Among the third photographed image data relating to the third photographed image, those having the same photographing direction are assigned the same ID and associated. Further, the second photographed image data and the third photographed image data by the second photographing device 11b and the third photographing device 11c are the X-axis and Y-axis among the coordinates of the first three-dimensional coordinate system to be described later in each photographed image. Corresponds to the coordinate values in the axial direction.

<Composite imaging unit 10B>
The other compound eye photographing unit 10B is a device that photographs the back of the ceiling from the second inspection port, and like the compound eye photographing unit 10A, the first photographing device 11a, the second photographing device 11b, and the third photographing device 11c, The drive part 12 and the control part 13 are provided.

  Here, each compound eye photographing unit 10A, 10B and a designated point display image generating device 30 to be described later are connected so as to be capable of wireless communication using an SD card or the like. Each captured image captured by the units 10A and 10B can be received in real time and displayed on the display device 40.

<Input device 20>
The input device 20 includes a keyboard, a mouse, and the like, and in accordance with a user operation, a designated point display image generation device 30 that issues an image data designation instruction for a monocular panoramic image, a designated point instruction for the second photographed image and the third photographed image, and the like. Output to.

<Specified Point Display Image Generating Device 30>
The designated point display image generation device 30 includes a CPU, a ROM, a RAM, an input / output circuit, and the like. As a functional unit, a captured image data storage unit 31, a map storage unit 32, and a monocular panoramic image data generation unit 33 are provided. An image data output unit 34, an image data designation acquisition unit 35, a designated point determination unit 36, a three-dimensional coordinate calculation unit 37, and a three-dimensional distance calculation unit 38.

<< Photographed Image Data Storage Unit 31 >>
In the captured image data storage unit 31, a plurality of captured image data relating to a plurality of captured images output from the control unit 13 of each compound-eye imaging device 10A, 10B is input and stored. Each captured image data includes the corresponding three-dimensional coordinates (second three-dimensional coordinates described later) of the monocular imaging device at the time of imaging, the imaging direction, the focal length, the plane equation of the captured image, and the plane equation. The ranges of x, y, and z are stored in association with each other.

≪Map storage unit 32≫
The map storage unit 32 is used when generating a monocular panoramic image based on a plurality of first photographed images (first photographed image data) of the first photographing device 11a of any one of the compound eye photographing units 10A and 10B. A monocular panoramic map is stored in advance.

≪Monocular panoramic image data generation unit 33≫
The monocular panoramic image data generation unit 33 reads a plurality of first captured image data output from the captured image data storage unit 31, and generates monocular panoramic image data related to a monocular panoramic image based on the read captured image data. To do.
Taking the case of using the first captured image data captured by the first imaging device 11a of the compound eye unit 10A as an example, the monocular panoramic image data generating unit 33 is a monocular panoramic map 50 (prestored in the map storage unit 32). 2), and the first captured image data captured by the first imaging device 11a is mapped to the plate model 51 of the monocular panorama map 50, thereby generating monocular panoramic image data.

  Here, the panorama map 50 will be described with reference to FIG. As shown in FIG. 2, the panoramic map 50 is configured by arranging a plurality of plate models 51 in a spherical shape. The center of the panorama map 50 (a sphere composed of a plurality of plate models 51) corresponds to the installation position of the first imaging device 11a, and each plate model 51 corresponds to each first captured image by the first imaging device 11a. It corresponds to the shooting range. That is, the number of plate models 51 is the same as the number of first photographed images by the first photographing device 11a, and the angle of an angle formed by two line segments respectively connecting the center of the panoramic map 50 and the centers of the adjacent plate models 51. Corresponds to the angle difference in the photographing direction by the first photographing device 11a when photographing the first photographed image adjacent at the installation position of the first photographing device 11a, and the aspect ratio of each plate model 51 is the first It corresponds to the aspect ratio of each first photographed image by the photographing device 11a, and the distance from the center of the sphere formed by the plurality of plate models 51 to the plate model 51 and the vertical and horizontal size of the plate model 51 are as follows. It corresponds to the focal length of 11a and the vertical and horizontal sizes of the captured image. Each plate model 51 is assigned an ID in the same manner as the first captured image data, and the monocular panoramic image data generation unit 33 is assigned to the ID and the plate model 51 assigned to the first captured image data. On the basis of the ID, the first captured image data is mapped to the corresponding plate model 51 to generate monocular panoramic image data, and the generated monocular panoramic image data is output to the image data output unit 34. Or stored in the storage unit 32. Here, the monocular panoramic image data generation unit 33 can generate monocular panoramic image data in a predetermined range centered on the viewpoint position based on the viewpoint position instruction output from the input unit 20.

  The number of plate models 51 in the panorama map 50, the vertical and horizontal sizes, and the like can be changed as appropriate according to the shooting location.

<< Image Data Output Unit 34 >>
The image data output unit 34 outputs monocular panoramic image data to the display device 40. In the present embodiment, the monocular panoramic image data output from the monocular panoramic image data generation unit 33 is output to the display device 40. Or the second captured image data and the third captured image data of one (compound eye photographing unit 10A side) or the other (compound eye photographing unit 10B side) read from the photographed image data storage unit 31 are output to the display device 40. To do.

<< Image Data Specification Acquisition Unit 35 >>
The image data designation obtaining unit 35 obtains designation of one first photographed image data from the monocular panoramic image displayed on the display device 40 in accordance with a user operation, and performs such designation result (for example, the designated first ID of the captured image data) is output to the image data output unit 34. When obtaining the designation result, the image data output unit 34 reads out the second photographed image data and the third photographed image data corresponding to the designation result from the photographed image data storage unit 31, and displays the read photographed image data on the display device 40. And the captured image corresponding to the specified result is displayed on the display device 40.

≪Specified point determination unit 36≫
The designated point determination unit 36 determines a designated point for display, and acquires designated point instructions in the second captured image and the third captured image output from the input device 20. In addition, the designated point determination unit 36 uses the obtained designated point instruction to obtain two-dimensional coordinates (X-axis and Y-axis coordinates of the designated point for display in each captured image data stored in the captured image data storage unit 31. Value) is output to the three-dimensional coordinate calculation unit 37.

≪Three-dimensional coordinate calculation unit 37≫
The three-dimensional coordinate calculation unit 37 outputs the two-dimensional coordinates of the designated point for display output from the designated point determination unit 36, the distance d between the second imaging device 11b and the third imaging device 11c, the second imaging device 11b, Based on the focal length f of the third imaging device 11c, the first three-dimensional coordinates of the designated point are calculated. Here, the first three-dimensional coordinate is a coordinate in the first three-dimensional coordinate system unique to each photographing direction with reference to the photographing direction of the photographing devices 11a to 11c, and the first three-dimensional coordinate system is a photographing tertiary. Also called the original coordinate system, the first three-dimensional coordinate is also called the photographing three-dimensional coordinate. That is, the first three-dimensional coordinate system is common to the second and third captured images having the same shooting direction, and is different between images having different shooting directions.

(First 3D coordinate calculation principle)
Here, the calculation principle of the first three-dimensional coordinate in the three-dimensional coordinate calculation unit 37 will be described with reference to FIGS. Hereinafter, the compound eye photographing unit 10A side will be described, the arrangement direction of the photographing devices 11a, 11b, and 11c is the X axis, and the optical axis direction of the photographing devices 11a, 11b, and 11c is the Z axis, and is orthogonal to the X axis and the Z axis. Let us consider a three-dimensional coordinate space of a right-handed coordinate system consisting of Y-axis. When the first imaging device 11a is arranged at the origin in the first three-dimensional coordinate system and the distance between the second imaging device 11b on the left side and the third imaging device 11c on the right side is d, the first imaging device 11a three-dimensional coordinate O _{1 of} the first three-dimensional coordinate O ₂ of the second imaging device _11b, the first three-dimensional coordinates O ₃ of the third imaging unit 11c is as follows.
O ₁ = (0, 0, 0)
O ₂ = (− d / _2, 0, 0)
O ₃ = (d / 2, 0, 0)

An object located at relative three-dimensional coordinates R = (x _R , y _R , z _R ) is imaged using the second imaging device 11b and the third imaging device 11c, and the object is captured by the second imaging device 11b. Two-dimensional coordinates N ₂ = (x ₂ , y ₂ ) of the two photographed images, that is, the first three-dimensional coordinates (x ₂ , y ₂ , −f), and the third photographed image by the third photographing device 11c. It is assumed that the two-dimensional coordinates N ₃ = (x ₃ , y ₃ ), that is, the first three-dimensional coordinates (x ₃ , y ₃ , −f).
Here, since the triangle RO ₂ O ₃ and the triangle RN ₂ N ₃ are similar, the following relationship is established.
_{x R = d (x 2 +} x 3) / {2 (x 2 -x 3 + d)}
_{y R = d (y 2 +} y 3) / {2 (y 2 -y 3 + d)}
z _R = −df / {2 (x ₂ −x ₃ + d)}
That is, the three-dimensional coordinate calculation unit 37 uses the predetermined focal length f and interval d and the value of the two-dimensional coordinates of the photographed image, so that the first three-dimensional coordinates R = ( x _R , y _R , z _R ) can be calculated.

  Further, the three-dimensional coordinate calculation unit 37 captures the first three-dimensional coordinates of the designated point output from the first three-dimensional coordinate calculation unit 37 and the photographing of the second photographing device 11b and the third photographing device 11c at the time of photographing. Based on the direction, the second three-dimensional coordinates of the designated point are calculated and output to the three-dimensional distance calculation unit 38 and the image data output unit 34. By doing so, the calculated second three-dimensional coordinates are displayed on the display device 40 described later. Here, the second three-dimensional coordinate is a coordinate in the second three-dimensional coordinate system that is common to all photographing directions, and is integrated with the first three-dimensional coordinate system, and the second three-dimensional coordinate system. Is also called a panoramic 3D coordinate system, and the second 3D coordinate is also called a panoramic 3D coordinate system. That is, the second three-dimensional coordinate system is common to all captured images.

(Second 3D coordinate calculation principle)
Here, the calculation principle of the second three-dimensional coordinate in the three-dimensional coordinate calculation unit 37 will be described. The three-dimensional coordinate calculation unit 37 sets the rotation angle around the Y axis from the reference of the shooting direction of the second imaging device 11b and the third imaging device 11c in the second three-dimensional coordinate system to θ, and the rotation angle around the X axis. When φ is set, the second three-dimensional coordinate A = (x _A , y _A , z _A ) of the designated point can be calculated by the following formula.

ここで、回転角度θ，φとしては、パノラママップ５０の板モデル５１に関連付けてマップ記憶部３２に予め記憶されたものを用いることができる。

Here, as the rotation angles θ and φ, those stored in advance in the map storage unit 32 in association with the plate model 51 of the panorama map 50 can be used.

Here, as shown in FIGS. 4A and 4B, the rotation of the angle φ around the X axis (X _R axis) of the relative three-dimensional coordinate and the rotation around the Y axis (Y _R axis) of the relative three-dimensional coordinate. by applying rotation and an angle theta, to the first three-dimensional coordinate system, X _R-axis relative three-dimensional coordinates, Y _R and Z axes (Z _R axis), X of the second three-dimensional coordinate system axis _{(X a-axis),} Y axis _{(Y a-axis)} and Z-axis and _{(Z a} axis) coincides respectively. Incidentally, phi, when viewed from the positive side of the X _A-axis counterclockwise positive, negative clockwise. θ, when viewed from the positive side of Y _A-axis counterclockwise positive, negative clockwise.
That is, the rotation around the Y axis (rotation in the pan direction) and the rotation around the X axis are converted into the first three-dimensional coordinates R = (x _R , y _R , z _R ) of the designated point in the first three-dimensional coordinate system. The second three-dimensional coordinates A = (x _A , y _A , z _A ) of the designated point are calculated by performing the rotation (rotation in the tilt direction). In the present embodiment, the origin in the second three-dimensional coordinate system and the origin in the first three-dimensional coordinate system, that is, O ₁ are the same position, and the rotation angle θ is equal to the Y-axis of the second three-dimensional coordinate system. The rotation angle φ is an angle formed by the X axis of the second three-dimensional coordinate system and the X axis of the first three-dimensional coordinate system. In the present embodiment, the X-axis in the relative three-dimensional coordinate system is always maintained in the horizontal direction, so that rotation conversion around the Z-axis is not necessary.

  The three-dimensional coordinate calculation unit 37 includes the first three-dimensional coordinates of the display designated point in the captured image data by the second imaging device 11b and the third imaging device 11c, and the corresponding second imaging device 11b and Based on the second three-dimensional coordinates of the third imaging device 11c, the imaging direction, the focal length, and the plane equation of the captured image, in the captured image data by the second imaging device 11b and the third imaging device 11c. It is also possible to calculate the second three-dimensional coordinate of the designated point for display and calculate the three-dimensional coordinate of the designated point based on the second three-dimensional coordinate of the designated point for display.

  The installation positions of the compound eye unit 10A and the compound eye unit 10B are set in advance. In other words, the second three-dimensional coordinates of the compound eye unit 10B are known, and the three-dimensional coordinate calculation unit 37 uses the second three-dimensional coordinates of the compound eye unit 10B to obtain the second imaging device 11b in the compound eye unit 10B and It is possible to calculate the second three-dimensional coordinate of the designated point for display in the image data captured by the third photographing device 11c, or to calculate the three-dimensional coordinate of the designated point.

≪Three-dimensional distance calculation unit 38≫
The three-dimensional distance calculation unit 38 acquires the second three-dimensional coordinates of the two specified points output from the three-dimensional coordinate calculation unit 37, and based on the acquired second three-dimensional coordinates, the two specified points The three-dimensional distance between them is calculated and output to the image data output unit 34. Then, the image data output unit 34 can output the calculated three-dimensional distance to the display device 40 and display it on the display device 40.

<Display device 40>
The display device 40 displays a monocular panoramic image using the monocular panoramic image data output from the designated point display image generation device 30, or displays the second captured image and the second captured image data using the second captured image data and the third captured image data. Or displaying a third photographed image.

<Operation example>
Subsequently, an operation example of the panoramic image display system 1 according to the embodiment of the present invention will be described with reference to FIGS.

  In this operation example, the compound-eye imaging unit 10A is arranged in the ceiling 100 from the first inspection port 101 as shown in FIGS. 5 and 6A, and images the ceiling 100 from one side. Further, as shown in FIGS. 5 and 6B, the compound eye photographing unit 10B is disposed in the ceiling 100 from the second inspection port 102, and photographs the ceiling 100 from the other side. In this operation example, it is desired to display a state in which a reinforcing material (brace) is installed between the field edge 103 and the slab 104, but the installation location of the brace 104 is not visible by the beam 105 from the first inspection port 101 side. Will be described as an example. 5 and 6A and 6B, the photographing devices 11a to 11c of one compound eye photographing unit 10A are separated by a predetermined distance, and the photographing devices 11a to 11c of the other compound eye unit 10B are separated. They are separated by a predetermined distance.

  First, in the compound eye photographing unit 10A, the control unit 13 rotates the driving unit 12 to direct the photographing devices 11a, 11b, and 11c in a desired photographing direction, and in this state, the shutters of the photographing devices 11a, 11b, and 11c. Photographing is performed by controlling, and photographed image data relating to the photographed image is output to the designated point display image generating device 30. In the compound eye photographing unit 10B, the control unit 13 rotates the driving unit 12 to direct the photographing devices 11a, 11b, and 11c in a desired photographing direction, and in this state, the shutters of the photographing devices 11a, 11b, and 11c. Photographing is performed by controlling, and photographed image data relating to the photographed image is output to the designated point display image generating device 30. Note that the compound eye photographing units 10A and 10B may photograph simultaneously or separately. Further, the compound eye photographing unit 10A that has finished photographing at the first inspection port 101 side can be re-installed at the second inspection port 102 side and used as the compound eye photographing unit 10B.

  Subsequently, the monocular panoramic image data generation unit 33 reads a plurality of first photographed image data photographed by the first photographing device 11a of the compound eye photographing unit 10A from the photographed image data storage unit 31, and the read first photographing. Monocular panoramic image data is generated based on the image data and output to the image data output unit 34. The image data output unit 34 outputs the monocular panoramic image data to the display device 40, and the display device 40 displays the monocular panoramic image (see FIG. 7).

  In this state, when the input device 20 outputs an image data designation for designating one first captured image 111 in the monocular panoramic image in accordance with the operation of the input device 20 by the user, the image data designation acquisition unit 35. Acquires the image data designation and outputs it to the image data output unit 34. The image data output unit 34 captures a pair of the second photographed image data and the third photographed image data having the same ID as the ID included in the image data designation. Read from the image data storage unit 31.

  The image data output unit 34 outputs a pair of second photographed image data and third photographed image data to the display device 40, and the display device 40 has a pair of second photographed images in the same photographing direction as the first photographed image 111. 112 and the third photographed image 113 are displayed on the same screen (see FIGS. 8A and 8B). In this operation example, the pair of the second captured image 112 and the third captured image 113 are arranged side by side.

  In this state, when the user operates the input device 20 and designates the display designated point (start point) S1 (corresponding to the designated point S) in the second photographed image 112, the designated point instruction in the second photographed image 112 is designated. The data is input to the determination unit 36.

Subsequently, when the user operates the input device 20 and designates a display designated point (start point) S2 (corresponding to the designated point S) in the third photographed image 113, the designated point instruction in the third photographed image 113 determines the designated point. Input to the unit 36.
When the designated point determination unit 36 acquires the two designated point instructions, the two-dimensional coordinates (x ₁₁ , y ₁₁ ) of the designated point S1 for display in the second photographed image 112 stored in the photographed image data storage unit 31 and The two-dimensional coordinates (x ₂₁ , y ₂₁ ) of the display designated point S 2 in the third captured image 113 are acquired and output to the three-dimensional coordinate calculation unit 37.

Subsequently, the three-dimensional coordinate calculation unit 37 calculates the second three-dimensional coordinate S = (x _S , y _S , z _S ) of the designated point S based on the above-described method, and the three-dimensional distance calculation unit 38. Output to. The three-dimensional coordinates of the designated point S are a straight line passing through the second imaging device 11b of the compound eye imaging unit 10A and the designated point for display S1, and a straight line passing through the third imaging device 11c of the compound eye imaging unit 10A and the designated point for display S2. The three-dimensional coordinates of the intersection of

  Subsequently, when the user operates the input device 20, the monocular panoramic image data generation unit 33 receives a plurality of first captured image data captured by the first imaging device 11 a of the compound eye imaging unit 10 </ b> B from the captured image data storage unit 31. The monocular panoramic image data is generated based on the read first captured image data, and is output to the image data output unit 34. The image data output unit 34 outputs the monocular panoramic image data to the display device 40, and the display device 40 displays the monocular panoramic image (see FIG. 9).

<Display Method of Display Designated Point S3>
Here, a display method of the display designated point S3 in the first captured image data by the compound eye imaging unit 10B by the three-dimensional coordinate calculation unit 37 will be described.

The imaging position (second three-dimensional coordinates) of the first imaging device 11a of the compound-eye imaging unit 10B is P21 (x ₂₀₁ , y ₂₀₁ , z ₂₀₁ ), and the second-eye imaging unit 10B that should display the display designated point S3. A plane equation of one captured image data is set to ax + by + cz + d = 0.
The vector V from the designated point S to the shooting position P21 is
V = (vx, vy, vz) = (x ₂₀₁ −x _RS , y ₂₀₁ −y _RS , z ₂₀₁ −z _RS )
The distance L between the designated point S and the shooting position P21 is
^{_{L = {(x 201 -x RS}} ) 2 + (y 201 -y RS) 2 + (z 201 -z RS) 2} 1/2
The unit vector E of the vector V is
E = (ex, ey, ez) = (vx / L, vy / L, vz / L)
Therefore, the second three-dimensional coordinates of the linear display designated point S3 located at the distance K from the photographing position P21 are expressed by Expression (1).
S3 = S + k · E = (x _S + k · ex, y _S + k · ey, z _S + k · ez) (1)

Here, if the display designated point S3 is on the plane ax + by + cz + d = 0,
a (x _S + k · ex) + b (y _S + k · ey) + c (z _S + k · ez) + d = 0
Therefore, k is expressed by equation (2).
_{k = - (a · x S} + b · y S + c · z S + d) / (a · ex + b · ey + c · ez) ... Equation (2)
By substituting k in Equation (2) into Equation (1), the second three-dimensional coordinates of the display designated point S3 in the first photographed image data by the first photographing device 11a of the compound eye photographing unit 10B are obtained.
The image data output unit 34 processes the first captured image data so as to display the display designated point S3 based on the second three-dimensional coordinates of the display designated point S3, and outputs the processed image data to the display device 40.

  Subsequently, when the input device 20 outputs an image data designation for designating the first photographed image 121 in the monocular panoramic image by the first photographing device 11a of the compound eye photographing unit 10B, the image data designation acquiring unit 35 designates the image data. Is acquired and output to the image data output unit 34. The image data output unit 34 stores a pair of second captured image data and third captured image data having the same ID as the ID included in the image data designation. Read from 31.

  The image data output unit 34 outputs a pair of second photographed image data and third photographed image data to the display device 40, and the display device 40 has a pair of second photographed images in the same photographing direction as the first photographed image 121. 122 and the third photographed image 123 are displayed on the same screen (see FIGS. 10A and 10B). Here, in the second captured image 122, a display designated point S4 corresponding to the designated point S is displayed by the same method as the display designated point S3, and the display designated point is also displayed in the third photographed image 123. A display designated point S5 corresponding to the designated point S is displayed by the same method as S3.

  In this state, when the user operates the input device 20 and designates the display designated point (end point) E1 (corresponding to the end point E) in the second photographed image 122, the designated point instruction in the second photographed image 122 determines the designated point. Input to the unit 36.

Subsequently, when the user operates the input device 20 to designate a designated point (end point) E2 (corresponding to the end point E) in the third photographed image 123, a designated point instruction in the third photographed image 123 is sent to the designated point determination unit 36. Entered.
When the designated point determination unit 36 acquires the two designated point instructions, the two-dimensional coordinates (x ₁₂ , y ₁₂ ) of the designated point E1 for display in the second photographed image 122 stored in the photographed image data storage unit 31 and The two-dimensional coordinates (x ₂₂ , y ₂₂ ) of the display designated point E 2 in the third captured image 123 are acquired and output to the three-dimensional coordinate calculation unit 37.

Subsequently, the three-dimensional coordinate calculation unit 37 calculates the second three-dimensional coordinate E = (x _E , y _E , z _E ) of the designated point E and outputs it to the three-dimensional distance calculation unit 38. The three-dimensional coordinates of the designated point E are a straight line passing through the second imaging device 11b of the compound eye photographing unit 10B and the designated point for display E1, and a straight line passing through the third photographing device 11c of the compound eye photographing unit 10A and the designated point for display E2. The three-dimensional coordinates of the intersection of

Subsequently, the three-dimensional distance calculation unit 38 calculates a three-dimensional distance D between the designated points S and E based on the following formula. Thereby, the three-dimensional distance is displayed on the display device 40.
^{_{D = {(x E -x S}} ) 2 + (y E -y S) 2 + (z E -z S) 2} 1/2

  In addition, the three-dimensional coordinate calculation unit 37 is the second three-dimensional of the display designated point (start point) S6 and the display designated point (end point) E3 in the first photographed image data by the first photographing device 11a of the compound eye photographing unit 10A. The coordinates are calculated, and the first captured image data is processed so as to display the virtual brace material 106 installed on the display designated point S6 and the display designated point E3, and is output to the display device 40 (see FIG. 11). ). In FIG. 11, the visible part of the virtual brace material 106 is displayed with a solid line and the part hidden behind the beam 105 is displayed with a dotted line, but the invisible part is not displayed. Good.

  The panoramic image display system 1 according to the embodiment of the present invention calculates the three-dimensional coordinates of the designated point based on the two captured image data, and displays the designated point based on the calculated three-dimensional coordinate. Since the other photographed image data is processed, the designated point can be suitably displayed even in an environment such as a ceiling behind an obstacle in photographing.

  In addition, since the panoramic image display system 1 according to the embodiment of the present invention performs compound-eye photography from two viewpoints, the three-dimensional distance between the two designated points even in an environment such as the back of a ceiling where obstacles in photography exist. Can be suitably calculated, or a virtual member constructed between two specified points can be suitably displayed.

  Moreover, the panoramic image display system 1 according to the embodiment of the present invention can calculate the second three-dimensional coordinates that can be used in common for all images included in the panoramic image. Further, the panoramic image display system 1 can calculate a three-dimensional distance between two designated points. In addition, the panorama image display system 1 has a second image corresponding to two captured images for obtaining a specified point rather than the first captured image data corresponding to one captured image for a monocular panoramic image in one imaging direction. By setting the shooting range of each of the shot image data and the third shot image data to be wider, even if the vicinity of the periphery in the first shot image data is used as the display designated point, the display designation is made. The point surely falls within the photographing range of the second photographed image data and the third photographed image data, and the second three-dimensional coordinate and the three-dimensional distance can be calculated reliably.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, A design change is possible suitably in the range which does not deviate from the summary of this invention. For example, the method of notifying the user of the calculated second three-dimensional coordinates and the three-dimensional distance is not limited to being displayed on the display device 40, and may be configured to utter a speaker as a notification unit. When the rotation around the Z axis (rotation angle ψ) is also taken into consideration, the second three-dimensional coordinate can be calculated by the following equation.

  Further, in the present invention, the right-handed coordinate system is adopted as the first three-dimensional coordinate system and the second three-dimensional coordinate system, but the case where the left-handed coordinate system is adopted is also included in the technical scope of the present invention.

Moreover, the 1st imaging device 11a can also be abbreviate | omitted by producing | generating monocular panoramic image data using either imaging | photography image data of the 2nd imaging device 11b and the 3rd imaging device 11c.
Further, the arrangement direction of the photographing devices 11a, 11b, and 11c is not limited to the left-right direction, and may be, for example, the up-down direction.
Further, the one compound eye photographing unit 10A and the other compound eye photographing unit 10B may be set such that the distance between the photographing devices 11a to 11c, the focal length of the photographing devices 11a to 11c, and the like are different.
In addition, the designated point determination unit 36 acquires a designated point for display designated in the first photographed image data in accordance with an operation of the input device 20 by the user, and designates the display in the first photographed image data by image processing. The configuration may be such that the display designated points in the second photographed image data and the third photographed image data corresponding to the points are determined.

DESCRIPTION OF SYMBOLS 1 Panorama image display system 30 Specified point display image generation apparatus 31 Image data storage part 33 Monocular panoramic image data generation part 34 Image data output part 35 Image data designation | designated acquisition part 36 Specified point determination part 37 Three-dimensional coordinate calculation part 38 Three-dimensional distance Calculation unit

Claims (3)

First image data photographed by the first photographing device, second image data photographed by the second photographing device separated from the first photographing device by a predetermined distance, and third image separated by a predetermined distance from the first photographing device. An image data storage unit that stores the third image data captured by the imaging device in association with each other;
A panorama image data generation unit that generates panorama image data based on the first image data;
An image data output unit for outputting image data to a display device;
A designated point determining unit that determines a designated point for display in each of the second image data and the third image data in accordance with an operation of the input device by a user;
A three-dimensional coordinate calculation unit that calculates a three-dimensional coordinate of a designated point that is an intersection of two straight lines connecting the designated point for display and the imaging device corresponding to the designated point for display based on the designated point for display; ,
With
The image data storage unit includes one first image data photographed by one of the first photographing devices, one second photographing device, and one third photographing device, and one second image data. And the third image data of one of them is stored in association with each other, and the other of the first images taken by the other first photographing device, the other second photographing device and the other third photographing device. One image data, the other second image data, and the other third image data are stored in association with each other,
The three-dimensional coordinate calculation unit
Based on the first designated point for display in one of the second image data and one of the third image data determined by the designated point determination unit, the three-dimensional coordinates of the first designated point are calculated, Based on the other first image data so that the first designated point for display in the other first image data is displayed on the image data output unit based on the three-dimensional coordinates of the first designated point. Process and output panoramic image data,
Based on the second designated point for display in the other second image data and the other third image data determined by the designated point determination unit, the three-dimensional coordinates of the second designated point are calculated, Based on the three-dimensional coordinate of the first designated point and the three-dimensional coordinate of the second designated point, the first designated point for display and the second in the first image data are output to the image data output unit. A designated point display image generating device, wherein panoramic image data based on one of the first image data is processed and output so as to display a designated point for display. Three-dimensional distance calculation for calculating a three-dimensional distance between the first designated point and the second designated point based on the three-dimensional coordinates of the first designated point and the three-dimensional coordinates of the second designated point. The designated point display image generation device according to claim 1, further comprising a unit. The image data output unit, one of the first to display a virtual member that is bridged to the a first display for the specified point and the second display specified point in one of the first image data The specified point display image generation apparatus according to claim 2, wherein panorama image data based on the image data is processed and output.

Images