JP2021064267A - Image processing apparatus and image processing method - Google Patents

Abstract

To provide an image processing apparatus configured to superimpose a different part between a designed design member and a design member arranged in a real space on a real-space image, and an image processing method.SOLUTION: An image processing apparatus includes an alignment unit, a difference processing unit, an image processing unit, and a display unit. The alignment unit aligns a three-dimensional model space generated based on a design member designed by computer-assisted design where a virtual object is arranged with a coordinate system of a real space. The difference processing unit generates a difference between the virtual object and three-dimensional member data corresponding to the virtual object, as a second virtual object. The image processing unit generates an image of the second virtual object to be observed from a viewing position of a user, in association with a real-space image, on the basis of the alignment. The display unit superimposes the second virtual object on the real-space image.SELECTED DRAWING: Figure 1

Description

Embodiments of the present invention relate to an image processing apparatus and an image processing method.

An image processing device that superimposes a virtual object on a real space image and displays it is known. On the other hand, at the construction site, a construction drawing for two-dimensional construction is created using the information of the three-dimensional design member designed by the three-dimensional CAD system.

However, when constructing a three-dimensional design member from a two-dimensional construction drawing, there is a risk that the construction will be mistaken due to a human recognition error or the like.

Japanese Unexamined Patent Publication No. 2012-94102

The problem to be solved by the present invention is to provide an image processing device capable of superimposing a difference between a design member designed by computer-aided design and a design member arranged in the real space on a real space image, and an image processing method. It is to be.

The image processing apparatus according to the present embodiment includes an alignment unit, a difference processing unit, an image processing unit, and a display unit. The alignment unit aligns the coordinate system of the three-dimensional model space in which the virtual object generated based on the design member designed by the computer-aided design is arranged and the coordinate system of the real space. The difference processing unit generates a difference between the virtual object and the three-dimensional member data generated based on the design member in the real space as the second virtual object. Based on the alignment, the image processing unit generates an image of the second virtual object observed from the user's viewpoint position in correspondence with the real space image observed from the user's viewpoint position. The display unit superimposes and displays the real space image and the second virtual object.

According to this embodiment, the difference between the design member designed by the computer-aided design and the design member arranged in the real space can be superimposed on the real space image.

The block diagram which shows the structure of the image processing apparatus which concerns on this embodiment. The figure which shows the calculation grid which consists of a plurality of calculation meshes. The figure which shows the attribute information given to the design member. The figure which shows typically the design member generated by the design processing part. The figure which shows typically the design member of the existing flange pipe generated by the design processing part. Sectional view of a virtual object based on a design member. The figure which shows the flange tube in the real space. Cross-sectional view of 3D member data based on the design member in the real space. A cross-sectional view of a virtual object based on a design member superimposed on the real space. The figure which superposed the secondary virtual object of the L-shaped angle material on the real space. The flowchart which shows the flow of image processing in an image processing apparatus.

Hereinafter, the image processing apparatus and the image processing method according to the embodiment of the present invention will be described in detail with reference to the drawings. The embodiments shown below are examples of the embodiments of the present invention, and the present invention is not construed as being limited to these embodiments. Further, in the drawings referred to in the present embodiment, the same parts or parts having similar functions are designated by the same reference numerals or similar reference numerals, and the repeated description thereof may be omitted. In addition, the dimensional ratio of the drawing may differ from the actual ratio for convenience of explanation, or a part of the configuration may be omitted from the drawing.

(One Embodiment)
FIG. 1 is a block diagram showing a configuration of an image processing device 1 according to the present embodiment. A configuration example of the image processing apparatus 1 will be described with reference to FIG. As shown in FIG. 1, the image processing device 1 according to the present embodiment is a device that aligns a real space photographed by a camera with a three-dimensional model space and superimposes and displays a real space image and an image of a virtual object. is there. The image processing device 1 includes a head-mounted image unit 10, a design unit 20, a display unit 30, and a marker 40.

The head-mounted image unit 10 is, for example, a wearable computer, which is a device worn on the operator's head. The detailed configuration of the head-mounted image unit 10 will be described later.

The design unit 20 is configured to include, for example, a CPU (Central Processing Unit), and arranges design members in the coordinate system of the design space. The design unit 20 is, for example, a CAD (CAD: computer-aided design), generates a design member using a design support tool having a three-dimensional CAD model, and arranges the design member in the three-dimensional model space for design. .. The design unit 20 includes an attribute information giving unit 202, a design processing unit 204, and an input operation unit 206.

The attribute information giving unit 202 gives attribute information to the design member generated by the design processing unit 204 described later. For example, the attribute information giving unit 202 gives the three-dimensional CAD model attribute information for selecting the three-dimensional CAD model.

The design processing unit 204 generates a design member by using the design support tool, and arranges the design member in the three-dimensional model space for design. The design processing unit 204 generates a design member based on the attribute information given by the attribute information giving unit 202, and arranges the design member in the three-dimensional model space for design. For example, the design processing unit 204 selects a design member from the three-dimensional CAD model for design based on the three-dimensional CAD model attribute information given by the attribute information giving unit 202. The details of the attribute information giving unit 202 and the design processing unit 204 will be described later.

The input operation unit 206 inputs information necessary for operating the head-mounted image unit 10 including the design unit 20. The input operation unit 206 includes, for example, a keyboard, a mouse, a pointing device, and the like. Further, the input operation unit 206 can be used by projecting it on the head-mounted image unit 10 or the like as a virtual keyboard or numeric keypad.

The display unit 30 displays various types of information. For example, the display unit 30 displays various information generated by the design unit 20. Further, the display unit 30 can superimpose and display the real space image and the virtual object by using the output signal of the output processing unit 110d described later. For example, the display unit 30 is composed of a liquid crystal display, a CRT (Cathode Ray Tube) display, or the like.

The marker 40 is a marker having a two-dimensional geometric pattern, and is used to associate the coordinate system in the real space with the coordinate system in the three-dimensional model space for design.

Here, the detailed configuration of the head-mounted image unit 10 will be described. The head-mounted image unit 10 includes a CPU, and includes a plurality of cameras 102, a microphone 104, a three-dimensional measurement unit 105, an input processing unit 106, a data storage unit 107, and a recognition processing unit 108. It has an image control processing unit 110, a liquid crystal screen 112, and a speaker 114.

The camera 102 is composed of a color digital camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary MOS) camera, for example. Each camera 102 outputs a real space image of the real space as image data. The image data captured by the camera 102 is output to the image control processing unit 110 via the input processing unit 106.

The microphone 104 collects the voice generated by the operator, for example, as a gesture, and outputs the voice data. The microphone 104 outputs the collected voice data to the voice recognition processing unit 108a via the input processing unit 106.

The three-dimensional measuring unit 105 is, for example, a 3D laser measuring device, and by non-contact measurement, the position of the surface of the target structure (for example, the inner wall surface in the case of an indoor (passage or room) of a building) and the design member 3 It is a device that measures three-dimensional member data (as-built data).

For example, when the three-dimensional measuring unit 105 is a 3D laser measuring device, the 3D laser measuring device is arranged at a reference position (for example, a reference mark) in the real space. The 3D laser measuring device then scans the laser toward the target structure. Next, the 3D laser measuring device receives the laser reflected by the target structure, measures the distance from the reference position, and calculates the three-dimensional coordinates of the surface point of the target structure from the output direction and distance of the laser.

That is, the three-dimensional measuring unit 105 measures the distance to the surface of the target structure based on the reference position where it is installed by non-contact measurement, and collects the shape data (for example, point cloud data) of them. obtain. The shape data is point cloud data indicating the range of space measured by laser measurement, and indicates a box shape in a box-shaped room where no object is placed, for example. Alternatively, when the design member is arranged, the three-dimensional member data in the shape of the design member and the box-shaped shape are shown. These shape data are stored in the data storage unit 107. In this case, a plurality of reference positions can be used to measure the entire range of the design member in the real space.

As the three-dimensional measuring unit 105, devices such as an electromagnetic wave measuring device, an ultrasonic measuring device, and a stereovision measuring device having directivity from light to radio waves can be used in addition to the 3D laser. The three-dimensional measurement unit 105 may be attached to the head-mounted image unit 10 for use. In this case, it is possible to determine the base point coordinates and the coordinate axes in time series by using the position / orientation information of the three-dimensional space recognition processing unit 108c described later.

The input processing unit 106 includes an alignment unit 106a, a coordinate integration unit 106b, and a conversion unit 106c.
The alignment unit 106a aligns the coordinate system of the three-dimensional model space in which the virtual object generated based on the design member is arranged and the coordinate system of the real space. More specifically, using the information of the marker 40 captured by the camera 102, the position and orientation of the head-mounted image unit 10 are recognized, and the coordinate system in the three-dimensional model space and the coordinate system in the real space are aligned. I do. For example, the marker 40 according to the present embodiment is arranged at the coordinate origin of the coordinate system in the three-dimensional model space. Therefore, the marker 40 is arranged at a position in the real space corresponding to the coordinate origin of the coordinate system in the three-dimensional model space, and the position and orientation of the head-mounted image unit 10 are recognized based on the captured image data of the marker 40. ..

The coordinate integration unit 106b integrates the point cloud data of the local coordinate system measured by scanning from the reference position into the coordinate system of the real space, and generates integrated shape data including the three-dimensional member data. When multiple point cloud data at different measurement positions in the real space are obtained, the multiple point cloud data of the local coordinate system measured at each reference position are integrated, and the integrated shape of the coordinate system in the real space is integrated. Generate data. At this time, the three-dimensional member data is associated with the coordinate system in the real space.

The conversion unit 106c converts the information of the design member generated by the design processing unit 204 into a virtual object. More specifically, the conversion unit 106c uses the information of the design member arranged in the 3D model space, for example, the information of the 3D CAD model, to make the design member into a virtual object arranged in the 3D model space. Convert.

The data storage unit 107 is composed of, for example, a hard disk drive device or a memory. As described above, the data storage unit 107 stores the shape data acquired via the coordinate integration unit 106b. In addition, integrated shape data including three-dimensional member data associated with the coordinate system in the real space integrated by the coordinate integration unit 106b is accumulated.

The recognition processing unit 108 includes a voice recognition processing unit 108a, an image recognition processing unit 108b, and a three-dimensional space recognition processing unit 108c.
The voice recognition processing unit 108a converts the voice data collected by the microphone 104 into an instruction code. For example, the head-mounted image unit 10 performs a control operation according to this instruction code.

The image recognition processing unit 108b acquires the distance information between the coordinates in the real space image in the superimposed and displayed image and the coordinates in the image of the virtual object.

The three-dimensional space recognition processing unit 108c calculates the three-dimensional coordinates of the natural feature points extracted from the image in the real space based on the image data captured by the camera 102, for example, by SfM (Structure from Motion), and the landmark data. Remember as. Then, the three-dimensional space recognition processing unit 108c estimates the position and orientation of the head-mounted image unit 10 based on the correspondence between the landmark data and the natural feature points obtained from the image data captured by the camera 102. In this way, even when the marker 40 is not imaged, the position and orientation of the head-mounted image unit 10 in the coordinate system in the real space and the reference coordinates of the three-dimensional measuring unit 105 when mounted on the head-mounted image unit 10 It can be estimated.

The image processing control unit 110 includes a control unit 110a, a difference processing unit 110b, an image processing unit 110c, and an output processing unit 110d.
The control unit 110a controls the entire image processing device 1.

The difference processing unit 110b generates a difference between the virtual object and the three-dimensional member data corresponding to the virtual object as the second virtual object. The coordinate system in the real space is associated with the virtual object as attribute information by the alignment of the coordinate system by the alignment unit 106a. The difference processing unit 110b uses this attribute information to generate a second virtual object. The details of the difference processing unit 110b will be described later.

The image processing unit 110c is the at least one of the real space image observed from the user's viewpoint position and the second virtual object and the virtual object observed from the user's viewpoint position based on the alignment by the alignment unit 106a. Combine with the image. More specifically, the image processing unit 110c is a design member based on the position and orientation of the head-mounted image unit 10 estimated by the three-dimensional space recognition processing unit 108c after the alignment by the alignment unit 106a is completed. The position and shape of the virtual object generated based on the above and the second virtual object generated by the difference processing unit 110b are changed and superimposed on the real space image. Further, the image processing unit 110c generates, for example, a three-dimensional member object showing the three-dimensional member data in a mesh shape and superimposes the three-dimensional member object on the real space image. Also in this case, the position and shape of the three-dimensional member object are changed based on the position and orientation of the head-mounted image unit 10 estimated by the three-dimensional space recognition processing unit 108c, and superimposed on the real space image.

The output processing unit 110d converts the data format input from the image processing unit 110c or the like and outputs the data format to the display unit 30, the liquid crystal screen 112, and the speaker 114.

The liquid crystal screen 112 superimposes and displays a real space image observed from the user's viewpoint position and an image of at least one of the second virtual object and the virtual object observed from the user's viewpoint position. Further, the liquid crystal screen 112 can superimpose and display, for example, a three-dimensional member object of three-dimensional member data on a real space image.

The liquid crystal screen 112 is, for example, a head-mounted display. Further, the liquid crystal screen 112 may be realized by an optical see-through method. For example, it is realized by displaying an image of a virtual object generated according to the position and orientation of the viewpoint of the observer on an optical see-through liquid crystal display. In this case, the image of the virtual object is superimposed so as to overlap the actual real space, not the image in the real space.

The communication between the head-mounted image unit 10 and the design unit 20 may be wireless communication or wired communication. Similarly, the communication between the head-mounted image unit 10 and the display unit 30 may be wireless communication or wired communication. Further, the design unit 20 may be configured in the head-mounted image unit 10. Furthermore, the communication between the head-mounted image unit 10 and the three-dimensional measurement unit 105 may be wireless communication or wired communication.

Further, the design member may be generated by using the design support tool while the head-mounted image unit 10 is worn by the operator, and the design member may be arranged in the three-dimensional model space for design. That is, the design member designed at the site can be observed as an image of a virtual object. At this time, for the operation of the design support tool, the input operation unit 206 may be projected on the head-mounted image unit 10 or the like as a virtual keyboard or numeric keypad and used. In this way, it is possible to confirm, verify, and modify the virtual object of the design member that overlaps with the actual real space while performing the design act, and the design efficiency can be further improved. As a result, the process of 3D laser scanning work has been required in the past, but it is no longer necessary, and it is possible to significantly shorten the process.

The speaker 114 outputs voice based on the recognition result by the recognition processing unit 108. For example, the voice based on the instruction code output by the voice recognition processing unit 108a is output. For example, it is possible to give a voice instruction to the design support tool.

Here, the detailed configuration of the attribute information giving unit 202 and the design processing unit 204 will be described with reference to FIGS. 2 to 5.
FIG. 2 is a diagram showing a grid point model showing a model space and an arrangement of design members. The arrangement of the design members in the grid point model will be described with reference to FIG. Here, a part of the horizontal plane of the three-dimensional grid point model 300 is shown. As shown in FIG. 2, the attribute information adding unit 202 generates a three-dimensional lattice point model 300 as a three-dimensional model space corresponding to the real space to be constructed based on the actual dimensions of the real space to be constructed. To do. Coordinates B1 to B3 and coordinates C1 to C3 indicate the position of the grid point model 300. For example, the coordinates B1 to B3 and the coordinates C1 to C3 correspond to the positions of the cores in the real space. Further, the marker position M1 indicates a position where the marker 40 is arranged when the alignment unit 106a aligns the coordinate space.

The attribute information giving unit 202 gives the position where the design member is arranged in the three-dimensional model space as the attribute information. More specifically, the attribute information giving unit 202 gives the position where the design member is arranged as the attribute information based on the coordinates on the three-dimensional grid point model 300 (FIG. 2) instructed by the input processing unit 106. .. For example, the attribute information giving unit 202 gives the position where the design member is arranged as the attribute information corresponding to the position of the street core on the three-dimensional grid point model 300 (FIG. 2) instructed by the input processing unit 106. It is also possible to do. Further, the attribute information giving unit 202 may arrange the design member in the three-dimensional model space by using the distance information acquired by the image recognition processing unit 108b.

FIG. 3 is a diagram showing attribute information given to the design member. The attribute information given to the design member will be described with reference to FIG. As shown in FIG. 3, the attribute information imparting unit 202 also imparts attribute information indicating the type of the member to the design members A1 to A8 generated by the design processing unit 204. For example, the attribute information giving unit 202 stores information such as a pipe having a diameter of 100A and a flange pipe having a diameter of 100A as data of attribute information that can be generated by the design processing unit 204. Then, the attribute information giving unit 202 gives the attribute information instructed by the input processing unit 106 to the design members A1 to A8.

Design members R1 to R5 already arranged in the real space are added to the design members A4 to A8 as attribute information. That is, the attribute information giving unit 202 can give the design members R1 to R5 arranged in the real space as the attribute information of the design members A4 to A8.

Here, an example of the design members A1 to A3 arranged in the three-dimensional model space will be described with reference to FIG. FIG. 4 is a diagram schematically showing design members A1 to A3 generated by the design processing unit 204. As shown in FIG. 4, the design processing unit 204 generates the design members A1 to A3 in the three-dimensional model space based on the attribute information given to the design member. More specifically, the design processing unit 204 generates the design members A1 to A3 in the three-dimensional model space by using the information and the position information of the three-dimensional CAD model given to the design member.

Here, the design member A4 based on the existing design data will be described with reference to FIG. FIG. 5 is a diagram schematically showing a design member A4 of the existing flange pipe R1 generated by the design processing unit 204. As shown in FIG. 5, the design processing unit 204 arranges the design member A4 in the three-dimensional model space based on the existing design data of the design member R1 that should be originally constructed in the real space. Although not shown in FIG. 5, the design members A5 to A7 are also arranged in the three-dimensional model space based on the existing design data of the design members R2 to R4.

Here, the virtual objects ZA4 to ZA7 of the design members A4 to A7 will be described with reference to FIG. FIG. 6 is a diagram schematically showing a cross section of virtual objects ZA4 to ZA7 generated by the conversion unit 106c in a virtual cross section Z1 (FIG. 7). The virtual objects ZA4 to ZA7 correspond to the design members A4 to A7. As shown in FIG. 6, the conversion unit 106c arranges virtual objects ZA4 to ZA7 based on the existing design data of the design members R1 to R4 that should be originally constructed in the real space in the three-dimensional model space. In this way, the conversion unit 106c can arrange the members that should be originally constructed in the real space as virtual objects ZA4 to ZA7 in the three-dimensional model space.

Here, the measurement data of the flange tube R1 in the real space will be described with reference to FIG. 7. FIG. 7 is a diagram showing a flange tube R1 in a real space. The above-mentioned three-dimensional measurement unit 105 measures the flange tube R1 in the real space using the marker M1 and a plurality of other reference positions as reference coordinates, and the coordinate integration unit 106b associates the coordinate system with the coordinate system in the real space. The generated 3D member data is generated. Z1 is a virtual cross section. In FIG. 7, the support steel materials R3, R4, and L-shaped angle material R5 are not shown.

Here, an example of three-dimensional member data will be described with reference to FIG. FIG. 8 is a diagram schematically showing a cross-sectional view of the three-dimensional member data ZR1, ZR3, ZR4 based on the design members R1, R3, R4 in the real space in the virtual cross section Z1. In this way, the design members R1, R3, R4 and the like already arranged in the real space are generated as the three-dimensional member data ZR1, ZR3, ZR4 by the coordinate integration unit 106b. Note that FIG. 8 shows an example in which the design member R2 is not constructed due to a construction error.

Here, with reference to FIGS. 3, 6 and 8, a detailed example of the processing of the difference processing unit 110b will be described with reference to FIGS. 9 and 10. FIG. 9 is a cross-sectional view of the virtual cross section Z1 in which the virtual objects ZA4 to ZA7 based on the design members R1 to R4 are superimposed on the real space. ZA5 (R2) shows an example of the second virtual object.

The difference processing unit 110b uses the three-dimensional member data stored in the data storage unit 107 to perform difference processing between, for example, the virtual object ZA5 and the three-dimensional member data ZR2. More specifically, the difference processing unit 110b extracts the three-dimensional member data ZR2 in the coordinate system in the real space corresponding to the virtual object ZA5 from the data storage unit 107. Next, since the three-dimensional member data ZR2 is surface data, the difference processing unit 110b generates filling data obtained by performing filling processing on the range included in the surface data. Then, the difference processing unit 110b performs difference processing between the virtual object ZA5 and the filling data corresponding to the virtual object ZA5. In this way, the difference processing unit 110b calculates the area of the three-dimensional member data ZR2 that is insufficient for the virtual object ZA5 as the secondary virtual object ZA5 (R2) of the virtual object ZA5. As shown in FIG. 8, since the three-dimensional member data ZR2 does not exist in the real space, the filling data is processed as 0.

When only the design member R1 corresponding to the virtual object ZA4 is arranged in the real space, the difference processing unit 110b stores the three-dimensional member data in a predetermined range from the space in which the virtual object ZA4 is arranged in the real space. Extract from. For example, the three-dimensional member data in the area including the range of 30 cm from the space where the virtual object ZA4 in the real space is arranged is extracted from the data storage unit 107. Next, since the three-dimensional member data is surface data, the difference processing unit 110b generates filling data in which the filling process is performed on the range included in the surface data. Then, the difference processing unit 110b performs difference processing between the virtual object ZA4 and the filling data. As a result, the area of the three-dimensional member data ZR1 that is excessive or insufficient with respect to the virtual object ZA4 can be calculated as a secondary virtual object of the virtual object ZA4. In this way, the difference processing unit 110b performs difference processing between the three-dimensional member data in a predetermined range from the space where the virtual object ZA4 in the real space is arranged and the virtual object ZA4, and the excess / deficiency area data is the second virtual. Create as an object.

Further, as shown in FIG. 6, the difference processing unit 110b may perform the above-mentioned processing on the joined virtual objects ZA4 to ZA7 when the virtual objects ZA4 to ZA7 are joined. For example, as shown in FIG. 3, the support steel material R2 should be arranged in the real space according to the design specifications. However, as shown in FIG. 8, the support steel material R2 is not arranged in the real space. In this case, when the above processing is performed, the difference between the joined virtual objects ZA4 to ZA7 and the filling data of the three-dimensional member data ZR1, ZR3, and ZR4 is calculated, and the second virtual object ZA5 (R2) is used as insufficient area data. Is generated. Then, the output processing unit 110d generates an image signal in which the second virtual object ZA5 (R2) is superimposed and displayed in the real space.

For example, the second virtual object ZA5 (R2) corresponding to the shortage area is displayed in a color different from the colors (for example, green) of the other virtual objects ZA4, ZA6, and ZA7, for example, red. On the other hand, the second virtual object corresponding to the surplus area is displayed in a color different from other virtual objects, for example, blue.

FIG. 10 is a diagram showing an example of difference processing when the L-shaped angle member R5 is attached in the opposite direction. The upper left figure shows the three-dimensional member data ZR1 and ZR5, the lower left figure shows the virtual objects ZA4 and ZA8, and the right figure shows the figure after the difference processing. As shown in the right figure, when the virtual objects ZA4 and ZA8 are joined, the difference processing unit 110b performs the above-mentioned difference processing on the joined virtual objects ZA4 and ZA8. In this case, when the above-mentioned difference processing is performed, the difference between the joined virtual objects ZA4 and ZA8 and the filling data of the three-dimensional member data ZR1 and ZR5 is calculated, and the second virtual object ZA8 (R5) is generated as insufficient area data. Will be generated. Then, the output processing unit 110d generates an image signal in which the second virtual object ZA8 (R5) is superimposed and displayed in the real space.

In this way, the difference between the virtual objects ZA4 to ZA8 corresponding to the design members in the construction drawing designed by the computer-aided design and the design members R1 to R5 arranged in the real space is shown in the real space image as the second virtual. It can be superimposed as objects ZA5 (R2) and ZA8 (R5). As a result, the difference between the design member in the construction drawing that should be originally constructed and the construction result in the real space is visualized as the second virtual objects ZA5 (R2) and ZA8 (R5) in the real space. be able to.

FIG. 11 is an example of a flowchart showing the flow of image processing in the image processing device 1.
As shown in FIG. 11, the operator inputs the attribute information of the design member and the marker 40 used for construction by the attribute information adding unit 202 (step S100). The operator instructs, for example, the coordinates corresponding to the position where the design member is arranged in the three-dimensional model space displayed on the display unit 30 by the input operation unit 206. Further, the operator selects a design member from, for example, a list of three-dimensional CAD models displayed on the display unit 30. The attribute information giving unit 202 gives such information as attribute information of the design member. At this time, the coordinates for arranging the marker 40 in the three-dimensional model space are also set.

Next, the design processing unit 204 generates a design member based on the attribute information given by the attribute information giving unit 202, and arranges the design member in the three-dimensional model space for design (step S102). These design members are displayed on the display unit 30, for example, and are visually recognized by the operator. Subsequently, the conversion unit 106c converts the information of the design member generated by the design processing unit 204 into a virtual object. As a result, the virtual object generated based on the design member designed by the computer-aided design is arranged in the three-dimensional model space.

Next, the three-dimensional measuring unit 105 is arranged at the reference position, the laser is scanned toward the target structure, and the design member is measured (step S104). Three-dimensional member data and the like are stored in the data storage unit 107. At this time, if there is an unmeasured area, the reference position is changed and the measurement is repeated. Subsequently, the coordinate integration unit 106b integrates the measured three-dimensional member data and the like into the coordinate system in the real space, and generates integrated shape data including the three-dimensional member data.

Next, the marker 40 is placed at a position in the real space corresponding to the coordinates of the marker 40 set in the three-dimensional model space, and an image is taken by the camera 102 (step S106). Subsequently, the alignment unit 106a aligns the coordinate system in the three-dimensional model space with the coordinate system in the real space based on the information of the imaged marker 40 (step S108).

The difference processing unit 110b generates a difference between the virtual object and the three-dimensional member data corresponding to the virtual object as the second virtual object based on the coordinate system in the real space (step S110).

Next, the three-dimensional space recognition processing unit 108c acquires natural feature points from the image data captured by the camera 102, and based on the correspondence between the landmark data and the natural feature points, the position and orientation of the head-mounted image unit 10 Is estimated (step S112).

Next, the image processing unit 110c superimposes the real space image observed from the user's viewpoint position and the image of the second virtual object observed from the user's viewpoint position based on the estimation result of the position and orientation. , Displayed on the liquid crystal screen 112 and the display unit 30 via the output processing unit 110d (step S114).

The control unit 110a determines whether to continue the image processing control (step S116). When continuing the image processing control (NO in step S116), the processing from step S112 is repeated. On the other hand, when the image processing control is terminated (YES in step S116), the entire processing is terminated.

As described above, according to the present embodiment, the alignment unit 106a aligns the coordinates of the three-dimensional model space in which the virtual object generated based on the design member is arranged and the coordinates of the real space, and makes a difference. The processing unit 110b generates the difference between the virtual object of the design member and the three-dimensional member data in the real space corresponding to the virtual object as the second virtual object. Then, the image processing unit 110c generates an image of the second virtual object observed from the user's viewpoint position in correspondence with the real space image observed from the user's viewpoint position based on the alignment. As a result, it is possible to visualize the difference between the design member that should have been originally constructed and the construction result in the real space as a second virtual object in the real space.

At least a part of the data processing method in the image processing apparatus 1 according to the present embodiment may be configured by hardware or software. When configured by software, a program that realizes at least a part of the functions of the data processing method may be stored in a recording medium such as a flexible disk or a CD-ROM, read by a computer, and executed. The recording medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed recording medium such as a hard disk device or a memory. Further, a program that realizes at least a part of the functions of the data processing method may be distributed via a communication line (including wireless communication) such as the Internet. Further, the program may be encrypted, modulated, compressed, and distributed via a wired line or wireless line such as the Internet, or stored in a recording medium.

Although some embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel devices, methods and programs described herein can be implemented in a variety of other forms. In addition, various omissions, substitutions, and changes can be made to the forms of the apparatus, method, and program described in the present specification without departing from the gist of the invention.

1: Image processing device, 10: Head-mounted image unit, 20: Design unit, 30: Display unit, 40: Marker, 105: Three-dimensional measurement unit, 106a: Alignment unit, 106c: Conversion unit, 108: Recognition processing Unit, 108b: Image recognition processing unit, 110b: Difference processing unit, 110c: Image processing unit, 206: Input operation unit

Claims (9)

An alignment unit that aligns the coordinate system of the 3D model space where the virtual objects generated based on the design members designed by computer-aided design are placed and the coordinate system of the real space.
A difference processing unit that generates a difference between the virtual object and the three-dimensional member data generated based on the design member in the real space as a second virtual object.
An image processing unit that generates an image of the second virtual object observed from the user's viewpoint position in correspondence with the real space image observed from the user's viewpoint position based on the alignment.
A display unit that superimposes and displays the real space image and the second virtual object, and
An image processing device comprising. Further provided with a three-dimensional measuring unit for three-dimensionally measuring the design member in the real space as three-dimensional member data.
The image processing apparatus according to claim 1, wherein the difference processing unit generates the second virtual object using the three-dimensional member data. Based on the alignment, the image processing unit generates a three-dimensional member object of the three-dimensional member data in correspondence with the real space image observed from the viewpoint position of the user.
The image processing device according to claim 1, wherein the display unit superimposes and displays the real space image and the three-dimensional member object. The difference processing unit performs difference processing between the three-dimensional member data within a predetermined range from the space in which the virtual object is arranged in the real space and the virtual object, and obtains excess or deficiency area data in the second. The image processing apparatus according to any one of claims 1 to 3, which is generated as a virtual object. The image processing apparatus according to claim 4, wherein the difference processing unit extracts the three-dimensional member data in the predetermined range based on the position information in the coordinate system in the real space associated with the virtual object. Based on the alignment, the image processing unit generates an image of the virtual object observed from the user's viewpoint position in correspondence with the real space image observed from the user's viewpoint position.
The image processing device according to any one of claims 1 to 5, wherein the display unit superimposes and displays the real space image and the virtual object. A design unit that arranges the design member in the three-dimensional model space based on the design data of the member already arranged in the real space, and a design unit.
A conversion unit that converts the design member into the virtual object is further provided.
The image processing device according to claim 6, wherein the image processing unit superimposes and displays the virtual object based on the design data of the arranged member. Further having an input operation unit for operating the design unit,
The image processing apparatus according to claim 7, wherein the conversion unit converts the design member generated by the design unit via the input operation unit into the virtual object. An alignment process that aligns the coordinate system of the 3D model space where the virtual objects generated based on the design members designed by computer-aided design are placed and the coordinate system of the real space.
A difference processing step of generating a difference between the virtual object and the three-dimensional member data generated based on the design member in the real space as a second virtual object.
An image processing step of generating an image of the second virtual object observed from the user's viewpoint position in correspondence with the real space image observed from the user's viewpoint position based on the alignment.
A display process for superimposing and displaying the real space image and the second virtual object, and
An image processing method comprising.

Images