JP2020024497A - Image processing device, and image processing method - Google Patents

Abstract

To provide an image processing device, and an image processing method which can superimpose an installation result image of a designed member designed by computer-aided design on a real space image.SOLUTION: An image processing device comprises an alignment unit, an image processing unit, and a display unit. The alignment unit aligns a coordinate system of a three-dimensional model space in which a virtual object generated based on a designed member designed by computer-aided design is disposed with a coordinate system of a real space. The image processing unit generates an image of the virtual object observed from a view position of a user, in association with a real space image observed from the view position of the user, on the basis of alignment. The display unit displays the real space image and the virtual object in a superimposed manner.SELECTED DRAWING: Figure 1

Description

  An embodiment of the present invention relates to an image processing device and an image processing method.

  2. Description of the Related Art An image processing apparatus that displays a virtual object superimposed on a real space image is known. On the other hand, at a construction site, a construction drawing for two-dimensional construction is generated using information of three-dimensional design members designed by a three-dimensional CAD system.

  However, when constructing a three-dimensional design member from a two-dimensional construction drawing, there is a risk that construction may be erroneously performed due to a human recognition error or the like.

JP 2012-94102 A

  An object of the present invention is to provide an image processing apparatus and an image processing method capable of superimposing a construction result image of a design member designed by computer-aided design on a real space image.

  The image processing device according to the present embodiment includes a positioning unit, an image processing unit, and a display unit. The alignment unit aligns a coordinate system in a three-dimensional model space, in which a virtual object generated based on a design member designed by computer-aided design is arranged, with a coordinate system in a real space. The image processing unit generates an image of the virtual object observed from the viewpoint position of the user, based on the alignment, in correspondence with the real space image observed from the viewpoint position of the user. The display unit superimposes and displays the real space image and the virtual object.

  According to the present embodiment, a construction result image of a design member designed by computer-aided design can be superimposed on a real space image.

FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment. The figure which shows the calculation grid comprised by a some calculation mesh. The figure which shows the attribute information given to the design member. The figure which shows typically the design member produced | generated by the design processing part. The figure which superimposed the virtual object based on the design member on the real space. The figure which superimposed the virtual object based on the design member scheduled to be removed on the real space. The figure which superimposed the movable virtual object on the real space. FIG. 4 is a diagram showing points on a virtual object and recognized fingertips. The figure which displayed the point which shows the point on a virtual object, and the fixed point. The figure which shows the point in real space, and the recognized fingertip. The figure which displayed the mark which shows the point in a real space, and a fixed point. The figure which shows the distance between the point on a virtual object, and the point on a real space image. The figure which shows the arrangement | positioning point of the virtual object in a design member. FIG. 3 is a diagram illustrating an arrangement point and a virtual object of a design member. The figure which shows the movement of the design member superimposed on the physical space image. The figure which shows the virtual object of the design member arrange | positioned at the arrangement point. The figure which shows the movement of the relocated design member. 5 is a flowchart illustrating a flow of image processing in the image processing apparatus. The figure which shows the result of having given the information of process order as attribute information for every design part. The figure which superimposed the virtual object of the design member in the first process. The figure which superimposed the virtual object of the design member in the next process. The figure which superimposed the virtual object of the design member in the 3rd process.

  Hereinafter, an image processing apparatus and an image processing method according to an embodiment of the present invention will be described in detail with reference to the drawings. The embodiments described below are examples of the embodiments of the present invention, and the present invention is not construed as being limited to these embodiments. In the drawings referred to in this embodiment, the same portions or portions having similar functions are denoted by the same reference numerals or similar reference numerals, and repeated description thereof may be omitted. Further, the dimensional ratios in the drawings may be different from the actual ratios for convenience of description, or some of the components may be omitted from the drawings.

(1st Embodiment)
FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus 1 according to the present embodiment. As shown in FIG. 1, an image processing apparatus 1 according to the present embodiment is an apparatus that positions a real space captured by a camera and a three-dimensional model space, and superimposes and displays a real space image and a virtual object image. is there. The image processing apparatus 1 includes a head-mounted image unit 10, a design unit 20, a display unit 30, and a marker 40.

  The head mounted image section 10 is, for example, a wearable computer, and is a device that is mounted on the operator's head. The detailed configuration of the head mounted image section 10 will be described later.

  The design unit 20 includes, for example, a CPU (Central Processing Unit), and arranges design members in a coordinate system of a design space. The design unit 20 is, for example, a CAD (computer-aided design), generates a design member using a design support tool having a three-dimensional CAD model, and arranges the design member in a three-dimensional model space for design. . The design unit 20 includes an attribute information providing unit 202, a design processing unit 204, and an input operation unit 206.

  The attribute information assigning unit 202 assigns attribute information to a design member generated by a design processing unit 204 described later. For example, the attribute information assigning unit 202 assigns three-dimensional CAD model attribute information for selecting a three-dimensional CAD model.

  The design processing unit 204 generates a design member using a design support tool, and arranges the design member in a three-dimensional model space for design. The design processing unit 204 generates a design member based on the attribute information given by the attribute information giving unit 202, and arranges the design member in a three-dimensional model space for design. For example, the design processing unit 204 selects a design member from a three-dimensional CAD model for design based on the three-dimensional CAD model attribute information provided by the attribute information providing unit 202. The details of the attribute information providing unit 202 and the design processing unit 204 will be described later.

  The input operation unit 206 inputs information necessary for operating the head mounted video unit 10 including the design unit 20. The input operation unit 206 includes, for example, a keyboard, a mouse, a pointing device, and the like.

  The display unit 30 displays various information. For example, the display unit 30 displays various information generated by the design unit 20. The display unit 30 displays the real space image and the virtual object in a superimposed manner. For example, the display unit 30 is configured by a liquid crystal display, a CRT (Cathode Ray Tube) display, or the like.

  The marker 40 is a marker having a two-dimensional geometric pattern, and is used to associate a coordinate system in the real space with a coordinate system in the three-dimensional model space for design.

  Here, a detailed configuration of the head-mounted video unit 10 will be described. The head-mounted image unit 10 includes a CPU, and includes a plurality of cameras 102, a microphone 104, an input processing unit 106, a recognition processing unit 108, an image control processing unit 110, a liquid crystal screen 112, 114.

  The camera 102 is configured by a color digital camera such as a CCD (Charge Coupled Device) camera or a CMOS (Complementary MOS) camera. Each camera 102 outputs an image of the real space as image data. Image data captured by the camera 102 is output to the image processing unit 110b via the input processing unit 106.

  The microphone 104 collects, for example, a voice generated by the operator as a gesture, and outputs voice data. The microphone 104 outputs the collected voice data to the voice recognition processing unit 108a via the input processing unit 106.

The input processing unit 106 has a positioning unit 106a and a conversion unit 106b.
The positioning unit 106a performs positioning between the coordinate system in the three-dimensional model space where the virtual objects generated based on the design members are arranged and the coordinate system in the real space. More specifically, the information of the marker 40 captured by the camera 102 is used to recognize the position and orientation of the head-mounted image unit 10, and to align the coordinate system of the three-dimensional model space with the coordinate system of the real space. I do. For example, the marker 40 according to the present embodiment is arranged at the coordinate origin of the coordinate system in the three-dimensional model space. For this reason, the marker 40 is arranged at a position in the real space corresponding to the coordinate origin of the coordinate system in the three-dimensional model space, and the position and orientation of the head-mounted video unit 10 are recognized based on the captured image data of the marker 40. .

  The conversion unit 106b converts the information on the design members generated by the design processing unit 204 into a virtual object. More specifically, the conversion unit 106b uses the information on the design members allocated in the three-dimensional model space, for example, the information on the three-dimensional CAD model to convert the design members into virtual objects arranged in the three-dimensional model space. Convert to

The recognition processing unit 108 includes a voice recognition processing unit 108a, an image recognition processing unit 108b, and a three-dimensional space recognition processing unit 108c.
The voice recognition processing unit 108a converts the voice data collected by the microphone 104 into an instruction code. For example, the head-mounted video unit 10 performs a control operation according to the instruction code.

  The image recognition processing unit 108b acquires distance information between the coordinates in the real space image in the superimposed image and the coordinates in the image of the virtual object. A more detailed configuration will be described later.

  The three-dimensional space recognition processing unit 108c calculates the three-dimensional coordinates of the natural feature points extracted from the image in the real space by using SfM (Structure from Motion) based on the image data captured by the camera 102, for example, and calculates landmark data. To be stored. Then, the three-dimensional space recognition processing unit 108c estimates the position and orientation of the head-mounted image unit 10 based on the correspondence between the landmark data and the natural feature points obtained from the image data captured by the camera 102. As described above, even when the marker 40 is not imaged, the position and orientation of the head-mounted image unit 10 in the coordinate system in the real space can be estimated.

The image processing control unit 110 has a control unit 110a, an image processing unit 110b, and an output processing unit 110c.
The control unit 110a controls the entire image processing apparatus 1.

  The image processing unit 110b combines the real space image observed from the viewpoint position of the user and the image of the virtual object observed from the viewpoint position of the user, based on the alignment by the alignment unit 106a. More specifically, after the positioning by the positioning unit 106a is completed, the image processing unit 110b determines the design member based on the position and orientation of the head-mounted image unit 10 estimated by the three-dimensional space recognition processing unit 108c. The position and shape of the virtual object generated based on are changed and superimposed on the real space image. Further, the image processing unit 110b generates a movable object of the virtual object and superimposes the movable object on the real space image.

  The output processing unit 110c converts a data format input from the image processing unit 110b or the like, and outputs the data to the display unit 30, the liquid crystal screen 112, and the speaker 114.

  The liquid crystal screen 112 superimposes and displays the real space image observed from the viewpoint position of the user and the image of the virtual object observed from the viewpoint position of the user. The liquid crystal screen 112 is, for example, a head mounted display. Further, the liquid crystal screen 112 may be realized by an optical see-through method. For example, this is realized by displaying an image of a virtual object generated according to the position and orientation of the observer's viewpoint on an optical see-through liquid crystal display. In this case, the image of the virtual object is superimposed so as to overlap not the image of the real space but the actual real space.

  Note that the communication between the head-mounted image unit 10 and the design unit 20 may be wireless communication or wired communication. Similarly, the communication between the head-mounted video unit 10 and the display unit 30 may be wireless communication or wired communication. Further, the design unit 20 may be configured in the head-mounted image unit 10.

  Furthermore, a design member may be generated using a design support tool in a state where the operator wears the head-mounted image unit 10, and the design member may be arranged in a three-dimensional model space for design. That is, the design member designed on site can be observed as an image of the virtual object. At this time, for the operation of the design support tool, the input operation unit 206 may be displayed on the head mounted image unit 10 or the like as a virtual keyboard or numeric keypad and used. As described above, it is possible to confirm, verify, and correct the virtual object of the design member overlapping the actual real space while performing the design action, and it is possible to further improve the design efficiency. Accordingly, a 3D laser scanning operation process was conventionally required, but is no longer necessary, and it is possible to significantly reduce the number of steps.

  The speaker 114 outputs a sound based on the recognition result by the recognition processing unit 108. For example, a voice based on the instruction code output by the voice recognition processing unit 108a is output. For example, it is also possible to instruct the design support tool by voice.

Here, a detailed configuration of the attribute information providing unit 202 and the design processing unit 204 will be described.
FIG. 2 is a diagram illustrating a lattice model indicating a model space and an arrangement of design members. Here, a part of the horizontal plane of the three-dimensional grid point model 300 is shown. As shown in FIG. 2, the attribute information providing unit 202 generates a three-dimensional grid point model 300 as a three-dimensional model space corresponding to the real space to be constructed, based on the actual dimensions of the real space to be constructed. I do. The coordinates B1 to B3 and the coordinates C1 to C3 indicate the positions of the grid point model 300. For example, the coordinates B1 to B3 and the coordinates C1 to C3 correspond to the position of the center of the real space. The marker position M1 indicates a position at which the marker 40 is arranged when the positioning unit 106a performs positioning in the coordinate space.

  The attribute information assigning unit 202 assigns, as attribute information, a position at which the design member is arranged in the three-dimensional model space. More specifically, the attribute information assigning unit 202 assigns the position at which the design member is to be arranged as attribute information based on the coordinates on the three-dimensional lattice model 300 (FIG. 2) specified by the input processing unit 106. . For example, the attribute information assigning unit 202 assigns the position at which the design member is to be arranged as attribute information in correspondence with the position of the center line on the three-dimensional lattice model 300 (FIG. 2) specified by the input processing unit 106. It is also possible. In addition, the attribute information providing unit 202 may arrange the design members in the three-dimensional model space using the distance information acquired by the image recognition processing unit 108b.

FIG. 3 is a diagram showing attribute information given to a design member. As shown in FIG.
The attribute information assigning unit 204 also assigns attribute information to the design members A1 to A4, D1, and MV1 to MV2 generated by the design processing unit 204. For example, the attribute information assigning unit 204 stores information on, for example, a pipe having a diameter of 100A and a flange pipe having a diameter of 100A as data of attribute information that can be generated by the design processing unit 204. Then, the attribute information assigning unit 204 assigns the attribute information specified by the input processing unit 106 to the design members A1 to A3. The attribute information providing unit 204 may provide information of the three-dimensional CAD model stored in the design processing unit 204 as information on the pipe having a diameter of 100A, the flange pipe having a diameter of 100A, and the like. Alternatively, information of another three-dimensional model may be added.

  To the design member A4, the design member R1 already arranged in the real space is given as attribute information. That is, the attribute information providing unit 204 can provide the design member R1 arranged in the real space as the attribute information of the design member A4. Similarly, the attribute information providing unit 204 can provide the design member R2 arranged in the real space as the attribute information of the design member D1 to be removed. Further, the attribute information providing unit 204 can provide the design member R3 arranged in the real space as the attribute information of the design member MV1 that generates the moving motion object. Similarly, as the attribute information of the design member MV2 that generates the moving motion object, it is possible to add a virtual object arranged in the three-dimensional model space, for example, an electric pump ZMV2 (FIG. 14) described later.

  FIG. 4 is a diagram schematically illustrating a design member generated by the design processing unit 204. As shown in FIG. 4, the design processing unit 204 generates and arranges the design members A1 to A3 in the three-dimensional model space based on the attribute information given to the design members. For example, the design processing unit 204 generates and arranges the design members A1 to A3 in the three-dimensional model space using the information and the position information of the three-dimensional CAD model assigned to the design member.

  The design processing unit 204 arranges the design member A4 in the three-dimensional model space based on the existing design data of the design member R1 already arranged in the real space. As described above, the design processing unit 204 can arrange the design members A4, D1, MV1, and the like in the three-dimensional model space with respect to the existing members.

  FIG. 5 is a diagram in which the design members A1 to A4 generated by the design processing unit 204 are converted into virtual objects ZA1 to ZA4 by the conversion unit 106b and are superimposed on the real space. The upper diagram is an image of the real space, and the lower diagram is a diagram in which the virtual object is superimposed on the real space. As shown in FIG. 5, the virtual objects ZA1 to ZA4 of the design members A1 to A4 can be superimposed on the construction position based on the design, and the construction drawing can be visualized in the real space. Further, by superimposing a virtual object of the design member A4 corresponding to the design member R1 arranged in the real space on the real space, it is possible to confirm whether the design member R1 is arranged at the designed position. It becomes possible.

  FIG. 6 is a diagram in which the design member D1 of the design member R2 to be removed is converted into the virtual object ZD1 and is superimposed on the real space. The upper diagram is an image of the real space, and the lower diagram is a diagram in which the virtual object ZD1 is superimposed on the real space image. As shown in FIG. 6, by superimposing the virtual object ZD1 of the design member D1 corresponding to the design member R2 placed in the real space on the real space image, the position of the design member R2 to be removed can also be confirmed. . For example, the image processing unit 110b can further superimpose a mark E1 or the like indicating the design member R2 to be removed.

  FIG. 7 is a diagram in which the design member MV1 of the movable design member R3 is converted into a virtual object ZMV1 and superimposed on the real space. Here, the movable design member R3 is a lid of the device that opens and closes. The upper diagram is an image of the real space, and the lower diagram is a diagram in which the virtual object ZD1 is superimposed on the real space. As shown in FIG. 7, the image processing unit 110b generates a plurality of virtual objects ZMV1 of the design member MV1 by changing the coordinates, and superimposes the virtual objects ZMV1 on the real space image in a time series. Thereby, it becomes possible to visually recognize the movable range of the movable member R3.

The detailed processing of the image recognition processing unit 108b will be described with reference to FIGS.
FIG. 8 is a diagram illustrating a mark point E1 on the virtual object superimposed on the real space and a fingertip E2 recognized by the image recognition processing unit 108b. As illustrated in FIG. 8, the image recognition processing unit 108b determines the point at which the line connecting the user's viewpoint position and the recognized fingertip E2 intersects either the real space or the virtual object superimposed on the real space. The coordinates are output to the image processing unit 110b. The image processing unit 110b superimposes the mark point E3 on the real space or a virtual object superimposed on the real space based on the three-dimensional coordinates.

  FIG. 9 is a diagram showing a mark point E3 on the virtual object superimposed on the real space and a mark E4 indicating a fixed point. As shown in FIG. 9, when the position is determined, the image processing unit 110b superimposes the mark E4 on the real space image.

  FIG. 10 is a diagram showing a mark point E5 in the real space and a fingertip E2 recognized by the image recognition processing unit 108b. As illustrated in FIG. 10, the image recognition processing unit 108b performs a three-dimensional intersection of a line connecting the viewpoint position of the user and the recognized fingertip E2 with either the real space or the virtual object superimposed on the real space. The coordinates are output to the image processing unit 110b. The image processing unit 110b superimposes the mark point E5 on the real space or the virtual object superimposed on the real space based on the three-dimensional coordinates.

  FIG. 11 is a diagram showing a mark point E5 in the real space and a mark E6 indicating a fixed point. As shown in FIG. 11, when the position is determined, the image processing unit 110b superimposes the mark E6 on the real space image.

  FIG. 12 is a diagram illustrating a distance E7 between the mark point E1 on the virtual object and the mark point E5 on the real space. As shown in FIG. 12, when the positions of the two points are determined, the image recognition processing unit 108b calculates the distance E7 between the two points and outputs the distance E7 to the image processing unit 110b. The image processing unit 110b superimposes information indicating the distance E7 on the physical space image. As described above, since the distance between the mark point E1 on the virtual object and the mark point E5 on the real space can be obtained, accurate distance information can be obtained even when the dimensions of the real space are different from the design drawing. Can be obtained. This makes it possible to correct the design information more accurately.

  For example, when a design member is generated using a design support tool while the head-mounted image unit 10 is worn by an operator, and the design member is arranged in a three-dimensional model space for design, the image recognition processing unit 108b It is possible to use the distance between the mark point E1 on the virtual object and the mark point E5 in the real space, which has been acquired by the user. As described above, the design unit 20 can arrange the design members in the three-dimensional model space using the distance information acquired by the image recognition processing unit 108b.

The free movement processing of the virtual object in the design member MV2 will be described with reference to FIGS.
FIG. 13 is a diagram illustrating a mark point E8, which is an arrangement point of the virtual object ZMV2 in the design member MV2 superimposed on the real space. As illustrated in FIG. 13, the image recognition processing unit 108b outputs, to the image processing unit 110b, the intersection of the line connecting the user's viewpoint position and the recognized fingertip with the real space as three-dimensional coordinates. The image processing unit 110b superimposes the mark point E8 on the real space or the virtual object superimposed on the real space based on the three-dimensional coordinates.

  FIG. 14 is a diagram illustrating the virtual object ZMV2 of the design member MV2 superimposed on the real space. As shown in FIG. 14, the image processing unit 110b arranges, for example, the mark point E8 with the center coordinates of the bottom surface of the virtual object ZMV2.

  FIG. 15 is a diagram illustrating movement of the virtual object ZMV2 superimposed on the real space. As illustrated in FIG. 15, the image processing unit 110b changes the position of the virtual object ZMV2 in the direction specified by the operator, and superimposes the virtual object ZMV2 on the real space in a time series. As can be seen from these, there is an obstacle on the movement path of the virtual object, and the design member MV2 cannot move in the indicated direction.

  FIG. 16 is a diagram illustrating the virtual object ZMV2 of the design member MV2 rearranged at a different position. The mark E indicates the movement direction instructed by the operation. Thus, the operator can instruct the moving direction of the design member MV2.

  FIG. 17 is a diagram showing the movement of the rearranged design member MV2. As shown in FIG. 16, there is no obstacle on the movement path of the virtual object ZMV2, and the design member MV2 can be moved in the indicated direction. In this way, it is possible to arrange the virtual object ZMV2 of the design member MV2 at an arbitrary position in the real space and move it in an arbitrary moving direction. This makes it possible to visually check whether the design member MV2 can be moved.

FIG. 18 is an example of a flowchart showing the flow of image processing in the image processing apparatus 1.
As shown in FIG. 18, the operator inputs attribute information of a design member used for construction by the attribute information providing unit 202 (step S100). The operator instructs the coordinates corresponding to the position where the design member is to be arranged in the three-dimensional model space displayed on the display unit 30 by using the input operation unit 206, for example. The operator selects a design member from a list of three-dimensional CAD models displayed on the display unit 30, for example. The attribute information assigning unit 202 assigns such information as attribute information of the design member. At this time, coordinates for arranging the marker 40 in the three-dimensional model space are also set.

  Next, the design processing unit 204 generates a design member based on the attribute information provided by the attribute information providing unit 202, and arranges the design member in a three-dimensional model space for design (step S102). These design members are displayed on the display unit 30, for example, and are visually recognized by the operator.

  Next, the marker 40 is arranged at a position in the real space corresponding to the coordinates of the marker 40 set in the three-dimensional model space, and an image is taken by the camera 102 (step S104). Subsequently, the positioning unit 106a performs positioning between the coordinate system in the three-dimensional model space and the coordinate system in the real space based on the information of the captured marker 40 (step S106).

  Next, the three-dimensional space recognition processing unit 108c acquires natural feature points from the image data captured by the camera 102 (step S108), and based on the correspondence between the landmark data and the natural feature points, the head-mounted video unit 10 position and orientation are estimated (step S110).

  Next, the conversion unit 106b converts the information on the design members generated by the design processing unit 204 into a virtual object. Thereby, the virtual object generated based on the design member designed by the computer-aided design is arranged in the three-dimensional model space. Subsequently, the image processing unit 110b superimposes the real space image observed from the user's viewpoint position and the image of the virtual object observed from the user's viewpoint position, based on the estimation result of the position and orientation (step). S112).

  The control unit 110a determines whether to continue the image processing control (Step S114). When the image processing control is continued (NO in step S114), the processing from step S108 is repeated. On the other hand, when ending the image processing control (YES in step S114), the entire process ends.

  As described above, according to the present embodiment, the positioning unit 106a performs positioning between the coordinates in the three-dimensional model space where the virtual objects generated based on the design members are arranged and the coordinates in the real space, and The processing unit 110b generates an image of the virtual object observed from the viewpoint of the user, based on the alignment, in correspondence with the real space image observed from the viewpoint of the user. Thereby, it becomes possible to superimpose and display the virtual object generated based on the design member on the real space image, and to check the construction result using the design member before construction.

(2nd Embodiment)
The image processing apparatus 1 according to the first embodiment is different from the image processing apparatus 1 according to the first embodiment in that the attribute information providing unit 202 can further provide process order information as attribute information to a plurality of design members. Is different from Hereinafter, points different from the first embodiment will be described.

  FIG. 19 is a diagram illustrating a result of the attribute information assigning unit 202 according to the second embodiment assigning information on a process order as attribute information for each design unit. As shown in FIG. 19, the attribute information assigning unit 202 according to the present embodiment can further assign a process order as attribute information.

FIG. 20 is a diagram in which the virtual objects ZA8 and ZA9 of the design members A8 and A9 to which the attribute information 2018/7/09 is added are superimposed and displayed on the real space image.
FIG. 21 is a diagram in which virtual objects ZA10 and ZA11 of design members A10 and A11 to which attribute information 2018/7/16 is added are further superimposed and displayed on a real space image.
FIG. 22 is a diagram in which virtual objects ZA13 and ZA13 of design members A12 and A13 to which attribute information 2018/7/23 is added are further superimposed and displayed on the real space image.

  As shown in FIGS. 20 to 22, the image processing unit 110b superimposes and displays virtual objects on a real space image in the order according to the attribute information. Further, the image processing unit 110b generates an image indicating a date corresponding to the attribute information and superimposes the image on the physical space image.

  As shown in FIG. 20, the image processing unit 110b superimposes and displays the virtual objects on the real space image in the order according to the attribute information.

  As described above, according to the present embodiment, the attribute information assigning unit 202 assigns process order information as attribute information for each design member. Thereby, the image processing unit 110b can superimpose and display the virtual objects on the real space image in the order according to the attribute information. In this way, it is possible to superimpose and display the virtual objects generated based on the design members on the real space image in the order of steps, and to check the construction process using the design members before the execution of the construction.

  At least a part of the data processing method in the image processing apparatus 1 according to the present embodiment may be configured by hardware or software. When configured by software, a program for realizing at least a part of the function of the data processing method may be stored in a recording medium such as a flexible disk or a CD-ROM, and read and executed by a computer. The recording medium is not limited to a removable medium such as a magnetic disk or an optical disk, but may be a fixed recording medium such as a hard disk device or a memory. Further, a program that realizes at least a part of the function of the data processing method may be distributed via a communication line (including wireless communication) such as the Internet. Further, the program may be encrypted, modulated, or compressed, and distributed via a wired or wireless line such as the Internet, or stored in a recording medium.

  While some embodiments have been described above, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. The novel devices, methods, and programs described herein can be implemented in various other forms. In addition, various omissions, substitutions, and changes can be made to the forms of the apparatus, method, and program described in this specification without departing from the spirit of the invention.

1: image processing apparatus, 10: head mounted image section, 20: design section, 30: display section, 40: marker, 106a: alignment section, 106b: conversion section, 108: recognition processing section, 108b: image recognition processing Unit, 110b: image processing unit, 202: attribute information giving unit, 204: design processing unit, 206: input operation unit

Claims (10)

A positioning unit for performing positioning between a coordinate system in a three-dimensional model space for arranging a virtual object generated based on a design member designed by computer-aided design and a coordinate system in a real space;
Based on the alignment, an image processing unit that generates an image of the virtual object observed from the viewpoint position of the user, corresponding to a real space image observed from the viewpoint position of the user,
A display unit that superimposes and displays the real space image and the virtual object;
An image processing apparatus comprising: A design unit for arranging design members in the three-dimensional model space;
A conversion unit that converts information of the design members arranged by the design unit into the virtual object,
The image processing apparatus according to claim 1, further comprising: Further comprising an input operation unit for operating the design unit,
The image processing according to claim 2, wherein the conversion unit converts a design member in the three-dimensional model space arranged via the input operation unit into the virtual object observed from a viewpoint position of the user. apparatus. The design unit arranges the design members in the three-dimensional model space based on design data of members arranged in the real space,
The image processing device according to claim 2, wherein the image processing unit superimposes and displays a virtual object of the placed member in the real space.   The image processing according to claim 4, further comprising an image recognition processing unit configured to acquire distance information between coordinates in the real space image in the superimposed image and coordinates on the image of the virtual object. apparatus.   The image processing device according to claim 5, wherein the design unit arranges a design member in the three-dimensional model space using the distance information acquired by the image recognition processing unit. An attribute information assigning unit that assigns process order information as attribute information to the plurality of design members arranged by the design unit,
The image processing apparatus according to claim 2, wherein the image processing unit superimposes and displays virtual objects corresponding to a plurality of design members on the real space image in accordance with the process order.   The image processing device according to claim 1, wherein the image processing unit generates a movable object of the virtual object and superimposes and displays the movable object on the real space in a time series.   The image processing device according to claim 8, wherein the image processing unit moves the movable object in a predetermined direction in the real space. An alignment step of aligning a coordinate system in a three-dimensional model space for arranging a virtual object generated based on a design member designed by computer-aided design with a coordinate system in a real space;
Based on the alignment, an image processing step of generating an image of the virtual object observed from the viewpoint position of the user, corresponding to a real space image observed from the viewpoint position of the user,
A display step of superimposing and displaying the real space image and the virtual object;
An image processing method comprising:

Images