JP2015011368A - Display control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a display control device capable of displaying and confirming a change in the state of an object to be imaged accompanied by a work by VR(Virtual Reality) or AR(Augmented Reality) or the like in a real time or by simulation.SOLUTION: A display control device includes: an imaged data input part 2 for inputting the imaged data of an object to be imaged which is imaged by a camera 21; a position and attitude input part 3 for inputting information on the position and attitude of the camera 21 or a user who holds the camera 21; an element information input part 4 for inputting information on an element causing a change in the state of the object to be imaged; a storage part 7 for storing the virtual model of the object to be imaged as a display object; an arithmetic part 5 for calculating a state virtual model indicating the state of the object to be imaged when viewed from the imaging direction of the camera 21 on the basis of the input information on the position and attitude and the information on the element and the information on the stored virtual model, and for synthesizing the input imaged data with the calculated state virtual model; and a display control part 6 for allowing a display 24 to display the synthesized synthetic image.

Description

  The present invention relates to a display control apparatus that displays a working state of an imaging target in virtual reality (VR) or augmented reality (AR).

  In recent years, VR and AR technologies have spread rapidly and are used on a daily basis due to the high performance of computers. As a technique for supporting daily life using AR and other sensors, for example, a technique shown in Non-Patent Literature 1 is disclosed. The technique shown in Non-Patent Document 1 is a technique for measuring a user's posture and position, capturing a real image, and displaying an annotated image in augmented reality on a head-mounted display (HMD) attached to glasses. It is disclosed.

  Also, Non-Patent Document 2 and Patent Document 1 disclose techniques that apply augmented reality and HMD techniques to the construction site and presentation inside the building after construction. Non-Patent Document 2 describes that it is possible to display the AR of the tension / measurement side line at the construction site, display the VR mark at the pile position, display the AR of the VR building, and walk through the interior. Moreover, the technique shown in Patent Document 1 acquires real images of an indoor space captured by a plurality of imaging devices, and compares the acquired actual image with a virtual image obtained by viewing a virtual model of the same indoor space from a virtual viewpoint. By calculating the correspondence between the real video and the virtual model, specify the surface area of the virtual model to map the texture in the real video, map the partial area in the real video after shape conversion, A VR image in which a virtual model after texture mapping is viewed from an arbitrary viewpoint is generated and displayed on a display device.

JP 2006-040053 A

Masatsugu Kido, “Development of Wear Oriented Information Partners that Extend Daily Life from Portable to Wearable”, [online], September 12, 2005, 11th NAIST Industry-Academia Collaboration Forum Presentation Material, [Search May 26, 2013 ] Internet <http://www.science-plaza.or.jp/about/sankangaku/forum11/kidode.pdf> Hiroshi Imai, “Study of field support system using AR technology” [online], Taisei Construction Technology Center Report 2008 NO. 41, [Search June 17, 2013], Internet <http://www.taisei.co.jp/giken/report/2008_41/intro/C041_056.htm>

  The techniques shown in the above non-patent documents and patent documents are techniques that support the user's work and operations, but cannot display changes in the imaging target as the work process proceeds.

  The present invention provides a display control apparatus capable of displaying and confirming a change in the state of an object to be imaged due to work in real time using VR, AR or the like or by simulation.

  The display control apparatus according to the present invention includes an imaging data input unit that inputs imaging data of an imaging target captured by a camera, and a position and orientation that inputs information about the position and orientation of the camera or a user who holds the camera. Input means; element information input means for inputting information relating to an element that causes a change in the state of the imaging target; storage means for storing a virtual model of the imaging target to be displayed; and the input position And a state virtual model indicating the state of the imaging target when viewed from the imaging direction of the camera based on the information on the posture and the information on the element and the stored information on the virtual model Image processing means for combining the captured image data and the calculated state virtual model, and display means for displaying the combined image. It is.

  As described above, in the display control apparatus according to the present invention, the information regarding the input camera or the position and orientation of the user holding the camera, the information regarding the element that causes the change in the state of the imaging target, and Based on the information of the virtual model of the imaging target stored in advance, a state virtual model indicating the state of the imaging target when viewed from the imaging direction of the camera is calculated, and calculated with imaging data captured by the camera By displaying a synthesized image obtained by synthesizing the state virtual model, the state of the imaging target can be confirmed by the AR, which can be used to improve work efficiency.

  In the display control device according to the present invention, the information related to the element is based on the operation information of the element.

  As described above, in the display control device according to the present invention, the state virtual model is calculated based on the information on the position and orientation of the camera and the operation information of the element that causes a change in the state of the imaging target. Therefore, it is possible to display a real AR from the viewpoint of the user, and it is possible to efficiently perform work while viewing the AR image while viewing the actual reality.

  In addition, since the state virtual model is calculated from the motion information of the element that causes a change in the state of the imaging target, the state virtual model is detected from the motion information of the element and the accurate state virtual model is calculated. There is an effect that can be.

  In the display control device according to the present invention, the information on the element is based on information on members constituting the imaging target.

  As described above, in the display control device according to the present invention, information on the position and orientation of the camera and information on members constituting the imaging target, which is information on elements that cause a change in the state of the imaging target. Based on this, the state virtual model is calculated, so that it is possible to display a realistic AR from the user's viewpoint, and it is possible to work efficiently while looking at the AR image while viewing the actual reality. There is an effect.

  In addition, since the state virtual model is calculated from the information of the members constituting the imaged object, which is information on the element that causes a change in the state of the imaged object, the state change of the imaged object is detected from the constituent members of the imaged object. It is possible to detect and calculate an accurate state virtual model.

  In the display control device according to the present invention, the imaging data is obtained by imaging the imaging target in the direction of the face of the user wearing the glasses with the camera mounted on the glasses. The information on the position and posture of the person is based on the information measured by the sensor attached to the glasses, and the display means is a monocular head mounted display attached to the glasses.

  As described above, in the display control apparatus according to the present invention, the eyeglasses have a camera that captures the face direction of the user, various sensors that acquire information on the position and orientation of the camera, and a monocular head that displays a composite image. Since a mounted display is attached and a state virtual model is calculated based on information on the position and orientation of the camera and element information that causes a change in the state of the imaged object, it is realistic from the user's perspective. As a result, it is possible to display an AR, and it is possible to efficiently perform an operation while viewing an AR image while viewing the actual reality.

  The display control apparatus according to the present invention includes a line-of-sight information input unit that inputs information captured by a line-of-sight camera that is mounted on the eyeglasses and captures an image of the user's eye area; A visual direction detection means for detecting the visual line direction of the user based on the detected visual direction, and an input instruction specifying means for specifying the input instruction of the user based on the detected visual line direction; The composite image is corrected based on the specified input instruction.

  As described above, in the display control device according to the present invention, the user's line of sight is detected and the input instruction by the line of sight is specified, so that the user's operation is simplified. There is an effect that the display state of the display can be controlled only by movement.

  In the display control device according to the present invention, when the input instruction specifying unit specifies an input instruction for an arbitrary viewpoint and line-of-sight direction from the user, the image calculation unit is configured to specify the specified viewpoint and line-of-sight direction. The state virtual model viewed from the above is calculated, and the display means displays only the state virtual model of the composite image.

  Thus, in the display control device according to the present invention, when an input instruction for an arbitrary viewpoint and line-of-sight direction from the user is specified, a state virtual model viewed from the specified viewpoint and line-of-sight direction is calculated, Since only the state virtual model is displayed on the display, the state of the imaging target from an arbitrary viewpoint or line-of-sight direction specified by the user can be confirmed with VR, and the working efficiency can be improved.

  In the display control apparatus according to the present invention, the storage means stores information related to a work process for the object to be imaged, and the image calculation means determines the state virtual in a predetermined work process based on the information related to the work process. The model is calculated.

  As described above, in the display control device according to the present invention, the information on the work process for the imaging target is stored, and the state virtual model in the predetermined work process is calculated based on the information on the work process. The state of the object to be imaged can be confirmed by AR every time, and the working efficiency can be improved.

It is the figure which showed the field in the case of performing ground improvement typically. It is a functional block diagram which shows the structure of the display control apparatus which concerns on 1st Embodiment. It is a figure which shows an example of imaging data and a three-dimensional state virtual model. It is a figure which shows an example of the synthesized image which synthesize | combined imaging data and the three-dimensional state virtual model. It is a figure which shows an example of imaging data and a three-dimensional state virtual model. It is a figure which shows an example of the synthesized image which synthesize | combined imaging data and the three-dimensional state virtual model. It is a flowchart which shows operation | movement of the display control apparatus which concerns on 1st Embodiment. It is a figure which shows the mounting image of the display control apparatus which concerns on 2nd Embodiment. It is a figure which shows the change of the three-dimensional state virtual model accompanying a user's operation | movement in the display control apparatus which concerns on 2nd Embodiment. It is a functional block diagram which shows the structure of the display control apparatus which concerns on 3rd Embodiment. It is a functional block diagram which shows the structure of the gaze calculating part in the display control apparatus which concerns on 3rd Embodiment. It is a functional block diagram which shows the structure of the display control apparatus which concerns on 4th Embodiment.

  Embodiments of the present invention will be described below. The present invention can be implemented in many different forms. Also, the same reference numerals are given to the same elements throughout the present embodiment.

(First embodiment of the present invention)
A display control apparatus according to the present embodiment will be described with reference to FIGS. The display control apparatus according to the present embodiment is capable of displaying and confirming a working state of an imaging target using a virtual reality (VR) or augmented reality (AR) technique. In addition, display of the imaging target after the work is performed, and display of the imaging target can be controlled in various states according to the user's operation using various sensors.

  In the present embodiment, for example, display control of ground improvement at a construction site or building state of a building is performed using AR or VR technology. In addition, the display control apparatus according to the present embodiment can be applied not only to a construction site but also to various places (for example, a factory for manufacturing, a site for planting, a restaurant, and the like).

  FIG. 1 is a diagram schematically showing the site in the case of ground improvement. Here, it is assumed that ground improvement is performed for an area surrounded by a river flowing from north to south and a road passing from east to west. In the ground improvement, as shown in FIG. 1, the target area is usually divided into a grid and work is performed for each section. The work is performed by stirring the earth and sand and the solidified material while excavating the ground by the rotation of a stirrer attached to the arm tip of the construction machine. By dividing the target area into a grid, it is possible to identify and identify the work target area and the area where the work has been completed, but it is necessary to actually draw a grid-like partition line on the site. It is the cause of the decrease in efficiency.

  In the present embodiment, a ground improvement state viewed from the user's viewpoint is created from a user's position and orientation information using a three-dimensional virtual model, and is displayed superimposed on actual imaging. By doing so, it is not necessary to actually perform lane markings or the like on the site, and work management can be easily performed on a computer.

  FIG. 2 is a functional block diagram showing the configuration of the display control apparatus according to the present embodiment. The display control apparatus 1 inputs image data input unit 2 that inputs image data captured by the camera 21 and information on the position and orientation of the camera 21 measured by the sensor unit 22 or the user holding the camera 21. A position / orientation input unit 3 to be read, an element information input unit 4 for inputting element information 23 relating to an element that causes a change in the object to be imaged, and a three-dimensional virtual model of the object to be imaged stored in the storage unit 7 A calculation unit 5 that generates a three-dimensional state virtual model viewed from the viewpoint of the camera 21 from the position / posture information and the element information, and combines the generated three-dimensional state virtual model with the input imaging data; Display control means 6 for displaying the synthesized image on the display 24.

  In the case of the ground improvement of FIG. 1, the camera 21 is fixedly installed so as to capture an image in a predetermined direction with an arbitrary position outside the construction machine or in the driver's seat as a viewpoint. The predetermined direction is, for example, a direction in which an object to be operated is present, and in this case, the predetermined direction is the front of the construction machine. That is, the camera 21 always captures the direction of the work area as a moving image or a still image. FIG. 3A shows an example of a still image captured by the camera 21. It is the image which imaged the direction of the ground used as a work object from the driver's seat of the construction machine in Drawing 1, ie, the front direction of a construction machine.

  Similarly to the camera 21, the sensor unit 22 is also installed at an arbitrary position outside the construction machine or in the driver's seat, and the position (for example, using a GPS receiver) and posture (for example, using an acceleration sensor or a gyro sensor) of the camera 21. Measure. At this time, since each sensor is difficult to install on the camera 21 itself, changes in the position and orientation of the construction machine are measured and replaced with information on the position and orientation of the camera 21.

  In addition, when a GPS receiver is installed in the driver's seat of the construction machine, there is a possibility that position information cannot be acquired. Therefore, it is desirable that the GPS receiver is installed outside the construction machine. In that case, based on the relative relationship between the position where the GPS receiver is installed and the position where the camera 21 is installed, the position information acquired by the GPS receiver is corrected by the processing of the calculation unit 5 to be described later. You may enable it to acquire exact position information.

  The element information 23 is information on the path and depth along which the stirrer has moved, and this information may be acquired from a sensor attached to the tip of the arm of the construction machine, or the movement of the arm from the operation history of the construction machine. May be calculated as element information 23. That is, the element information 23 is information (here, movement information) relating to an element (here, a stirrer) that causes a change in the ground to be imaged.

  The calculation unit 5 reads the three-dimensional virtual model of the ground improvement stored in the storage unit 7, the position and orientation information of the camera 21 input by the position and orientation input unit 3, and the agitator input by the element information input 4. Based on the movement information, a three-dimensional state virtual model of ground improvement viewed from the camera 21 is generated. FIG. 3B shows an example of the generated three-dimensional state virtual model. The hatched area indicates the path along which the stirrer has moved, that is, the area where the ground improvement work has been completed. In FIG. 3 (B), the sections 31 to 33 indicate that the work has been performed through the maximum depth that the stirrer needs to be improved. For the sections 34 and 35, the stirrer is improved. This indicates that the work has been carried out through 2/3 of the maximum depth required, and for the compartment 36 and compartment 37, the work is carried out through 1/3 of the maximum depth at which the stirrer needs improvement. It shows that it was broken. That is, the three-dimensional state virtual model indicates that the ground improvement work is completed for the sections 31 to 33, the sections 34 to 37 are in the middle of the process, and the other sections are not started. .

  Then, the calculation unit 5 combines the image data captured by the camera 21 and the three-dimensional state virtual model, and the combined image is displayed on the display 24 by the display control unit 6. FIG. 4 is a diagram illustrating an example of a composite image display. As shown in FIG. 4, the ground improvement work state by AR can be displayed by superimposing the real imaging data and the virtual three-dimensional state virtual model.

  In addition, the sensor which measures the information regarding the camera 21, the sensor part 22, the stirrer as the element information 23, the display 24, etc. may be integrally provided in the construction machine, or may be installed externally. Similarly, the display control apparatus 1 itself may be regarded as an apparatus integrated with the construction machine, or may be installed as a separate body.

  Next, processing of the display control apparatus according to the present embodiment when building a building will be described with a specific example. FIG. 5A is a diagram illustrating an example of a still image captured by the camera 21. This is an image of the state where only the foundation of the house is completed on the land to be built. FIG. 5B shows a three-dimensional state virtual model generated by the calculation unit 5. The three-dimensional state virtual model is generated based on the input position and orientation information and the element information 23. Here, the element information 23 includes information on members constituting the house to be imaged. Has been. That is, in the example of FIG. 5B, based on the information such as the design drawing, the information on the type and quantity of the wood constituting the housing of the first floor portion of the house is input as the element information 23, and configured by these members A three-dimensional state virtual model of the object to be imaged is generated.

  Then, as shown in FIG. 6, the calculation unit 5 combines the image data captured by the camera 21 with the three-dimensional state virtual model, and the combined image is displayed on the display 24 by the display control unit 6. By superimposing the actual imaging data and the virtual three-dimensional state virtual model, it is possible to display the work state of the building with AR.

  FIG. 7 is a flowchart showing the operation of the display control apparatus according to the present embodiment. First, the position / orientation input unit 3 inputs the measurement result from each sensor unit 22 as information on the position and orientation of the camera (S1). The element information input unit 4 provides element information (for example, movement information of the arm tip in the case of ground improvement, information on members constituting the building in the case of building construction) that causes a change in the imaging target. It inputs as information 23 (S2). The calculation unit 5 reads the three-dimensional model from the storage unit 7 in which the three-dimensional model to be imaged is stored in advance (S3), and based on the input camera position and orientation information and element information. Then, the current state of the three-dimensional model viewed from the viewpoint of the camera 21 is calculated to generate a three-dimensional state virtual model (S4). The imaging data input unit 2 inputs the video of the camera as imaging data (S5), the calculation unit 5 combines the generated three-dimensional state virtual model and the input imaging data, and the display control unit 6 combines them. The synthesized image thus displayed is displayed on the display 24 (S6), and the process is terminated.

  As described above, according to the display control apparatus 1 according to the present embodiment, information on the position and orientation of the camera, element information 23 on an element that causes a change in the state of the imaging target, and the storage unit 7 in advance. Based on the stored information of the three-dimensional virtual model of the imaging target, a three-dimensional state virtual model indicating the state of the imaging target when viewed from the imaging direction of the camera is calculated, and imaging data captured by the camera 21 By displaying a composite image obtained by combining the calculated three-dimensional state virtual model, the state of the imaging target can be confirmed by AR, which can be used to improve work efficiency.

  In addition, a three-dimensional state virtual model is calculated based on information on the position and orientation of the camera 21, operation information of elements that cause a change in the state of the imaging target, and information on members constituting the imaging target. Therefore, it is possible to display a realistic AR from the viewpoint of the user, and it is possible to efficiently work while viewing the AR image while viewing the actual reality.

(Second embodiment of the present invention)
The display control apparatus according to the present embodiment will be described with reference to FIGS. In addition, in this embodiment, the description which overlaps with the said 1st Embodiment is abbreviate | omitted.

  In the case of FIG. 8A, a camera 21 that captures the direction of the user's face and a display 24 that outputs an image to the user's eyes are arranged on the glasses 80, and the user wears the glasses 80. The imaging target is imaged and displayed in the state. In FIG. 8A, a sensor (GPS receiver, acceleration sensor, gyro sensor, etc.) for detecting the position and posture of the user wearing the spectacles 80 is built in the detection device 81, and each measurement is performed. . The management of measurement data and other data, the control of each process, the control of communication, etc. may be performed by the detection device 81, or the control device can be held by the user holding the control device 82 as necessary. 82 may be performed.

  In this case, based on the measurement information from the detection device 81 held by the user, the calculation unit 5 generates a three-dimensional state virtual model, and the imaging data of the user facing direction captured by the camera 21 Synthesize. The synthesized composite image is displayed on the display 24 disposed on the glasses 80. The user can perform work or the like while comparing the actual landscape with the AR image displayed on the display 24.

  In FIG. 8B, a camera 21 that captures the direction of the user's face and a monocular display 24 that outputs an image to the user's eyes are disposed on the glasses 80, and further, a GPS receiver 83, An acceleration sensor 84 and a gyro sensor 85 are provided. In addition, the user holds a control device 82 or a detection device 81 for managing each device. In this case, the calculation unit 5 generates a three-dimensional state virtual model based on the information of each sensor arranged in the glasses 80 and synthesizes it with imaging data in the direction of the face of the user captured by the camera 21. . The synthesized composite image is displayed on a monocular head mounted display 24 disposed on the glasses 80. The user can perform work or the like while comparing the actual landscape with the AR image displayed on the display 24.

  In any configuration, since a user can wear a necessary device, the device configuration can be simplified and the position, posture, and the like can be accurately detected. In addition, by using a portable terminal that is currently making significant progress as the detection device 81 and the control device 82, the manufacturing process of the device can be simplified and each device can be used flexibly according to the application.

  FIG. 9 is a diagram illustrating an example of an operation of a display image due to a change in position and orientation in a state where the display control apparatus according to the present embodiment is worn. Here, an example of a three-dimensional state virtual model of a scene viewed from a user in a state of being in a three-dimensional virtual building is shown. FIG. 9A shows a case where the user looks up, and FIG. 9B shows a case where the user looks down. The ceiling part and the floor part of the building are respectively displayed, and it can be seen that the three-dimensional state virtual model changes correspondingly according to the posture of the user.

  FIG. 9C shows the three-dimensional state virtual model when the user turns to the left, and FIG. 9D shows the three-dimensional state virtual model when the user turns to the right. As in the case of FIGS. 9A and 9B, it can be seen that the three-dimensional state virtual model changes correspondingly depending on the posture of the user.

  FIG. 9E shows a three-dimensional state virtual model when the user moves backward and FIG. 9F shows a case where the user moves forward. As the user's position changes, it can be seen that the three-dimensional state virtual model changes correspondingly.

  FIG. 9G shows the first floor, and FIG. 9H shows the second floor. In many cases, the user cannot actually move in the height direction, so the jump operation is detected. When the user jumps, the vibration and direction are detected, and a three-dimensional virtual model of the second floor at the same position is generated. Further, it is possible to perform a moving operation to a higher floor by further jumping, and to perform a moving operation to a lower layer by jumping on the top floor.

  As described above, in the display control apparatus according to the present embodiment, the glasses 21 detect the camera 21 that images the user's face direction, the position and posture of the user, or the position and posture of the glasses 80, Various sensors that acquire the detected information as position and orientation information of the camera 21 and a head-mounted display that displays a composite image obtained by synthesizing the imaging data and the generated three-dimensional state virtual model are mounted. It is possible to display a real AR from the viewpoint of the above, and it is possible to efficiently work while viewing the AR image while viewing the actual reality.

  In addition, when applying the display control apparatus 1 according to the present embodiment in the ground improvement described in the first embodiment, since the user is in the driver's seat of the construction machine, the GPS receiver disposed in the glasses 80 is It may exist in the driver's seat and position information may not be acquired. In that case, a GPS receiver may be installed outside the construction machine, and the position information of the camera 21 may be determined from the relative relationship between the position of the GPS receiver and the position of the driver's seat.

(Third embodiment of the present invention)
The display control apparatus according to the present embodiment will be described with reference to FIGS. In addition, in this embodiment, the description which overlaps with each said embodiment is abbreviate | omitted.

  FIG. 10 is a functional block diagram showing the configuration of the display control apparatus according to this embodiment. In FIG. 10, the difference from the case of FIG. 2 in the first embodiment is that the line of sight that performs the calculation of the line of sight based on the eye image captured by the line-of-sight camera 9 and identifies the user's input instruction The calculation unit 8 is provided.

  The line-of-sight calculation unit 8 calculates the user's line-of-sight direction based on the eye image data captured by the line-of-sight camera 9, specifies an input instruction, and sends the specified input instruction information to the calculation unit 5. The calculation unit 5 calculates a three-dimensional state virtual model and corrects a composite image according to the acquired input instruction information.

  The processing of the line-of-sight calculation unit 8 will be described in more detail. FIG. 11 is a functional block diagram illustrating a configuration of the line-of-sight calculation unit 8 in the display control apparatus according to the present embodiment. The line-of-sight calculation unit 8 includes a line-of-sight information input unit 11 that inputs image data of the line-of-sight camera 9 that captures the eye side of the glasses 80, a line-of-sight direction detection unit 12 that specifies the line-of-sight direction from the input eye image data, Based on the detected line-of-sight direction and a line-of-sight input storage unit 14 that stores input instruction information corresponding to a previously registered line-of-sight direction, an input instruction specifying unit 13 that specifies input instruction information input by the user with the line of sight With. The detection of the gaze direction can be performed using an existing technique. For example, the movement of the black eye part may be detected, or the gaze direction may be detected based on a change in the area of the white eye part. .

  The calculation unit 5 calculates a three-dimensional state virtual model and corrects the composite image based on the specified input instruction information. For example, when a viewpoint position and a viewpoint direction are specified as input information, a three-dimensional state virtual model viewed from the specified viewpoint position and viewpoint direction is generated. At this time, since the viewpoints of the captured image captured by the camera 21 and the generated three-dimensional state virtual model do not coincide with each other, display control in VR is performed without performing image synthesis. Also, for example, when there is a deviation between the combined image of the image data captured by the camera 21 and the three-dimensional state virtual model, the correction information is input by the line of sight (for example, the three-dimensional state virtual model By changing the position and angle according to the direction of the line of sight, the display state of the composite image is corrected and consistency with the imaging data can be obtained.

  As described above, in the display control apparatus 1 according to the present embodiment, the user's line of sight is detected and the input instruction by the line of sight is specified, so that the user's operation becomes simple, for example, even during other work. The display state of the display can be controlled only by moving the line of sight.

  In addition, when an input instruction for an arbitrary viewpoint and line-of-sight direction is specified from the user, the state virtual model viewed from the specified viewpoint and line-of-sight direction is calculated, and only the state virtual model is displayed on the display. The state of the object to be imaged from any viewpoint or line-of-sight direction designated by the person can be confirmed with VR, and work efficiency can be improved.

(Fourth embodiment of the present invention)
The display control apparatus according to the present embodiment will be described with reference to FIG. In addition, in this embodiment, the description which overlaps with each said embodiment is abbreviate | omitted.

  FIG. 12 is a functional block diagram showing the configuration of the display control apparatus according to this embodiment. In FIG. 12, the difference from the case of FIG. 2 in the first embodiment is that the storage unit 7 stores information regarding the work process of the work on the imaging target. The information regarding the work process is, for example, schedule information indicating how much work is completed by when. In addition, materials and parts to be used in addition to each schedule may be associated.

  The calculation unit 5 is based on information about the time inputted as one of the element information 23 (for example, information including temporal elements such as XX month XX day, XX day, XX time, etc.) The work state applicable to this time is calculated from the work process information stored in the storage unit 7, and a three-dimensional state virtual model is generated so as to be in the work state. The generated model is combined with the imaging data and displayed on the display 24 as described above.

  As described above, in the display control apparatus according to the present embodiment, the information on the work process for the imaging target is stored, and the state virtual model in the predetermined work process is calculated based on the information on the work process. The state of the object to be imaged can be confirmed by AR for each process, and work efficiency can be improved.

DESCRIPTION OF SYMBOLS 1 Display control apparatus 2 Imaging data input part 3 Position and orientation input part 4 Element information input part 5 Calculation part 6 Display control part 7 Memory | storage part 8 Gaze calculation part 9 Gaze camera 11 Gaze information input part 12 Gaze direction detection part 13 Input direction specification 13 Unit 14 line-of-sight input storage unit 21 camera 22 sensor unit 23 element information 24 display 80 glasses 81 detection device 82 control device 83 GPS receiver 84 acceleration sensor 85 gyro sensor

Claims (7)

Imaging data input means for inputting imaging data of an imaging target imaged by a camera;
Position and orientation input means for inputting information on the position and orientation of the camera or the user holding the camera;
Element information input means for inputting information on an element that causes a change in the state of the imaging target;
Storage means for storing a virtual model of the imaging target to be displayed;
A state virtual model indicating the state of the imaging target when viewed from the imaging direction of the camera, based on the input information on the position and orientation, the information on the element, and the stored information on the virtual model Image calculation means for synthesizing the input imaging data and the calculated state virtual model;
A display control apparatus comprising: display means for displaying a synthesized composite image. The display control device according to claim 1,
The display control apparatus according to claim 1, wherein the information related to the element is based on operation information of the element. The display control device according to claim 1,
The display control apparatus according to claim 1, wherein the information on the element is based on information on a member constituting the imaging target. The display control device according to any one of claims 1 to 3,
The imaging data is obtained by imaging the imaging target in the direction of the face of the user wearing the glasses with the camera mounted on the glasses, and information on the position or orientation of the camera or the user, A display control apparatus based on information measured by a sensor mounted on the glasses, wherein the display means is a monocular head mounted display mounted on the glasses. The display control device according to claim 4,
Line-of-sight information input means for inputting information captured by a line-of-sight camera that is mounted on the glasses and images the eye area of the user;
Gaze direction detection means for detecting the gaze direction of the user based on the input imaging information of the eye area;
Input instruction specifying means for specifying the user's input instruction based on the detected line-of-sight direction;
The display control apparatus, wherein the image calculation unit corrects the composite image based on the specified input instruction. The display control device according to claim 5,
When the input instruction specifying unit specifies an input instruction for an arbitrary viewpoint and line-of-sight direction from the user, the image calculation unit calculates the state virtual model viewed from the specified viewpoint and line-of-sight direction. Then, the display means displays only the state virtual model of the composite image. The display control device according to any one of claims 1 to 6,
The storage means stores information related to a work process for the imaging target;
The display control apparatus, wherein the image calculation means calculates the state virtual model in a predetermined work process based on information on the work process.