JP2011044130A - Image processing system and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing system and an image processing method for displaying new data so as to visually obtain processing results, such as strain and stress, of a sensor. <P>SOLUTION: The image processing system for analyzing secular changes in information of the sensors arranged in an actual space includes a means for acquiring photographed images in the actual space; a means for obtaining an obtaining time range of each frame of the photographed image; a means for obtaining sensor information sent from the sensor; a means for acquiring time at which sensor information is acquired as well as sensor information; a means for obtaining sensor positions; a means for forming virtual images related to the sensor information; and a means for displaying the virtual images as static images for each obtaining time, selecting the photographed images corresponding to the frames having time areas in which the obtaining time of the sensor information corresponding to the virtual images displayed as the static images, and outputting the virtual images as the static images by overlapping and combining the virtual images at positions corresponding to the sensor positions of the photographed images. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing system that displays a virtual image superimposed on a real image, and an image processing method thereof.

  Conventionally, in the inspection and investigation of buildings such as bridge members, data obtained from sensors such as strain gauges and displacement meters have been analyzed and displayed in a graph. However, these data have a problem that it is difficult to visually grasp what kind of distortion is in which place.

  Therefore, visualization software (for example, AVS, Visualizer, Paraview) for effectively displaying data has been proposed. According to this, the complement function is enriched, and it is excellent in imaging and vector display. However, it takes time because it is necessary to create a model to be a base for visualization. In addition, the display method has a problem that it is not clear where the data is displayed unless the accuracy of the model is increased.

  As another visualization technique, an augmented reality (AR) technique is used in the field of information communication. By using these technologies, an image in a virtual space such as CG is superimposed on a real space image, and what is not actually visible or supplementary information can be displayed to the user according to the situation. It can be displayed (for example, Patent Document 1, Non-Patent Documents 1 and 2).

JP 2006-99188 A

Nao Hashimoto ARTToolKit Introduction to Augmented Reality Programming By Taniji Toyosu ARTToolkit programming technique to realize augmented reality

  It is an object of the present invention to provide an image processing system and an image processing method for displaying new data so that sensor processing results such as strain and stress can be visually grasped.

  In the engineering field, in order to show objectivity, it is necessary to leave (show) a physical quantity acquired by a sensor or the like and an external force that induces it in a document. Animation is not a suitable format to leave on paper as a report. There is a need for a way to leave engineeringly meaningful information on paper. Therefore, the present invention provides means for easily leaving sensor information such as a physical quantity acquired by a sensor or the like on a sheet.

  A system that measures physical quantities such as strain and stress by simply using conventional AR technology. Even if it is observed dynamically (by animation), it can grasp the change sensuously, but details the changes in these behaviors. It was difficult to analyze. In fact, it is assumed that it will be used for applications that measure the strength of civil engineering buildings, etc., but in examining the reinforcing means, external forces over bridges (for example, especially the force accompanying the passage of vehicles, wind power, It is necessary to analyze in detail the strain, stress and acceleration generated according to temperature stress, seismic force, etc.). In such a case, a composite image in which a real image (the position of a running vehicle) and a virtual image (physical quantity or the like) are synchronized in strict time is required.

  Therefore, the present invention provides an image processing system and an image processing method for displaying an analysis result from a sensor or the like superimposed on a real image, enabling sensor information to be displayed and analyzed, and further providing sensor information and a real image. It is an object of the present invention to provide an image processing system and an image processing method that can analyze how sensor information changes based on a phenomenon that occurs in real space.

The present invention (1) is an image processing system for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time range acquisition means for acquiring an acquisition time range of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image can be displayed at each acquisition time, and the captured image corresponding to a frame having a time range including the acquisition time of sensor information corresponding to the virtual image displayed on the still image is selected. Image output means for superimposing and synthesizing the virtual image at a location corresponding to the position of the sensor in the captured image and outputting as a still image;
Is an image processing system.

The present invention (2) is an image processing system for analyzing changes over time in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time range acquisition means for acquiring an acquisition time range for acquiring sensor information together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image for each acquisition time range, and the captured image corresponding to a frame having a time included in the acquisition time range of sensor information corresponding to the virtual image displayed as the still image is selected. And an image output means for superimposing and synthesizing the virtual image at a position corresponding to the position of the sensor of the photographed image and outputting as a still image;
Is an image processing system.

The present invention (3) is an image processing system for analyzing changes over time in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Sensor information estimation for calculating an estimated value of sensor information at the captured image acquisition time of the frame from a plurality of sensor information at close times for at least one selected from the acquisition time of the frame of the captured image Means,
Virtual image forming means for forming a virtual image related to the estimated value of the sensor information;
A still image can be displayed for each frame of the captured image, and a virtual image related to the estimated value of the sensor information corresponding to the acquisition time of the captured image of the frame corresponds to the position of the sensor of the captured image Image output means for superimposing and synthesizing and outputting as a still image;
Is an image processing system.

The present invention (4) is an image processing system for analyzing changes over time in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image at each acquisition time, and the captured image corresponding to a frame having an acquisition time closest to the acquisition time of sensor information corresponding to the virtual image displayed as the still image is displayed. Selecting and outputting the virtual image as a still image by superimposing and synthesizing the virtual image at a location corresponding to the position of the sensor in the captured image;
Is an image processing system.

  The present invention (5) is the image processing system according to any one of the inventions (1) to (4), further comprising print output means for printing the still image by a printing apparatus.

The present invention (6) includes second captured image acquisition means for acquiring a second captured image in a real space different from the captured image;
Second captured image acquisition time or time range acquisition means for storing the acquisition time or acquisition time range of the second captured image;
Any one of the inventions (1) to (5), wherein the image output means displays two images of the second photographed image and the photographed image in synchronization with the photographed image or the virtual image. One image processing system.

The present invention (7) is an image processing method for analyzing temporal changes of sensor information arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time range acquisition step of acquiring an acquisition time range of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time acquisition step for acquiring a time at which the sensor information is acquired together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A virtual image forming step of forming a virtual image related to the sensor information;
The virtual image can be displayed at each acquisition time, and the captured image corresponding to a frame having a time range including the acquisition time of sensor information corresponding to the virtual image displayed on the still image is selected. An image output step of superimposing and synthesizing the virtual image on a location corresponding to the position of the sensor of the captured image and outputting as a still image;
Is an image processing method.

The present invention (8) is an image processing method for analyzing temporal changes of sensor information arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time acquisition step of acquiring an acquisition time of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time range acquisition step for acquiring an acquisition time range for acquiring sensor information together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A virtual image forming step of forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image for each acquisition time range, and the captured image corresponding to a frame having a time included in the acquisition time range of sensor information corresponding to the virtual image displayed as the still image is selected. And an image output step of superposing and synthesizing the virtual image on a location corresponding to the position of the sensor of the captured image and outputting as a still image;
Is an image processing method.

The present invention (9) is an image processing method for analyzing a change with time of information of sensors arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time acquisition step of acquiring an acquisition time of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time acquisition step for acquiring a time at which the sensor information is acquired together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A sensor information estimation step of calculating an estimated value of sensor information at the captured image acquisition time of the frame from a plurality of sensor information at close times for at least one selected from the acquisition time of the frame of the captured image When,
A virtual image forming step of forming a virtual image related to the estimated value of the sensor information;
A still image can be displayed for each frame of the captured image, and a virtual image related to the estimated value of the sensor information corresponding to the acquisition time of the captured image of the frame corresponds to the position of the sensor of the captured image An image output process for superimposing and combining and outputting as a still image;
Is an image processing method.

The present invention (10) is an image processing method for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time acquisition step of acquiring an acquisition time of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time acquisition step for acquiring a time at which the sensor information is acquired together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A virtual image forming step of forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image at each acquisition time, and the captured image corresponding to the frame having the acquisition time closest to the acquisition time of the sensor information corresponding to the virtual image displayed as the still image is displayed. An image output step of selecting and superimposing the virtual image on a position corresponding to the position of the sensor of the photographed image, and outputting as a still image;
Is an image processing method.

  The present invention (11) is a program for causing a computer to execute any one of the image processing methods (7) to (10).

  The meanings of various terms used in this specification will be described. Here, the “acquisition time range” of each frame of the captured image means a time range from the image acquisition start time of each frame to the image acquisition start time of the next frame. Here, the “acquisition time range” of the sensor information means a time range from the acquisition start time of the sensor information to the image acquisition start time of the next sensor information. The “acquisition time” of each frame of the photographed image is not particularly limited, but means a time for which a certain reference is set in each frame, for example, an acquisition start time of the image frame.

The present invention (A) is an image processing system for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time range acquisition means for acquiring an acquisition time range of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image and the captured image corresponding to a frame having a time range including the acquisition time of sensor information corresponding to the virtual image are selected, and the virtual image is selected at a location corresponding to the position of the sensor in the captured image. Image output means for superposing and synthesizing and outputting the images;
Is an image processing system.

The present invention (B) is an image processing system for analyzing changes over time of information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time range acquisition means for acquiring an acquisition time range for acquiring sensor information together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image and the captured image corresponding to a frame having a time included in the acquisition time range of sensor information corresponding to the virtual image are selected, and the virtual image is selected at a location corresponding to the position of the sensor in the captured image. Image output means for superposing and synthesizing and outputting the images;
Is an image processing system.

The present invention (C) is an image processing system for analyzing changes over time in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image and the captured image corresponding to the frame having the acquisition time closest to the acquisition time of the sensor information corresponding to the virtual image are selected, and the location corresponding to the position of the sensor in the captured image is selected. Image output means for superimposing and synthesizing and outputting the virtual image;
Is an image processing system.

  According to the present invention, there is an effect that it is possible to provide a data display format that can visually grasp the processing result of the sensor. Further, it is not necessary to create a model as in the conventional visualization software, and it is possible to provide a display form that can clearly grasp where the display of the processing result of each sensor is.

  Generally, the captured image acquisition frequency per second is different from the sensor information acquisition frequency, but information with high acquisition frequency is stored together with the time range, and information with high acquisition frequency is stored with time, and these information Are combined to generate a composite image. By configuring in this way, it is possible to superimpose and display captured images according to the sensor acquisition time, so it is possible to analyze changes in sensor information over time while comparing with phenomena in the real space where the sensors are arranged. It becomes possible.

  Even when the captured image acquisition frequency and the sensor information acquisition frequency are different, the acquisition time for each frame of the captured image and the acquisition time of the sensor information are stored, and the time closest to the sensor information is stored. A composite image is generated by selecting a captured image in the frame of the captured image. By configuring in this way, it is possible to superimpose and display captured images according to the sensor acquisition time, so it is possible to analyze changes in sensor information over time while comparing with phenomena in the real space where the sensors are arranged. It becomes possible.

  In addition, even if the captured image acquisition frequency and the sensor information acquisition frequency are different and the image and the sensor information are not acquired at the same time, the estimated value of the sensor information at the captured image acquisition time is calculated and the estimated By generating virtual images based on the values and displaying them in a superimposed manner, it is possible to analyze changes over time in the sensor information while collating with phenomena in the real space.

FIG. 1 is a block diagram illustrating a functional configuration of the image processing system according to the first embodiment. FIG. 2 is a diagram conceptually showing sensor information acquisition timing and image frame acquisition timing. FIG. 3 is a diagram illustrating a method for calculating an estimated value of sensor information at the image acquisition time. FIG. 4 is an image processed by the system according to the first embodiment with respect to a real image taken by the camera ((b) is an enlarged view of (a)). FIG. 5 is a flowchart of main processing of the image processing system according to the first embodiment. FIG. 6 is a flowchart showing details of the processing performed in step S360. FIG. 7 is a flowchart of an analysis process for analyzing a measurement result by the image processing system according to the present invention. FIG. 8 is a flowchart showing details of the process performed in step S360A for analyzing the measurement result by the image processing system according to the present invention. FIG. 9 is a flowchart of an analysis process for analyzing a measurement result by the image processing system according to the present invention. FIG. 10 is a flowchart of analysis processing for calculating an estimated value of sensor information by the image processing system according to the present invention. FIG. 11 is a flowchart of main processing of the image processing system according to the third embodiment. FIG. 12 is a block diagram illustrating a functional configuration of an image processing system according to the fourth embodiment. FIG. 13 is a diagram illustrating an example of an image generated by the image processing system according to the fourth embodiment.

First Embodiment FIG. 1 is a block diagram showing a functional configuration of an image processing system according to the best mode. As shown in FIG. 1, the system according to the best mode includes a camera 110 that captures an image of the real space R in which the marker M is arranged, a sensor 120 that is arranged around the marker M in the real space, and the sensor. A data logger 130 capable of acquiring the above information, an image processing system 140, a display device 160, and an input device 170. Here, the real space is not limited to the space shown on the screen, but means an actual space.

  The camera 110 captures a moving image in the real space. Each captured frame image is input to the image processing system 140 as an image signal. The camera 110 may be a camera for high-speed shooting with a large number of frames per second (for example, 50 frames or more per second). In this case, the number of frames taken by the camera may be higher than the sensor information acquisition frequency.

  A marker M is arranged in the real space, and a sensor 120 is arranged around the marker M. The form information of the marker M is known, and the form information of the marker M is stored in the arrangement information storage unit 141 in advance. Specifically, the color, shape and size of the marker M, the distance from the reference point to each side, and the like. Although only one marker M is shown in FIG. 1, there may be a plurality of markers. By providing two or more markers, it is possible to display a virtual image even when one marker is hidden, or to obtain an effect of improving the accuracy of specifying the sensor position. When a plurality of markers are present, the pattern of these markers is different for each marker. In addition, the process in the case of using a plurality of markers is well-known as described in publicly known literature (for example, Taniji Toyosu, ARTToolkit programming technique for realizing augmented reality, 2008). The marker M is not particularly limited as long as the position and orientation can be identified, but is preferably an asymmetric figure. In addition, when a marker is identified by color difference, it is preferable that the marker is composed of white and black that are most likely to cause color difference. In addition, the frame of the marker M has a quadrangular shape in the figure, but the shape is not particularly limited. For example, the frame may be any one of a triangle, a pentagon, a hexagon, a heptagon, and an octagon. Alternatively, it may be a two-dimensional code or a fractal graphic. Among these, a quadrangle is particularly suitable. Furthermore, it is preferable that, for example, a figure that can identify the direction or a figure composed of characters is drawn in the frame of the marker.

  Various known sensors can be used as the sensor 120 and are not particularly limited. For example, a strain gauge, a load cell, a displacement meter, a temperature sensor, a humidity sensor, an acceleration sensor, a gyro sensor, a distance sensor, a magnetic sensor, and an optical position sensor. And a light wave distance meter. Further, the number of sensors 120 is not limited to one and may be plural. These sensors 120 are connected to the data logger 130, process the measurement results by the sensors, and supply the processing results to the image processing system together with the acquisition time of the sensor information. In addition, among these sensors, by using a triaxial strain gauge in which three strain gauges are stacked, it becomes possible to calculate vector quantities such as principal strain and principal stress. The direction can be displayed more clearly than the conventional method.

  A known display device can be used as the display device 160 and is not particularly limited. For example, a display for a computer or a head mounted display (HMD) to be mounted on the user's head can be used. . Among these, it is preferable to use a computer display in order to display the analysis result later. The input device 170 is not particularly limited, and for example, a mouse or a keyboard can be used.

  Next, the image processing system 140 will be described.

  Upon receiving the captured image of each frame sent from the camera 110, the image input unit 142 receives the captured image from the acquisition time of the captured image (for example, the acquisition start time of the frame image) or the acquisition start time of the frame. It is sent to the image storage unit 143 and the marker detection unit 144 together with the frame acquisition time range up to the frame acquisition time. The acquisition time or the acquisition time range of these captured images may be those sent from imaging means such as a camera, or may be calculated in an image processing system.

  When the marker detection unit 144 receives a captured image from the image input unit 142, the marker detection unit 144 detects a marker shown in the captured image and calculates image coordinates. In addition, about the method of calculating | requiring an image coordinate and the attitude | position of the marker mentioned later, well-known literature (For example, the author of Hirokazu Kato, the augmented reality system based on marker tracking, its calibration, TVRSJ Vol.4 No.4 1999). As described, it is well known.

  The arrangement information generation unit 145 calculates marker arrangement information (position / posture information) based on the image coordinate information from the marker detection unit 144 and the form information of the marker M stored in the arrangement information storage unit 141. .

  The arrangement information storage unit 141 stores the arrangement information generated by the arrangement information generation unit 145 in addition to the shape information of the marker M described above.

  The imaging means position / orientation estimation unit 146 obtains (estimates) the position / orientation of the camera 110 in the current frame from the marker arrangement information obtained by the marker detection unit 144. The position and orientation of the camera 110 are nonlinearly optimized so that the error between the position (image coordinates) projected on the imaging surface of the camera 110 estimated for the immediately preceding frame and the image coordinates obtained by the marker detection unit 144 is minimized. Calculated by repeating the process. The position and orientation obtained here is the position and orientation of the camera 110 in the current frame.

  The image storage unit 143 temporarily stores the captured image received from the camera 110 via the image input unit 142 together with the acquisition time range of each frame of the image.

  The operation input unit 147 sends the instruction signal sent from the operation input device 170 to the imaging means position / orientation estimation unit 146 and the arrangement information editing unit 148.

  The arrangement information editing unit 148 has a function for editing the relative position information of the marker and the sensor. Here, the relative position information of the marker and the sensor indicates, for example, information on the coordinates where the sensor is located on the marker coordinates with one point of the marker as the reference point and the reference point as the origin.

  The sensor information input unit 151 sends the data obtained from the sensor sent from the data logger 130 and the data acquisition time to the sensor information processing unit 152.

  The sensor information processing unit 152 analyzes the data and obtains an analysis result by calculation. The analysis result calculated by the sensor information processing unit 152 is sent to the sensor information image generation unit 154.

  The sensor information storage unit 153 temporarily stores the analysis result obtained by the sensor information processing unit 152 and the corresponding data acquisition time.

  The sensor information image generation unit 154 generates a sensor information image reflecting the analysis result obtained by the sensor information processing unit 152 in a format specified by the operator. Here, the formats that can be specified are stored in the image information storage unit 155. For example, an image for indicating the result obtained by information processing in color, a circular image with a different size reflecting the scalar value of the result, When the obtained result is a vector value, an arrow image for reflecting the size and direction, a numerical display image, a graph display image, and other 3D display images can be used.

  The synthesizing unit 149 generates a synthesized image indicating an image that reflects the processing result of the sensor information on the captured image acquired from the image storage unit 143 at a location reflected on the position of the sensor. Specifically, the synthesizing unit 149 generates the sensor information image generated by the sensor information image generation unit 154 based on the imaging unit position and orientation information from the imaging unit position and orientation estimation unit 146 and the relative position information of the marker and the sensor. Generates a virtual image arranged at a location corresponding to the sensor position in the captured image. The captured image acquired from the image storage unit 143 and the virtual image are synthesized in synchronism.

  In the image processing system according to the present invention, the sensor measurement information and the image information are not only superimposed and displayed in real time, but the sensor information and the captured image are recorded and the virtual image reflecting the sensor information later And a still image superimposed on the photographed image. Alternatively, the still image may be output as an image file in JPG or BMP format.

  In the analysis process using a still image, the analysis unit 190 instructs the synthesis unit 149 to generate a still image based on the sensor information or frame image specified by the operator. The sensor information processing unit 152 reads necessary sensor information from the sensor information storage unit 153 and sends the analysis processing result to the sensor information image generation unit 154 to generate a virtual image. In addition, a necessary frame image is read from the image storage unit 143 and sent to the image input unit 142 and the composition unit 149. The image sent to the image input unit 142 is sent to the marker detection unit 144 and used for the space recognition process.

  More specifically, when the number of sensor data acquisition times per unit time is larger than the number of image frame acquisition times, a virtual image corresponding to the sensor information at the sensor information acquisition time t + x, and a captured image obtained by capturing a real space As shown in FIG. 2, the image information is synthesized in correspondence with the frame of the captured image having the acquisition time range including the acquisition time t + x of the sensor information corresponding to the virtual image, and is displayed as a still image.

  For example, when the sensor information acquisition time is t + 5/10, f1 having the time range t to t + 1 is combined as a corresponding captured image frame. In addition, a virtual image corresponding to the sensor information acquisition time from t, t + 1/10, t + 2/10, t + 3/10,... T + 9/10 is combined with f1 and output. Further, the virtual image corresponding to the sensor information acquisition time from the sensor information acquisition time t + 1 + 1/10 to t + 1 + 9/10 is synthesized with f2. In this way, it is possible to display a virtual image in conjunction with the sensor data acquisition timing, so even if a very fast change over time occurs, it will be analyzed later to track the change in sensor information over time. It becomes possible to do.

  Contrary to the above, when the number of acquisition times of image frames per unit time is larger than the number of acquisition times of sensor data, still images are displayed by selecting a frame having an image acquisition time included in the acquisition time range of sensor information. To do.

  In addition, the sensor information acquisition time and the image acquisition time may be stored, and a virtual image corresponding to the sensor information may be selected and displayed in an overlapping manner by selecting an image frame having a time closest to the sensor information acquisition time. it can.

  For example, when the sensor information acquisition time is t + 4/10, the image f1 of the frame having t that is the closest image acquisition time is selected and displayed in a superimposed manner. If the sensor information acquisition time is t + 6/10, the image f2 of the frame having the nearest image acquisition time t + 1 is selected and displayed as a still image. When the sensor information acquisition time is t + 5/10, f1 or f2 is selected and a still image is displayed. When either the sensor information acquisition frequency or the image acquisition frequency is high by selecting and overlapping the virtual image corresponding to the sensor information acquisition time by selecting the captured image of the frame with the closest acquisition time and displaying it. Even in this case, it is possible to display the images in a superimposed manner without changing the processing.

In addition to the still image display method as described above, a still image corresponding to the frame of the captured image is selected, an estimated value of sensor information at the acquisition time of the captured image is calculated, and the estimated value is related There is a method of forming a virtual image and displaying it superimposed. In this case, the estimated value of the sensor information at the acquisition time of the captured image is calculated from the sensor information having the acquisition time close to the acquisition time of the captured image. The calculation method of the estimated value of the sensor information is not particularly limited, and for example, it can be obtained by a proportional distribution method estimated from sensor information having acquisition times immediately before and after the captured image acquisition time. As shown in FIG. 3, it is assumed that the sensor information value V1 is acquired at the sensor acquisition time Tm1, and the sensor information value V2 is acquired at the sensor acquisition time Tm2. Further, it is assumed that the captured image acquisition time in the selected frame is Tc1. At this time, the sensor information value Vx to be combined with Tc1 is estimated by calculating, for example, a linear relationship from previous and subsequent data.
That is,
Vx = (V2-V1). (Tc1-Tm1) / (Tm2-Tm1) + V1 (1)
In addition, the estimated value can be calculated by a known mathematical method, and can be obtained by polynomial approximation in addition to the above-described proportional distribution method.

  Further, the image processing system according to the present invention may be configured to be output to a printing apparatus 180 such as a printer that prints the still image. By configuring in this way, the result analyzed by this system can be used as a report.

  Here, FIG. 4 shows a real image captured by the camera 110 and a captured image according to the present invention. In this way, an image reflecting the measurement result obtained by the sensor is shown in the vicinity of the sensor existing on the real image.

  Next, a description will be given of a flowchart of measurement processing for the image processing system according to the present invention to display a virtual image in which a measurement result is reflected on a sensor portion on a photographed image (FIG. 5). An initialization process is performed before entering the measurement process. As initialization processing, initialization of GULT, video device setting, window setting, camera parameter setting, pattern file setting, and the like are performed. The initialization process is well known as described in publicly-known literature (for example, Toyotoshi Tanijiri, ARTToolkit programming technique for realizing augmented reality, 2008).

  First, in step S310, the image input unit 142 receives an image signal of a captured image transmitted from the camera 110, generates a captured image based on the image signal, and transmits the captured image together with an acquisition start time range of each frame. A well-known technique can be used as a method for generating an image based on an image signal.

  Next, in step S320, the marker detection unit 144 detects a marker as a projected image of geometric features from the captured image generated in step S310. As a method for detecting the marker, a known method can be used. For example, the marker can be recognized by converting the image obtained from the camera into a binary image.

  Next, in step S330, the arrangement information generation unit 145 performs a comparison process of the reliability of the marker M detected in step S320. Among the areas detected as markers, areas that are not markers are also included. Therefore, the detected marker is compared with the marker form information, and a reliability comparison process is performed. This prevents detection (false detection) of a region that is not a marker. When using a plurality of markers, reliability comparison processing is performed for each marker.

  Next, in step S340, the arrangement information generation unit 145 performs calculation processing of the position / orientation of the marker M. From the steps S320, S330, and S340, information on the position and orientation of each marker in the captured image can be acquired.

  Next, in step 350, the arrangement information generation unit 145 applies a coordinate transformation matrix. By this processing, the image coordinates of the marker are converted and converted into coordinates for drawing a virtual image.

  In step S360, an object (virtual image) is selected and drawn. The processing in this step will be described in detail later.

  In step S <b> 370, the operation input unit 147 determines whether an instruction to end the process is input from the input device 170. As a result of such determination, if no termination instruction is given, the process returns to step S310 at the beginning of the measurement process and the process is repeated. On the other hand, if an end instruction is given, this process ends.

  Next, the process of step S360 will be described in detail. FIG. 6 is a flowchart showing details of the processing performed in step S360.

  First, in step S361, the arrangement information generation unit 145 calculates the position of the sensor, or the position and orientation based on the arrangement information of the marker M. When the information obtained from the sensor is a scalar quantity, the calculation of the position and orientation may be performed even if only the position of the sensor is calculated. However, when the information is a vector quantity, the calculation of the position and orientation is performed. Is preferably performed. Here, for example, one point of the marker M is selected in advance as a reference point, and the position of the sensor or the calculation of the position and posture is calculated based on the position or position and posture information from the reference point input in advance. To do. The sensor position information and orientation information are input in advance from the input device 170 and are stored in the arrangement information storage unit 141 via the operation input unit 147 and the arrangement information editing unit 148.

  In step S362, the sensor information processing unit 152 performs calculation processing of sensor measurement results based on the data input from the sensor information input unit 151. Here, when a three-axis strain gauge with three elements stacked is used as the sensor, the calculation method of rosette analysis is used from the sensor information and the information of the type of gauge, the pasting direction, and the connection sequence that have been input in advance. Thus, vector values such as principal strain and principal stress can be obtained.

  In step S363, the sensor information image generation unit 154 selects an object that reflects the measurement result of the sensor. The object is stored in the image information storage unit 155, and an object that reflects the measurement result of the sensor is selected from these stored objects.

  In step S364, an image of the virtual space in which the object selected in step S362 is arranged is drawn based on the information to which the coordinate transformation matrix is applied in step S350. At this time, the real space image and the virtual space image are synchronized. At this time, the frame interval of the moving image acquired by the camera 110 and the sensor data acquisition interval often do not match. In this case, for example, when the number of sensor data acquisition is larger than the number of frames of the moving image. It is preferable to superimpose images reflecting the data processing result of the sensor at the intermediate point in the time range of the frame. In addition, when a numerical value is selected as an image to be superimposed, it is preferable to display it in parallel to the xy plane of camera coordinates.

  A flowchart of analysis processing for analyzing the measurement result by the image processing system according to the best mode will be described (FIG. 7). First, in step S301A, it is determined whether or not a time to be displayed has been input. As the numerical value, a time to be displayed by the input device 170 is input. If no numerical value has been input, the process proceeds to step S370A. If a numerical value has been input, the process proceeds to the next step S310A. In step S310A, a captured image frame having the time range input in the above step is selected and read, or a captured image frame having an acquisition time closest to the time range input in the above step is selected and read. . Subsequently, in steps S320A to S350A, processing similar to that of steps 320 to 350 described above is performed.

  In step S360A, a virtual image is generated and drawn based on sensor information corresponding to the numerically input time. As shown in FIG. 8, basically the same processing as step S360 in the measurement processing is performed, but in step S365A, sensor information corresponding to the designated time is read based on the above numerical input. By this step, it is possible to display a virtual image corresponding to the numerically input time.

  In step S380A, it is determined whether there is a printout instruction. If there is an instruction, the displayed still image is output to the printing apparatus 180 in the next step S385A. If there is no printout instruction, the process proceeds to step S390A.

  In step S390A, it is determined whether there is an instruction to end composite image display. If there is an end instruction, the process proceeds to the next step S370A, and if there is no instruction, the process returns to the top of step S380A.

  In step S370A, it is determined whether there is an end instruction. If there is an end instruction, the process ends. If there is no end instruction, the process returns to the beginning of the analysis process.

  As described above, when the virtual image corresponding to the sensor information and the captured image of each frame are combined and displayed, the two images are displayed so that the acquisition time of the sensor information is included in the acquisition time range of the frame. Still images are displayed so as to be superimposed.

  As a modification of the above analysis method, when the image frame acquisition frequency is larger than the sensor information acquisition frequency, it is determined whether or not an image frame is selected in step S301A, and the image frame acquisition frequency is selected in step S310A. Load the frame image. In step S365A, sensor information having a time range including the acquisition time of the selected image frame is read.

  In addition, in step S301A, it is determined whether the sensor information acquisition time or the image frame is selected. If the sensor information acquisition time is selected, the acquisition closest to the sensor information acquisition time is performed in step S310A. A captured image of a frame having a time is selected and read. On the other hand, if an image frame is selected, the captured image is read in step S310A, and sensor information having an acquisition time closest to the frame acquisition time is read in step S365A.

  As another analysis method, an analysis method shown in FIG. 9 is used. Since there are many common parts between the above analysis method and the processing flow, step numbers corresponding to the above flow are attached and description thereof is omitted, and here, differences from the above processing method will be mainly described.

  In this analysis method, a still image is displayed for each frame of a captured image, an estimated value of sensor information at the acquisition time of the captured image displayed on the still image is calculated, and a virtual image reflecting the estimated value is generated and synthesized. . That is, as a difference from the above processing method, in step S301A-2, it is determined whether or not an image is designated. Here, the image is an image corresponding to the frame of the captured image. The image is selected by the input device 170. Subsequently, in step 310A-2, the image frame of the selected image is read.

  In step 360A-2, a virtual image reflecting the estimated value of the sensor information at the acquisition time of the selected image frame is generated and drawn. Here, the estimated value is calculated by a method as shown in FIG. In step S367A-2, a plurality of pieces of sensor information having an acquisition time close to the image acquisition time are read out. For example, when the estimated value is calculated by the proportional distribution method, the sensor information close to the sensor information V1 having the immediately preceding acquisition time Tm1 and the immediately following acquisition time Tm2 with respect to the image acquisition time Tc1. The sensor information V2 is read.

  Subsequently, in step S368A-2, an estimated value is calculated from the value read in the previous step. For example, an inclination is obtained from the above value, a difference between values generated by a time difference from V1 is calculated, and an estimated value is calculated in addition to V1. Here, when the value obtained from the sensor information has a plurality of parameters such as a vector quantity, an estimated value is calculated for each parameter. Thereafter, in step S369A-2, a measurement result is calculated from the estimated value.

  Through the processing as described above, an image corresponding to a desired frame can be selected from the captured image, and the analysis result of the sensor information at that time can be displayed.

Second Embodiment An image to be combined by the system according to the first embodiment is a display in which an image reflecting sensor information is combined with a real image and displayed. In addition to the above, the finite element method (FEM) analysis result, the boundary element method (BEM) analysis result, the frame analysis result, the difference method analysis result, and the analysis information image such as the temperature distribution of the structure can be displayed in a superimposed manner. It is possible to do.

Since the system 240 according to the second embodiment has substantially the same configuration as that of the first embodiment, the description of the common configuration is omitted or the same symbol is given and the description is omitted.
The difference between the first embodiment and the second embodiment is that the image output means, such as an FEM analysis result, a BEM analysis result, a framework analysis result, a difference method analysis result, a temperature distribution, etc. based on the marker arrangement information. It is a point which has a means to synthesize | combine the virtual image which reflected the analysis result. As analysis results, in more detail, structural analysis (for example, vibration analysis, eigenvalue analysis, earthquake analysis, etc.), temperature analysis, fluid analysis, using analysis methods such as FEM analysis, BEM analysis, frame analysis, and difference method analysis Stress distribution, displacement distribution, velocity distribution, and temperature distribution obtained by performing the above.
Hereinafter, a case where FEM analysis results are displayed in an overlapping manner will be described.

  Here, the FEM analysis can use a well-known FEM analysis method for calculating a structure of a civil structure such as a bridge, and may be a nonlinear analysis or a linear analysis. Images reflecting these analysis results are stored in the arrangement information storage unit 141 in advance, and the images are drawn together when drawing the composite image.

  More specifically, in step S364 in the first embodiment, the imaging means position estimation unit 146 reflects not only the object selected in step S362 but also the FEM analysis result stored in the arrangement information storage unit 141. Draw an image on a real image.

  As a modification of the second embodiment, there is a mode in which a plurality of markers are arranged to calculate more accurate spatial coordinate information of the real space, and a virtual image reflecting the FEM analysis result is synthesized based on the spatial coordinates. Can be mentioned.

  In this case, it is possible to perform more accurate superposition than superimposing FEM analysis results based on the arrangement information of one marker. For example, if you try to acquire the space coordinates of the real space with a single marker, a rotation error occurs due to a slight misalignment of the marker, a scale reading error occurs, and the perspective (depth scale) error is It has a problem that it occurs. For these problems, it is possible to correct rotation errors and scale reading errors by attaching two markers on the same plane. In addition, it is possible to correct perspective errors by attaching markers to two surfaces.

Third Embodiment The image processing system according to the third embodiment is basically the same as the configuration of the first embodiment, but differs in that a still image is used as a real image to be displayed. The system according to this embodiment is used, for example, when there is little change in sensor information over time, such as when using a temperature sensor as a sensor, or when the object from which sensor information is acquired has no movement such as a building It is preferable to do. Hereinafter, differences from the first embodiment will be described in detail. In addition, description is abbreviate | omitted about the structure which is common in 1st embodiment.

  The system in the third embodiment differs in measurement processing. FIG. 11 shows a flowchart of the measurement process according to this embodiment. The major difference is that after the end of S370, if there is no end instruction, the process returns to S360 and only object selection / drawing is repeated. First, the position and orientation information of the marker of the obtained image is calculated, and an image reflecting the measurement result of the sensor is displayed.

Fourth Embodiment The image processing system according to the fourth embodiment is basically the same as the configuration of the first embodiment, but differs in that a plurality of cameras are used. By acquiring images of parts that cannot be projected with a single camera and displaying them side by side with augmented reality images, for example, it is possible to see how much stress is applied to the pier when a car passes over the bridge. Therefore, it becomes possible to display easily. FIG. 12 is a block diagram illustrating a functional configuration of an image processing system according to the fourth embodiment. Constituent elements common to the first embodiment are denoted by the same reference numerals and description thereof is omitted.

  The image processing system according to the present embodiment is different in that two cameras (cameras 110a and 110b) are used. The cameras 110a and 110b capture a moving image or a still image in real space. Each captured frame image is input to the image processing system 140 as an image signal together with time range information. The image captured by the camera 110a is sent to the image input unit 142, and the image input unit sends the image and the image acquisition time range to the image storage unit 143 and the marker detection unit 144. On the other hand, the image captured by the camera 110b is sent to the second image input unit 156 as an image signal, and the image and the image acquisition time or acquisition time range are sent to the second image storage unit 157.

  An image input by the camera 110a may be processed in the same manner as in the first embodiment, or may be processed in the second and third embodiments. In addition, the image input from the camera 110b is displayed side by side in another window or the like by comparing the image obtained by synthesizing the real image and the virtual image by the processing and the image acquisition time range.

  In the analysis process, the analysis unit 190 instructs to display a still image obtained by combining the image captured by the camera 110a and the virtual image reflecting the sensor information, as in the first embodiment. At this time, the analysis unit 190 instructs the second image storage unit 157 in which an image from the camera 110b is stored to output a captured image corresponding to a frame having a predetermined acquisition time or acquisition time range. To do.

  When the acquisition time or acquisition time range of sensor information is specified and a still image is displayed, the same processing as in the first embodiment is performed from the captured images from the camera 110a and the camera 110b, and an appropriate A frame having an acquisition time or an acquisition time range can be selected and a still image can be displayed.

  On the other hand, when a captured image of the frame of the camera 110a or the camera 110b is designated and displayed as a still image, the acquisition time range or acquisition time of the captured image and the acquisition of sensor information are performed by the method according to the first embodiment. A virtual image can be displayed by selecting sensor information or calculating an estimated value in comparison with a time range or acquisition time. The synchronization method of the image frame of the camera 110a and the image frame of the camera 110b is, for example, acquiring the acquisition time range in the image frame of one camera and acquiring the acquisition time in the image frame of the other camera, There is a method of displaying in synchronization with a captured image of a frame having an acquisition time range in which the acquisition time of the camera image is included in the range. Alternatively, the acquisition times of the frames of both cameras may be stored, and the captured image of the frame having the closest time to the specified frame image may be selected and displayed.

  FIG. 13 is a diagram illustrating an example of an image generated by the image processing system according to the present embodiment. FIG. 13A is a diagram illustrating a schematic structure of a model of a structure used as an example. Here, when a tire B of a vehicle having a speed of 40 km / h passes through an upper portion of a structure having a horizontal rib 1 and a vertical reinforcing steel material 2 and the distance between the horizontal rib and the vertical reinforcing steel material is 1.25 m. The change with time of the main stress at the point of interest P is measured. FIG. 13B is a conceptual diagram of a still image displayed at the time of analysis of the measurement. A state in which the tire of the vehicle passes over the structure and the main stress at the point of interest P located therebelow. Changes are displayed as vector quantities as indicated by arrows. By configuring in this way, it is possible not only to visually capture temporal changes but also to be used as objective data analysis data, and therefore it is possible to take reinforcement measures based on the analysis results.

  As described above, the image processing system according to the best mode has been described in detail. However, the marker detection unit 144, the arrangement information generation unit 145, the photographing unit position / orientation estimation unit 145, and the arrangement information editing unit 148 that constitute the image processing system 140 are described. The synthesizing unit 149, the sensor information processing unit 152, the sensor information image generation unit 154, and the analysis unit 190 can be implemented as a computer program. Further, the image storage unit 143, the arrangement information storage unit 141, the sensor information storage unit 153, and the image information storage unit 155 can be implemented by a memory device such as a RAM or a hard disk. Further, the image input unit 142, the operation input unit 147, and the sensor information input unit 151 can be implemented by an interface. That is, when the computer program is installed in a computer having the memory device and the interface, and the computer executes the computer program, the computer performs the same operation as the image processing system 140. Here, a well-known apparatus can be used about computer hardware.

M: Marker R: Real space 110: Camera 120: Sensor 130: Data logger 140: Image processing system 160: Display device 170: Input device 190: Analysis unit

Claims (11)

An image processing system for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time range acquisition means for acquiring an acquisition time range of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image can be displayed at each acquisition time, and the captured image corresponding to a frame having a time range including the acquisition time of sensor information corresponding to the virtual image displayed on the still image is selected. Image output means for superimposing and synthesizing the virtual image at a location corresponding to the position of the sensor in the captured image and outputting as a still image;
An image processing system. An image processing system for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time range acquisition means for acquiring an acquisition time range for acquiring sensor information together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image for each acquisition time range, and the captured image corresponding to a frame having a time included in the acquisition time range of sensor information corresponding to the virtual image displayed as the still image is selected. And an image output means for superimposing and synthesizing the virtual image at a position corresponding to the position of the sensor of the photographed image and outputting as a still image;
An image processing system. An image processing system for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Sensor information estimation for calculating an estimated value of sensor information at the captured image acquisition time of the frame from a plurality of sensor information at close times for at least one selected from the acquisition time of the frame of the captured image Means,
Virtual image forming means for forming a virtual image related to the estimated value of the sensor information;
A still image can be displayed for each frame of the captured image, and a virtual image related to the estimated value of the sensor information corresponding to the acquisition time of the captured image of the frame corresponds to the position of the sensor of the captured image Image output means for superimposing and synthesizing and outputting as a still image;
An image processing system. An image processing system for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition means for acquiring a captured image of the real space;
Captured image acquisition time acquisition means for acquiring the acquisition time of each frame of the captured image;
Sensor information acquisition means for acquiring sensor information sent from sensors arranged in real space;
Sensor information time acquisition means for acquiring the time when the sensor information is acquired together with the sensor information;
Sensor position information acquisition means for acquiring the position of the sensor;
Virtual image forming means for forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image at each acquisition time, and the captured image corresponding to a frame having an acquisition time closest to the acquisition time of sensor information corresponding to the virtual image displayed as the still image is displayed. Selecting and outputting the virtual image as a still image by superimposing and synthesizing the virtual image at a location corresponding to the position of the sensor in the captured image;
An image processing system.   The image processing system according to claim 1, further comprising: a print output unit that prints the still image using a printing apparatus. Second captured image acquisition means for acquiring a second captured image in a real space different from the captured image;
Second captured image acquisition time or time range acquisition means for storing the acquisition time or acquisition time range of the second captured image;
6. The image according to claim 1, wherein the image output unit displays two images of the second captured image and the captured image in synchronization with the captured image or virtual image. Processing system. An image processing method for analyzing temporal changes in sensor information arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time range acquisition step of acquiring an acquisition time range of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time acquisition step for acquiring a time at which the sensor information is acquired together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A virtual image forming step of forming a virtual image related to the sensor information;
The virtual image can be displayed at each acquisition time, and the captured image corresponding to a frame having a time range including the acquisition time of sensor information corresponding to the virtual image displayed on the still image is selected. An image output step of superimposing and synthesizing the virtual image on a location corresponding to the position of the sensor of the captured image and outputting as a still image;
An image processing method. An image processing method for analyzing temporal changes in sensor information arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time acquisition step of acquiring an acquisition time of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time range acquisition step for acquiring an acquisition time range for acquiring sensor information together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A virtual image forming step of forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image for each acquisition time range, and the captured image corresponding to a frame having a time included in the acquisition time range of sensor information corresponding to the virtual image displayed as the still image is selected. And an image output step of superposing and synthesizing the virtual image on a location corresponding to the position of the sensor of the captured image and outputting as a still image;
An image processing method. An image processing method for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time acquisition step of acquiring an acquisition time of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time acquisition step for acquiring a time at which the sensor information is acquired together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A sensor information estimation step of calculating an estimated value of sensor information at the captured image acquisition time of the frame from a plurality of sensor information at close times for at least one selected from the acquisition time of the frame of the captured image When,
A virtual image forming step of forming a virtual image related to the estimated value of the sensor information;
A still image can be displayed for each frame of the captured image, and a virtual image related to the estimated value of the sensor information corresponding to the acquisition time of the captured image of the frame corresponds to the position of the sensor of the captured image An image output process for superimposing and combining and outputting as a still image;
An image processing method. An image processing method for analyzing temporal changes in information of sensors arranged in a real space,
A captured image acquisition step of acquiring a captured image of the real space;
A captured image acquisition time acquisition step of acquiring an acquisition time of each frame of the captured image;
A sensor information acquisition step of acquiring sensor information sent from sensors arranged in the real space;
A sensor information time acquisition step for acquiring a time at which the sensor information is acquired together with the sensor information;
A sensor position information acquisition step for acquiring the position of the sensor;
A virtual image forming step of forming a virtual image related to the sensor information;
The virtual image can be displayed as a still image at each acquisition time, and the captured image corresponding to a frame having an acquisition time closest to the acquisition time of sensor information corresponding to the virtual image displayed as the still image is displayed. An image output step of selecting and superimposing the virtual image on a position corresponding to the position of the sensor of the photographed image, and outputting as a still image;
An image processing method.   A program for causing a computer to execute the image processing method according to any one of claims 7 to 10.