JP2005260724A - Recording apparatus and method, and reproducing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve a problem wherein recording and reproducing are not performed to play back working in a compound reality space. <P>SOLUTION: An image of a virtual object is combined with a real space image that is photographed in a real space, to produce a compound real space image. Furthermore, position and posture information of a part of the body of a worker who observes the compound real space image, is detected and the compound real space image, the position and posture information of the worker and time information are correspondingly accumulated in a recording apparatus. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a recording apparatus and method, and a reproducing apparatus and method, and more particularly, to an apparatus and method for recording and reproducing an operation of an operator in a mixed reality space.

  In recent years, technological development relating to mixed reality (MR) aimed at seamless fusion of the real world and the virtual world has become active. MR is a technology that enhances VR for the purpose of coexistence between the virtual reality (VR) world and the real world (real space), which could be experienced only in situations where it has been separated from the real space. It is attracting attention as.

  A typical device for presenting mixed reality to an observer is a head-mounted device (HMD). There are two types of HMDs: video see-through type and optical see-through type HMDs. The former uses an opaque (normal) display device, and displays a real image (real space image) acquired by a video camera and a mixed real space image synthesized with a virtual space image generated by normal computer graphics (CG). By displaying on the screen, the viewer is given a mixed reality. On the other hand, the latter uses a semi-transparent display device, and the observer directly observes the real space through the display device, and displays only the virtual space image on the display device to give the observer a mixed reality. For example, Patent Document 1 discloses an example in which a mixed reality presentation method using an optical see-through type HMD is applied to an air hockey game.

Japanese Patent Laid-Open No. 11-84307 JP 2001-300131 A D. Reiners, D. Stricker, G. Klinker, and S. Muller, "Augmented Reality for Construction Tasks: Doorlock Assembly", In Proc. Of IWAR'98, A K Peters, Ltd., pp31-46, (1999) G. Klinker, O. Creighton, AH Dutoit, R. Kobylinski, C. Vilsmeier, and B. Brugge, "Augmented maintenamce of powerplants: A prototyping case study of a mobile AR system", In. Proc. Of ISAR'01, pp124-132 (2001)

  One area where MR technology is useful is work support. This includes, for example, work assistance for displaying a product assembly procedure as a virtual space image on a real product (part) in a factory. As an example of studying application to such a field, for example, Non-Patent Document 1 discloses an application example for training an automobile assembly worker using 3D (three-dimensional) CAD data. Then, an application example for maintenance of a nuclear power plant is reported.

  However, the examples described in these documents only support the work in real time, and accumulate data related to the work such as the work actually performed and the user's action at the time of the work. For example, there is no system for reproducing the accumulated data.

  On the other hand, Patent Document 1 discloses a game system that provides a replay image that can be grasped from an objective and global viewpoint after playing the game space. The replay image is intended to reproduce the progress of the game. Therefore, no consideration is given to the accumulation of information related to the body (motion) of the worker (player) who actually played the game and the use during replay.

  The present invention has been made in view of such problems of the prior art, and its main purpose is an image (mixed reality space image) that the operator is observing in the mixed reality space and a position related to the worker. An object of the present invention is to provide a recording apparatus and method for recording a work record including posture information and time information.

  Another main object of the present invention is to provide a playback apparatus and method for displaying various recorded information in association with each other.

  The above-mentioned purpose is to combine an image of a virtual object with a real space image obtained by photographing a real space to generate a mixed reality space image, and a body of an operator who observes the mixed reality space image. A recording method comprising: a position / orientation detection step for detecting a part of position / orientation information, a mixed reality space image, a position / orientation information, and a storage step for storing the time information in association with each other. Achieved by:

  In addition, the above-described object is to provide a mixed reality space image obtained by synthesizing a virtual object image with a real space image obtained by photographing a real space, and a part of the body of an operator who observes the mixed reality space image. A reproduction method for reproducing and displaying a mixed reality space image, position and orientation information, and time information from a recording device in which position / orientation information and time information are recorded in association with each other, the mixed reality recorded in the recording device It is also achieved by a reproduction method comprising a reproduction step of reproducing and displaying a spatial image and position and orientation information based on time information.

  In addition, the above-described object is to provide an image composition unit that generates a mixed reality space image by synthesizing a virtual object image with a real space image obtained by photographing the real space, and an operator who observes the mixed reality space image. Position and orientation detection means for detecting position and orientation information of a part of the body, storage means for storing the mixed reality space image, the position and orientation information of the body part of the worker, and time information in association with each other in the recording device It is also achieved by a recording apparatus characterized by comprising:

  In addition, the above-described object is to provide a mixed reality space image obtained by synthesizing a virtual object image with a real space image obtained by photographing a real space, and a part of the body of an operator who observes the mixed reality space image. Recording means in which the position and orientation information and time information are recorded in association with each other, and a reproducing means for reproducing the mixed reality space image and the position and orientation information recorded in the storage means based on the time information. This is also achieved by a reproducing apparatus characterized by the above.

  The above-described object is also achieved by a program that causes a computer to execute at least one of the recording method and the reproducing method of the present invention, or a computer-readable recording medium that stores the program.

  In this way, the work performed by the worker in the mixed reality space can be reviewed by associating and accumulating the mixed reality space image observed by the worker, the position and orientation of the worker's body part, and time information. , It can be used for other purposes such as other studies.

  Further, by reproducing and displaying these data in association with each other, it is possible to provide convenience for worker's work analysis and work education.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings. Here, a mixed reality video storage / reproduction system, which is an example of an embodiment of a recording apparatus and an example of an embodiment of a playback apparatus, will be described, but this requires both a recording function and a playback function in the present invention. It does not mean that.

● (first embodiment)
FIG. 1 is a block diagram showing a functional configuration example of a mixed reality video storage / playback system according to the present embodiment. The mixed reality video storage / playback system according to this embodiment includes a head-mounted display device (HMD) 20 and a mixed reality video storage / playback device 10.

  The HMD 20 is a so-called video see-through head-mounted display device having a camera 22, a three-dimensional position / orientation sensor 21, and a display 23. Usually, an operator wearing the HMD 20 is in a real space within a predetermined range. It is possible to experience mixed reality. The HMD 20 includes, for example, a three-dimensional position / orientation sensor 21 that measures position and orientation using magnetism, a camera 22 that captures a real space observed by the wearer of the HMD 20 and acquires a real space image, and a mixed reality image. A display 23 that is a color liquid crystal display, for example, is provided for presenting an image received from the storage / reproduction device 10 to the eyes of the wearer of the HMD 20. It is assumed that the relative positional relationship between the three-dimensional position and orientation sensor 21 and the camera 22 is measured in advance and is known and unchanged. A position / orientation signal, which is a measurement result, is output from the three-dimensional position / orientation sensor 21 to a position / orientation measurement unit 11 described later, and an imaging result is output from the camera 22 to an image input unit 12 described later. The display device 23 is a device that receives an image signal as an input and displays an image. The display device 23 receives the image signal from the image composition unit 15 and displays the contents thereof.

  In the mixed reality video storage / playback apparatus 10, the position / orientation measurement unit 11 receives the three-dimensional position / orientation signal output from the three-dimensional position / orientation sensor 21 of the HMD 20 and determines the position / orientation of the camera 22 from the known relative positional relationship. To generate camera viewpoint position and orientation data (position and orientation information). The image generation unit 13 is photographed by the camera 22 using the position / orientation data obtained by the position / orientation measurement unit 11 and the position information and model information (information such as shape and texture) of the virtual object stored in advance. A virtual space image having a positional relationship with the real space image is generated by a computer graphics technique. On the other hand, the image input unit 12 performs predetermined processing such as digitization processing on the signal of the captured video received from the camera 22 and supplies the signal to the image composition unit 15 as a real space image.

  The image composition unit 15 synthesizes the virtual space image generated by the image generation unit 13 on the real space image from the camera 22 obtained through the image input unit 12 to generate a mixed reality space image. Then, the mixed reality space image is output to the display 23 of the HMD 20. The wearer of the HMD 20 experiences mixed reality by observing a mixed reality space image in which a virtual space image based on the measured camera position and orientation is superimposed on the real-time image captured by the camera 22. To do.

  The control unit 16 is, for example, a CPU, and controls each element in the mixed reality video storage / playback device 10 by developing and executing a program recorded in a storage device (not shown) in a RAM (not shown), which will be described later. Implement various processes. The timer 17 is, for example, a clock that operates using a clock used by the control unit 16.

  Here, for the sake of simplicity, it has been described that one system of processing is performed. However, in reality, a parallax image is displayed on the HMD 20 and a three-dimensional mixed reality space is experienced. The processes leading to the generation of the virtual space image and the generation of the mixed reality space image are performed for two systems for the right eye and the left eye.

  Specifically, two cameras 22 are provided on the left and right sides of the HMD, and the position / orientation measurement unit 11 obtains position / orientation data of each camera from a known positional relationship between the three-dimensional position sensor and each camera. The image generation unit 13 generates right-eye and left-eye virtual space images based on the individual position and orientation data. In addition, the image composition unit 15 synthesizes the left-eye and right-eye virtual space images with the two real space images obtained from the left and right cameras 22 via the image input unit 12, and combines the left-eye and right-eye composite images. A real space image is generated and output to the left-eye display device and the right-eye display device of the display 23.

  The mixed reality space image generated by the image synthesizing unit 15 is also output to the storage unit 14, and the position / orientation information of the camera 22 obtained by the position / orientation measurement unit 11 corresponding to the mixed reality space image and the time of the timer 17. Is stored together with the time information obtained by using. The data stored in the storage unit 14 is synchronized with the control unit 16 in accordance with an input from an input unit 33 having at least one input device such as a mouse, a keyboard, or a jog dial, for example, CRT, liquid crystal display, etc. Are reproduced and displayed on the display device 31 in time series. Here, the position / orientation information and the mixed reality space image stored in the storage unit 14 are both the left and right two-line information obtained in the above-described parallax image generation process, and may be only one of them.

  Note that such a mixed reality video storage / playback apparatus 10 can be realized by executing an application program for realizing a function described later, for example, in a general-purpose computer having a video input / output function. In the present embodiment, it is assumed that an application program operates on an operating system (OS) using a graphical user interface (GUI), and the user of this system displays a GUI displayed on the screen of the display device 31 by the input unit 33. Various instructions are input to the apparatus by operating. Such GUI display can be realized, for example, when an application program uses an API (Application Programming Interface) of the OS.

● (Overall processing description)
Next, an overview of the overall processing of the system of this embodiment will be described with reference to FIG. In the present embodiment, as shown in FIG. 2 (a), the process in which the HMD wearer (worker) 40 performs work on the virtual objects 41, 42, and 43 placed in the real space is recorded and accumulated. Shall. The world coordinate system (x, y, z) which is an orthogonal coordinate system as shown in the figure is set in the real space, and the virtual objects 41, 42 and 43 placed in this coordinate system are inconsistent with the real space. It is drawn on the real space image so as to be observed with no positional relationship, and is displayed on the HMD.

  Any specific work performed by the worker 40 using the virtual objects 41 to 43 may be performed. For example, the virtual object is selected and moved using a stylus equipped with a magnetic sensor. It may be work.

  During the work in such a mixed reality space, the position and orientation information of the HMD of the worker 40 (the value converted from the output of the three-dimensional position sensor to the camera position and orientation as described above) and the HMD are displayed. The mixed reality space image and the time information are recorded and stored in the storage unit 14 in association with each other. These accumulated data can be reproduced and displayed in time series on the display device 31 by operating the input unit 33. Hereinafter, the position and orientation information of the HMD is treated as the same as the position and orientation of the camera, that is, the viewpoint information of the operator unless otherwise specified.

  In the present embodiment, the data recorded in the storage unit 14 is reproduced on a reproduction screen as shown in FIG. The playback screen includes a 3D trajectory display module 50, a moving image playback module 60, and an elapsed time display module 70. The 3D trajectory display module 50 three-dimensionally displays the mixed reality space (the space in the world coordinate system described above) on which the worker 40 has worked together with the virtual object, and includes the HMD of the worker 40 during the work. The movement locus 55 is also displayed three-dimensionally. The video playback module 60 includes a video display area 61 for video display, a slide bar 62 for controlling video display, a stop button 65 for stopping video playback, a stop button 66 for pausing video playback, and a video playback button 67. The image data recorded in the storage unit 14 is reproduced in the moving image display area 61. The elapsed time display module 70 displays the elapsed time with 0 as the recording start time.

  Each of the modules 50, 60, and 70 can be operated by the input unit 33. For example, the user of the system (which may or may not be the same as the worker) moves the mouse cursor into the area of each module. By moving the mouse and clicking it, it is possible to specify the playback start location, start playback, pause, stop, etc.

  The storage unit 14 is a data storage device represented by, for example, a hard disk drive, and data is recorded and managed in the storage unit 14 so as to be generally performed by a computer device. Therefore, when the user wishes to reproduce the data recorded in the storage unit 14, the controller 16 displays and displays the reproduction screen by designating the desired file from the GUI using a known method and instructing the reproduction. The read data file is read from the storage unit 14 and the reproduction process is started. Of course, it is also possible to present a reproducible data file to the user with the GUI of the application program, and to select and reproduce from there.

  When a playback instruction is given, the control unit 16 performs a playback screen initialization process. In other words, the position and orientation information of the HMD is extracted from the data file and plotted on the corresponding coordinates of the 3D trajectory display module 50 on the playback screen. Further, time information in the data file is acquired, and the time display “00:00:00” is performed on the elapsed time display module 70. Further, the corresponding image data (first frame image) is read and displayed in the moving image display area 61. The slide bar button is set at the left end of the slide bar 62.

  When the user presses the moving image playback button 67 in this state, the moving image is played back in the moving image display area 61 as in a general moving image player. The slide bar button moves in synchronization with the moving image reproduction, and the display of the elapsed time display area 70 is also updated. In addition, the corresponding portion of the locus drawn in the 3D locus display area 50 is displayed with high luminance. When the stop button 65 is pressed during moving image reproduction, the state returns to the initial state, and when the stop button 66 is pressed, the current processing state is stopped.

  Further, in the present embodiment, it is possible to designate and display a specific time point of the accumulated work data from any of the three display area modules 50, 60, and 70. That is, when a desired portion of the three-dimensional trajectory 55 of the 3D trajectory display module 50 is clicked, the corresponding plot is displayed with high brightness, and at the same time, the moving image frame corresponding to that point is displayed in the moving image display area 61 and the slide bar 62 is displayed. Move to the corresponding time position. The elapsed time display area 70 displays the corresponding elapsed time.

  When the slide bar 62 of the moving image display module 60 is moved, a moving image frame corresponding to the moved position is displayed in the moving image display area 61, and a corresponding locus plot in the 3D locus display area 50 is displayed with high brightness. In the elapsed time display area 70, the corresponding time is displayed. When the time data of the elapsed time display area 70 is designated, the display corresponding to the time is similarly performed in the 3D locus display area 50, the moving image display area 61, and the slide bar 62. As a result, any time point can be designated from any of the three display modules, ie, the 3D trajectory display area 50, the moving image reproduction module 60, and the elapsed time display area 70 described above, so that synchronized reproduction is always possible.

● (Recording process)
Next, the work data recording process will be described in more detail with reference to the drawings.
FIG. 3 is a flowchart of an accumulation process for accumulating work data in the mixed reality work space. It is assumed that the name of work data to be recorded is designated in advance by the user of the system before this process is started. In addition, the start of the recording process may be instructed by the user from the input unit, or the operator 40 may instruct using a switch or the like constituting a part of the input unit 31, for example.

  First, in step S100, a file to be accumulated with a name designated by the user is opened. In the next step S105, "0" is set in the variable Sequence #, and the process proceeds to the next step. In step S110, a set of six parameters in the world coordinate system (a coordinate system fixed and set in the real world as shown in FIG. 2) is input to the position / orientation measurement unit 11 from the three-dimensional position sensor 20. Sensor position and orientation data (x, y, z, α, β, γ) are generated. Here, α represents a rotation angle about the x axis, β represents a rotation angle about the y axis, and γ represents a rotation angle about the z axis.

  In the next step S120, the position / orientation data in the position / orientation coordinates of the camera viewpoint (in this embodiment, synonymous with the position / orientation coordinates of the HMD) from the relationship between the sensor position and the camera position measured in advance. Camera position and orientation data). As described above, since the positional relationship between the sensor and the camera is fixed, calibration can be performed in advance to obtain a coordinate conversion formula. Then, the sensor position / orientation data is converted into camera position / orientation data using this coordinate conversion formula. In step S <b> 130, the image from the camera 22 is written into the image composition unit 15 through the image input unit 12. In step S140, the image generation unit 13 uses the position and shape data represented in the world coordinate system of the virtual object prepared in advance, and the image of the virtual object (virtual space from the view of the camera position and orientation obtained in step S120. A mixed reality space image is obtained by drawing an image) and overwriting the image captured from the camera 22 written in the image composition unit 15 by overwriting.

In the next step S150, the mixed reality space image is displayed on the display 23. In step S200, the mixed reality space image output from the image composition unit 15, the camera position and orientation data, and the like are written in a file together with the elapsed time information. In step S300, for example, the input of the end key from the display screen operation input unit 33 is checked. If the input can be confirmed, the process proceeds to step S380. Otherwise, the variable Sequence # is incremented by 1 in step S350, and the process proceeds to step S110. Repeat the process. In step S380, the file written in the above process is closed and the process is terminated.
As described above, the data to be recorded can be arbitrarily set from one or both of the left and right parallax images and one or both of the camera position and orientation data.

  Next, details of the file writing process in step S200 will be described with reference to FIGS. In the file writing process, for example, a management table file as shown in FIG. 5 is created. Here, for convenience, one row of the table is called a record, and various data write areas (each column of the table) included in the record are called fields. In FIG. 5, the Sequence # field has the value of the variable Sequence # used in the above-described storage process, and the elapsed time field has the time of writing in the image storage process having the variable Sequence # value of 0 as 0. Each time is stored. The next Point 1 field stores the position of the HMD (camera viewpoint position and orientation data described above), and the next Point 2 to Point n fields sequentially store position and orientation data from other sensors. Other sensors can be attached to workers other than the operator's head, for example, when it is desired to record the position and orientation of hands and feet, etc. The data output by these other sensors is Point 2 to Recorded in the Point n field. When there is another sensor, the position / orientation measurement unit outputs it together with the HMD position / orientation data. The last field of the record is a pointer field for storing a pointer to the mixed reality space image data file (for example, a file path and a frame number). Since the image data has a large capacity and becomes variable length data when compressed, another file is generated for the image data so that it can be accessed from the management table with a pointer. Then, records having such a configuration are sequentially added for each cycle of the accumulation process, and are recorded as time passes with Sequence # as a main key.

FIG. 4 is a flowchart showing a procedure for such a writing process to the management table file.
First, in step S210, a new record is created and the current value of the variable Sequence # is written in the Sequence # field. In the case of the first record, the value of the timer 17 is stored as the reference time. In step S220, the mixed reality space image data generated by the image composition unit 15 is stored in another file in the storage unit 14, and a pointer indicating the head is written in the pointer field. Next, in step S240, the current time is acquired from the timer 17, the difference from the accumulation start time is taken, converted into elapsed time, and written in the corresponding column of the management table. In step S250, the values of the camera viewpoint position and orientation data of the six variables obtained in step S120 are written in the Point 1 field of the management table (if other sensors are used, the position and orientation data from these sensors are also combined. Write), and the process ends.

  By executing the above processing, the mixed reality space image displayed on the HMD of the worker 40, the position and orientation information of the HMD, and the time information are associated with each other and stored in a file.

● (Playback processing)
Next, details of the processing on the playback screen of FIG. 2B will be described using the flowchart of FIG. Prior to the playback of the mixed reality space image, it is assumed that the user designates the stored file name by operating the GUI with the input unit as described above, for example.

  In first step S400, the control unit 16 reads the designated file (management table) into a memory (not shown). In the next step S410, the moving image reproducing module is initialized. Specifically, referring to the pointer field of the record of Sequence # = 0 in the management table, the corresponding image is read and displayed in the moving image display area 61. Further, referring to the elapsed time field of the last record of the management table, the moving speed per unit time of the slide bar 62 is calculated in units of display pixel dots of the display device 31 from the total reproduction time of the stored images. For example, when the unit time of display is 1 second, the moving speed per unit time (1 second) of the slide bar is obtained by “the number of pixels of the length of the slide bar (dots) / total time (seconds)”.

  In the initialization process of the 3D trajectory display module in step S420, a virtual object whose position is defined based on the world coordinate system set in the real space is displayed in 3D together with the coordinate space on the screen according to the model. The display viewpoint at this time is set to a scale that allows the entire work space of FIG. 2A to be looked over, and is displayed in a bird's eye view format as viewed from diagonally above the coordinate space. Then, the position and orientation data in the Point 1 field of the management table are read from Sequence # 0 to the last Sequence # and plotted in the 3D display area. When playing back a moving image, the coordinates corresponding to Sequence # are displayed visually different from other plots, such as high-intensity display or display in a different color.

In the next step S430, the elapsed time display module is initialized. That is, the elapsed time is set to 0, and the time display on the display screen is set to “00:00:00”.
When such screen initialization processing is performed, the control unit 16 waits for input of an event (operation performed by a user or another program) in step S450. If an end event (for example, an end instruction from the program menu) is input here, the playback screen processing ends. In the case of other events, the process branches to a process for each area (module) in which the event has occurred. In other words, in the case of an event in the video playback module 60, the event processing in the video playback module in step S500, and in the case of an event in the 3D trajectory display area 50, the event processing in the 3D trajectory display area in step S800. In the case of an event in the display area 70, the process branches to the event processing in the elapsed time display area in step S900. When these event processes are completed, the process returns to the event waiting in step S450.

  Next, details of the event processing of the moving image playback module in step S500 will be described with reference to the drawings. The event in the video playback module 60 is one of the operation of the video playback button 67, the video playback stop button 66, the video playback stop button 65, or the movement of the slide bar 62. First, the step S510 moving image playback button process activated by operating the moving image playback button 67 will be described with reference to FIG.

  In step S515, the current elapsed time is compared with the image file accumulation time (= total playback time). If the former is larger, the process proceeds to the cancel button process in step S580. Otherwise, the process proceeds to step S520. In step S520, a moving image frame (mixed reality space image) corresponding to the elapsed time is displayed. Details of this processing will be described later with reference to FIG. In the next step S530, a slide bar display process corresponding to the elapsed time is performed. Details of this processing will be described later with reference to FIG. In the next step S540, 3D trajectory display processing is started with the current elapsed time as an argument. Details of this processing will be described later with reference to FIG. In the next step S550, the elapsed time display process is started with the current elapsed time as an argument. Details of this processing will be described later with reference to FIG. In the next step S560, it is determined whether or not there is an operation input to the moving image reproduction stop button 66 and the moving image reproduction stop button 65. When there is an operation input of the moving image reproduction stop button 66, the moving image reproduction button process is ended. If there is an operation input of the movie playback stop button 65, the process proceeds to the stop button process in step S580. If there is no operation input to these buttons, the process proceeds to step S570, and the unit time of display is added to the elapsed time to obtain a new elapsed time. That is, “elapsed time = elapsed time + display unit time” is calculated, and the process returns to step S515.

  Next, the moving image reproduction stop button process performed in step S580 will be described with reference to the flowchart of FIG. In step S710, the elapsed time = 0 is set, and the process proceeds to step S720. In step S720, a moving image frame display process described later with reference to FIG. 13 is performed. In the next step S730, slide bar display processing described later with reference to FIG. 15 is performed. In step S740, 3D trajectory display processing described later with reference to FIG. 12 is performed. In the next step S750, an elapsed time display process to be described later with reference to FIG. 14 is performed, and the process ends. Through the above processing, the playback screen returns to the initial display state.

  Next, the moving image slide bar process activated by operating the slide bar 62 will be described with reference to the flowchart shown in FIG. In step S610, when the operation of the moving image slide bar 62 by the user is stopped and the position is fixed, the position is read as the pixel position of the display device 31. From this pixel position, the relative position in the entire slide bar (for example, what percentage has advanced from the left end as the start time) can be known, so in the next step S620, the elapsed time is obtained from the entire time and this relative position. be able to. In step S630, a moving image frame display process described later with reference to FIG. 13 is performed. In step S640, a 3D trajectory display process described later with reference to FIG. 12 is performed. In step S650, an elapsed time display process which will be described later with reference to FIG. 14 is performed, and the process ends.

  Next, the event process of the 3D locus display area performed in step S800 of FIG. 6 will be described. A point on the 3D trajectory plot 55 in the 3D trajectory display area 50 of FIG. 2 is indicated by, for example, a mouse cursor, and the mouse is clicked to start the 3D trajectory click process shown in FIG. In step S810, the value of the three-dimensional coordinate of the clicked point (plot) is obtained. This corresponds to the inverse process of the process of converting the three-dimensional coordinate value of the sensor position / orientation data into the coordinates of the display area in the 3D trajectory display process, and generally used 3D graphics software such as Open Inventor (registered trademark). Such a function can be easily realized. In step S820, the elapsed time is read from the elapsed time field of the record corresponding to the three-dimensional coordinates of the clicked point with reference to the Point 1 field of the management table of FIG. The following steps S830 to S860 are performed using this elapsed time as an argument. In step S830, a moving image frame display process described later with reference to FIG. 13 is performed. In the next step S840, a slide bar display process described later using FIG. 15, a 3D trajectory display process described later using FIG. 12 in step S850, and an elapsed time display process described later using FIG. 14 in step S860. Each is done and the process is terminated.

  Next, the event processing of the elapsed time display area performed in step S900 of FIG. 6 will be described. When the user designates and clicks the value of the elapsed time display area 70 in FIG. 2 with, for example, a mouse cursor, the time input process shown in the flowchart of FIG. 11 is started.

  In step S910, a value input waiting state for the elapsed time display area is awaited, for example, waiting for the user to complete the input of the elapsed time by operating the keyboard. When the completion of input is notified by, for example, a mouse click in an area other than the time display area in the progress display area, the input value is set as the elapsed time. The following steps S920 to S950 are performed using this elapsed time as an argument. In step S920, an elapsed time display process described later using FIG. 14 is performed, in step S930, a moving image frame display process described later using FIG. 13 is performed. In step S940, a slide bar display process described later using FIG. In S950, the 3D trajectory display process described later with reference to FIG. 12 is executed, and the process ends.

● (Various display processes)
Next, various display processes commonly used in the processes described above will be described.
<3D locus display processing>
First, 3D locus display processing will be described with reference to FIG.
In step S1010, an elapsed time value is received, and in the next step S1020, the elapsed time ≥ referring to the elapsed time field value t (i) (i is the value of Sequence #) in FIG. The value i of Sequence # corresponding to the maximum t (i) satisfying t (i) is obtained. Then, the three-dimensional coordinates are acquired from the Point 1 field of the record having the value i of Sequence #, the corresponding plot in the 3D display area 50 is displayed with high luminance, and the other plots are displayed with low luminance, and the processing is terminated. .

<Video frame display processing>
Next, the moving image frame display process will be described with reference to FIG.
In step S1110, the value of the elapsed time is received. In the next step S1120, the value of t (i) in the management table of FIG. 5 is referred to, and the maximum t (i satisfying the elapsed time ≧ t (i) is satisfied. The value i of Sequence # corresponding to) is obtained. Then, referring to the pointer field of the record having the value i of Sequence #, the corresponding moving image frame (mixed reality space image) is read from the storage unit 14, displayed in the moving image display area 61, and the process is terminated.

<Elapsed time display processing>
Next, the elapsed time display process will be described with reference to FIG.
In step S1210, the value of the elapsed time is received, and in the next step S1220, for example, as shown by 70 in FIG. 2, it is displayed and processed in the format of XX (minute): XX (second): XX (1/100 second). Exit.

<Slide bar display processing>
Next, the slide bar display process will be described with reference to FIG.
In step S1310, the value of the elapsed time is received, and in the next step S1320, using the number of slide bar pixels per unit time calculated in step S410 of FIG. 6, “elapsed time × the number of slide bar pixels per unit time”. ”Is calculated, a slide button is displayed at a position moved to the right by the calculated number of pixels from the left end of the slide bar, and the process is terminated.

● (Modification)
Here, for ease of explanation and understanding, the value of the three-dimensional position and orientation sensor provided in the HMD worn by the worker 40 is displayed as a trajectory in the trajectory display area. However, as described above, It is also possible to mount a sensor on a place, for example, an operator's hand, shoulder, arm, etc., and measure and record and display the position and orientation.

  When recording position and orientation information from a plurality of sensors, data is stored in the Point 2 to Point n fields of the management table of FIG. During reproduction, it is possible to read an arbitrary number of data from the Point 1 to Point n fields and display a three-dimensional trajectory in the 3D trajectory region 50 of FIG. In this case, it is also possible to change the color for each sensor and display a plurality of trajectories at the same time, or to provide a selection area on the playback screen and selectively display the position and orientation data of an arbitrary sensor.

  For the position and orientation data of these other sensors, display processing such as the above-described 3D trajectory click processing and elapsed time display processing can be similarly performed as processing of the position and orientation data from the HMD sensor.

  Here, only the case where only the coordinate values (x, y, z) of the position and orientation data obtained from the three-dimensional position and orientation sensor having 6 degrees of freedom are displayed in the 3D display area 50 is described. In a certain 3D locus click process, the posture data is visualized by displaying posture data (α, β, γ) corresponding to the clicked plot as vector data. Also good. Further, since the entire 3D locus display area 50 is modeled in the world coordinate system (x, y, z), observation from a desired viewpoint is possible on the screen by zooming up or rotating. A GUI for moving the viewpoint for drawing the 3D locus display area 50 is added to the playback screen, and the drawing in the 3D locus display area is performed according to the operation of the GUI. Convenience is further improved.

● (Second Embodiment)
In the above-described embodiment, the magnetic sensor is used as the three-dimensional position / orientation sensor. However, a sensor using an optical method or another method may be used. Also, the camera position and orientation were obtained using the output of the three-dimensional position and orientation sensor provided in the HMD and the relative positional relationship between the sensor and the camera measured in advance. It is also possible to obtain the position and orientation of the camera from the marker coordinates in the captured image of the camera and the known marker coordinates. Then, conversely, the position and orientation of the HMD may be obtained from the obtained position and orientation of the camera.

  Further, as disclosed in, for example, Japanese Patent Application Laid-Open No. 2003-279310, position and orientation measurement using a three-dimensional position and orientation sensor and position and orientation measurement using a marker can be used together to improve the accuracy of position and orientation data.

  A block diagram of a second embodiment using this method is shown in FIG. Explaining only the differences from the first embodiment, the position / orientation measurement unit 11 acquires point data in the world coordinate system based on the measurement signal from the three-dimensional position / orientation sensor 21 and uses the coordinate values as camera coordinates. A viewing transformation is performed to obtain a transformation matrix for transformation into coordinate values of points in the system.

  The feature detection unit 18 obtains an image obtained by the camera 22 from the image input unit 12, detects a marker in the real world in the image, and simultaneously detects a marker position on the image plane. The feature point projection calculation unit 24 converts the position of the marker in the real world (known as the position in the world coordinate system) to the position in the camera coordinate system by the viewing conversion obtained by the position and orientation measurement unit 11, and Perspective projection calculation is performed to calculate the position on the image plane.

  The position / orientation correction unit 25 compares the position of the marker image plane obtained by the feature point projection calculation unit 24 with the position of the feature point obtained by the feature point detection unit 18 to further determine the position and orientation of the HMD 20. Correct with high accuracy. With this configuration, the position / orientation correction unit 25 can obtain HMD (camera) position / orientation data corrected with high accuracy. By using the obtained position and orientation data and performing the same processing as in the first embodiment, it is possible to record and reproduce with higher accuracy.

● (Other embodiments)
A function equivalent to the above-described mixed reality video storage / playback apparatus may be realized by a system including a plurality of devices. A software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium or to a system or apparatus having a computer that can execute the program using wired / wireless communication. The present invention includes a case where an equivalent function is achieved by a computer executing the supplied program.

  Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS.

  As a recording medium for supplying the program, for example, a magnetic recording medium such as a flexible disk, a hard disk, a magnetic tape, MO, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R, DVD- There are optical / magneto-optical storage media such as RW, and non-volatile semiconductor memory.

  As a program supply method using wired / wireless communication, a computer program forming the present invention on a server on a computer network, or a computer forming the present invention on a client computer such as a compressed file including an automatic installation function A method of storing a data file (program data file) that can be a program and downloading the program data file to a connected client computer can be used. In this case, the program data file can be divided into a plurality of segment files, and the segment files can be arranged on different servers.

  That is, the present invention includes a server device that allows a plurality of users to download a program data file for realizing the functional processing of the present invention on a computer.

  In addition, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to the user, and key information for decrypting the encryption for the user who satisfies a predetermined condition is obtained via, for example, a homepage It is also possible to realize the program by downloading it from the computer and executing the encrypted program using the key information and installing it on the computer.

  In addition to the functions of the above-described embodiments being realized by the computer executing the read program, the OS running on the computer based on the instruction of the program is a part of the actual processing. Alternatively, the functions of the above-described embodiment can be realized by performing all of them and performing the processing.

  Furthermore, after the program read from the recording medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion board or The CPU of the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments can also be realized by the processing.

It is a block diagram which shows the function structural example of the mixed reality image | video storage / reproducing apparatus and HMD in the 1st Embodiment of this invention. It is a figure which shows the mode of a mixed reality space, and the example of a reproduction | regeneration screen of a mixed reality video storage / reproduction apparatus. It is a flowchart of the accumulation | storage process of the mixed reality image | video accumulation | storage reproduction | regeneration apparatus in embodiment of this invention. It is a flowchart which shows the detail of the process of file write-in process step S200 of FIG. It is a figure which shows the example of the management table used with the mixed reality video storage reproduction | regeneration apparatus in embodiment of this invention. It is a flowchart of the reproduction | regeneration processing of the mixed reality video storage reproduction | regeneration apparatus in embodiment of this invention. It is a flowchart which shows the detail of the moving image reproduction | regeneration button process which is a part of process of reproduction | regeneration processing step S500 of FIG. It is a flowchart which shows the detail of the slide bar process which is a part of process of reproduction | regeneration process step S500 of FIG. It is a flowchart which shows the detail of the stop button process which is a part of process of reproduction | regeneration process step S500 of FIG. It is a flowchart which shows the detail of 3D locus | trajectory click processing which is a part of process of reproduction | regeneration processing step S800 of FIG. It is a flowchart which shows the detail of the time input process which is a part of process of reproduction | regeneration process step S900 of FIG. It is a flowchart which shows the detail of the 3D locus | trajectory display process of the mixed reality video storage reproduction | regeneration apparatus in embodiment of this invention. It is a flowchart which shows the detail of the moving image frame display process of the mixed reality image | video storage / reproducing apparatus in embodiment of this invention. It is a flowchart which shows the detail of the elapsed time display process of the mixed reality video storage reproduction | regeneration apparatus in embodiment of this invention. It is a flowchart which shows the detail of the slide bar display process of the mixed reality image | video storage / reproducing apparatus in embodiment of this invention. It is a block diagram which shows the function structural example of mixed reality image | video storage / reproducing apparatus and HMD in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Mixed reality image | video accumulation | storage reproduction | regeneration apparatus 11 Position and orientation measurement part 12 Image input part 13 Image generation part 14 Storage part 15 Image composition part 16 Control part 17 Timer 18 Feature point detection part 20 HMD (Head Mounted Display)
21 Three-dimensional position and orientation sensor 22 Camera 23 Display 24 Feature point projection calculation unit 25 Position and orientation correction unit 31 Display device 33 Display screen operation input unit 40 Workers 41, 42, 43 Work target virtual object 50 3D locus display area 55 3D Trail 60 Movie playback module 61 Movie display area 62 Slide bar 65 Movie playback stop button 66 Movie playback stop button 67 Movie playback button

Claims (10)

An image synthesis step of generating a mixed reality space image by synthesizing a virtual object image with a real space image obtained by photographing the real space;
A position and orientation detection step of detecting position and orientation information of a part of a worker's body observing the mixed reality space image;
A recording method comprising: a storage step of storing the mixed reality space image, the position and orientation information, and time information in association with each other in a recording device. A mixed reality space image obtained by synthesizing a virtual object image with a real space image obtained by photographing the real space, and position and orientation information of a part of the body of the operator who observed the mixed reality space image; A reproduction method for reproducing and displaying the mixed reality space image, the position and orientation information, and the time information from a recording device that is recorded in association with time information,
A reproduction method comprising: a reproduction step of reproducing and displaying the mixed reality space image and the position / orientation information recorded in the recording device based on the time information. The reproduction step performs the reproduction through a reproduction screen,
The playback screen displays a time display area for displaying the time information, a moving image display area for displaying the mixed reality space image associated with the time information displayed in the time display area, and the time display area 3. A reproduction method according to claim 2, further comprising a position and orientation information display area for displaying the position and orientation information associated with the time information displayed on the display. The reproduction step displays the three-dimensional coordinates included in the position and orientation information as a three-dimensional trajectory in the position and orientation information display area,
When selection of an arbitrary point constituting the locus is detected, the mixed reality space image associated with the same time information as the position and orientation information corresponding to the selected point is displayed in the moving image reproduction area. The reproduction method according to claim 3, wherein: An image synthesis means for generating a mixed reality space image by synthesizing a virtual object image with a real space image obtained by photographing the real space;
Position and orientation detection means for detecting position and orientation information of a part of a worker's body observing the mixed reality space image;
A recording apparatus comprising: a storage unit that stores the mixed reality space image, the position and orientation information of the worker, and time information in association with each other in a recording apparatus. A mixed reality space image obtained by synthesizing a virtual object image with a real space image obtained by photographing the real space, and position and orientation information of a part of the body of the operator who observed the mixed reality space image; Recording means recorded in association with time information;
A playback apparatus comprising: playback means for playing back the mixed reality space image and the position / orientation information recorded in the storage means based on the time information. The reproduction means performs the reproduction through a reproduction screen,
The playback screen displays a time display area for displaying the time information, a moving image display area for displaying the mixed reality space image associated with the time information displayed in the time display area, and the time display area 7. A playback apparatus according to claim 6, further comprising a position and orientation information display area for displaying the position and orientation information associated with the time information displayed on the display. The reproduction means displays the three-dimensional coordinates included in the position and orientation information as a three-dimensional trajectory in the position and orientation information display area,
When selection of an arbitrary point constituting the locus is detected, the mixed reality space image associated with the same time information as the position and orientation information corresponding to the selected point is displayed in the moving image reproduction area. The playback apparatus according to claim 7, wherein   A program for causing a computer to execute at least one of the recording method according to claim 1 and the reproducing method according to any one of claims 2 to 4.   A computer-readable recording medium storing the program according to claim 9.

Images