JP4235522B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to amusement facilities such as amusement parks and theme parks, various display facilities such as museums, aquariums, botanical gardens, and zoos, and display devices and display methods used in various exhibition venues such as expositions and exhibitions. .

  Technically, the present invention relates to an image processing apparatus and an image processing method for generating a mixed reality space image in which a real space image and a virtual space image are superimposed.

  A system that synthesizes and prints a virtual object image generated in a computer with a real-space image captured using an imaging device such as a camera that can capture still images and moving images, such as a so-called “printer”. It exists widely as something. In such a “printer”, a virtual object superimposed on a real image is a simple two-dimensional image, and a system user can confirm the presence of the virtual object only on the composite screen during imaging. In other words, at the time of imaging, the system experience person makes a picture and acts for an image that seems to interact with the virtual object while checking the positional relationship between the virtual image and itself on the composite screen. It was.

  Conventionally, there is a system that provides a space (a mixed reality space) in which a coordinate system of a real space and a virtual space is shared (publicly known document 1).

  The applicant has proposed a system that prints by combining the real image and the virtual image in the viewpoint position and line-of-sight direction desired by the photographer in a space (mixed reality space) where the coordinate system of the real space and the virtual space is shared. ing. Here, the virtual object can be recognized as if it is a three-dimensional object that exists in the real space by using a three-dimensional computer graphics (3DCG) technique instead of a two-dimensional image. Furthermore, the three-dimensional coordinate axes of the real space and the virtual space are the same, and the experiencer has a deep sense of immersion as if the virtual object is actually present in this real space-virtual space (mixed reality space). It is possible to perceive with In such an environment, by capturing images of interaction with a virtual object as an image, the system experience is encouraged to interact with the virtual object naturally without being conscious of acting or painting. It is possible to provide better printing as a “picture”.

  On the other hand, at the theme park attraction, equipment that automatically captures the experience that the customer is experiencing and enjoys by an event in the real space and prints and provides the image captured to the user who has finished the experience is used Has been. For example, in a roller coaster type attraction, an image pickup device is installed at a point where the boarded vehicle suddenly descends and the experiencer is surprised, and the experiencer and the vehicle are photographed when the vehicle suddenly descends, and the boarding is finished. One example is a system in which the user pays a fee to print an image of his surprised expression.

In such a mixed reality system, real-time characteristics are required, so in the prior art, the model of the virtual space object has to be limited by the amount of data and the amount of calculation, and it has a better sense of fusion as a still image. It was difficult to create a mixed reality space image. In addition, when virtual space objects are moving in the mixed reality space, the experience is less likely to pay attention to a decrease in the sense of fusion due to these restrictions, but the low sense of fusion in still images and paper media output. I often felt the problem of low image quality.
JP 2002-271893 A

  An object of the present invention is to improve the quality of an image in which an experienced person and a virtual object naturally interact with each other, and to make the mixed reality space image more interesting for the experienced person who is the subject.

  It is another object of the present invention to provide a paper medium output without interrupting the operation of a system that experiences a mixed reality space.

  It is another object of the present invention to prevent the virtual space object from accidentally obscuring the observer across the front of the observer who is the subject when acquiring the mixed reality space image.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

The present invention is an image processing apparatus that is connected to a printing unit that prints an image and that can output an image to be printed to the printing unit, the acquisition unit acquiring a real space image from an imaging unit, Based on the position and orientation of the imaging means, the generating means for generating a virtual space image representing the virtual space, the real space image acquired by the acquiring means, and the virtual space image generated by the generating means, Synthesis means for generating a mixed reality space image, display means for displaying the mixed reality image, and output means for outputting the mixed reality space image to the printing means for printing the mixed reality image by the printing means. with the door, said generating means, when said display means displays the MR image, and generating the virtual space image in the first method, the output means outputs the mixed reality space image A method of performing a detailed influence on the virtual object in the virtual space by reflection of light more than the first method, and increasing the number of bits of numerical accuracy for generating the virtual space image than the first method. The virtual space image is generated by any one of a method for setting a vertical and horizontal pixel size of the virtual space image to be within a product range of a resolution and a print size of the printing unit, and the combining unit includes: A cylindrical overlap determination area is set for each of the observer and the virtual object, and is set for the observer when the output means outputs the mixed reality space image and the image is taken from the imaging means. When the superimposition determination area set for the virtual object is superimposed on the superimposed determination area, the virtual object is moved to a position where it is not superimposed on the observer, and the mixed reality space image is generated. If the display means displays the mixed reality space image, and generating the mixed reality space image without moving the virtual object.

  In the present invention, it is possible to provide a high-quality mixed reality image paper medium output.

  Also, in the present invention, it is possible to solve the problems in the prior art by creating a mixed reality space image with a large amount of information for paper media output by using another means or another data at the time of output. Become.

  Furthermore, in the present invention, even when a single observer prints out a high-quality mixed reality image while a plurality of observers are experiencing the mixed reality system at the same time, It is possible to obtain a print output of a mixed reality space image with high image quality at the same time without impairing the operation quality.

  Furthermore, in the present invention, it is possible to prevent the observer from being obscured by a virtual object in the paper medium output of the mixed reality space image obtained by capturing an image of the observer experiencing the mixed reality space. .

  Hereinafter, the best mode for carrying out the invention will be described in detail.

  FIG. 1 is a diagram illustrating a configuration of a system according to the present embodiment that enables acquisition of a mixed reality space image based on a plurality of observers and different viewpoints of these observers.

(Observer image processing apparatus and program)
First, an observer image processing apparatus and a program applied thereto in the present embodiment will be described.

  In FIG. 1, reference numerals 100 and 101 denote observers who observe a mixed reality space image.

  Image processing for an observer that generates a mixed reality space image for one observer by a PC 110, a monitor 140, position sensor systems 300, 310, and 320, and a head mounted display (HMD) 400 with an imaging device. The apparatus is configured and provided for the observer 100.

  Similarly, for the observer 101, a PC 111, a monitor 141, position sensor systems 301, 311, 321, and a head-mounted display device (HMD) 401 with an imaging device are provided as an image processing apparatus for the observer. It has been.

  In this embodiment, the number of observers is two, 100 and 101. However, the number of observers is not limited to this, and an arbitrary number of observer image processing apparatuses can be connected, and two or more observers can be connected. Any number of observers can simultaneously observe and share the mixed reality space.

  Reference numeral 190 denotes a network represented by Ethernet (R). An observer image processing apparatus provided for each observer, an objective viewpoint image processing apparatus (to be described later), and a mixed reality space management apparatus are networks. 190 is connected.

  Next, a program running on the PC 110 will be described. In the PC 110, a mixed reality space image generation program and a position / orientation acquisition program are operating.

  The position and orientation acquisition program constantly updates the viewpoint position and orientation of the observer measured by the position and orientation sensor main body 300, the position and orientation sensor transmitter 310, and the position and orientation sensor receiver 320 (see FIG. 5) that constitute the position and orientation sensor system. Then, after being converted into a position / orientation value on the coordinate axis 200 described later, it is continuously transmitted to the mixed reality space management program operating in the mixed reality space management device.

  The mixed reality space image generation program is based on the mixed reality space management information transmitted from the mixed reality space management program (that is, the position and orientation of all virtual space objects and the position and orientation of real space objects). A virtual space image at that time is drawn on the memory, and a real space image acquired by an imaging unit (described later) provided on the HMD 400 attached to the head of the observer 100 is drawn on the memory. Synthesize. As a result, a mixed reality space image is drawn on the memory.

  The mixed reality space image generated on the memory is output and displayed on a display unit (to be described later) provided in the monitor 140 and the HMD 400. Therefore, the observer 100 can observe the mixed reality space image by looking at the mixed reality space image displayed on the display unit of the HMD 400.

  Although the update frequency of the mixed reality space image depends on the processing capability of the PC, if the PC processing capability is sufficient and an update frequency of 10 to 30 Hz can be realized, the continuously updated mixed reality space image is an observer. Can be recognized as a video.

  The same video is output to the monitor 140 and the HMD 400. This is because a person other than the observer (a person who does not wear the HMD, an operator of the system, etc.) knows what kind of image the observer wearing the HMD 400 is viewing, and a system message or system. This is because the HMD 400 is inconvenient and inconvenient when it is operated by displaying the setting screen or the like, and it is necessary to perform the operation while looking at the monitor 140.

  With the above configuration and program, the observer 100 can freely change the viewpoint position and orientation, that is, freely move in the mixed reality space, and observe the mixed reality space image according to the viewpoint position and orientation at that time. It becomes.

(Mixed reality space management device and program)
Next, the mixed reality space management apparatus and the program applied to it in this embodiment will be described.

  The PC 115 and the monitor 145 constitute a mixed reality space management apparatus, and only one is installed in the mixed reality system regardless of the number of observers. In the PC 115, a mixed reality space management program is executed.

  The mixed reality space management program acquires the state and position / orientation of a real space object, updates the state and position / orientation of a virtual space object, and uses the obtained information as mixed reality space management information for all mixed reality space images. Notify the generator.

  Acquisition of the state and position / orientation of a real space object is realized by communicating with a position / orientation acquisition program operating in all image processing apparatuses for observers. As a result, the viewpoint positions and orientations of all the observers and the fixed installation cameras are centrally managed and synchronized by the mixed reality space management program.

  The update of the state, position, and orientation of the virtual space object is realized by updating the state, position, and orientation of all the virtual space objects by advancing the virtual space time according to a scenario preset in the program. At this time, if the position and orientation relationship between virtual space objects and the position and orientation relationship between the virtual space object and the real space object (observer and fixed installation camera) were in a special state set in the scenario, Some event (for example, an explosion of a virtual space object) may occur.

  The mixed reality space management information updated as described above is notified to all the reality space image generation programs.

(Objective viewpoint image processing device)
The PC 117, the monitor 147, and the video camera 160 constitute an objective image processing apparatus. Note that the objective image processing apparatus is an observer image processing apparatus except that the HMD 400 and the position / orientation sensor systems 300, 310, and 320 are not connected and the video camera 160 that is fixedly installed as an imaging apparatus is used. It is the same composition as.

  Here, it is desirable to set the installation position and orientation of the video camera 160 so that the mixed reality space image generation possible area described later falls within the imaging range as much as possible. In the PC 117, as with the PC 110, a mixed reality space image generation program and a position / orientation acquisition program are operating.

  The mixed reality space image generation program acquires a real image from the video camera 160 connected to the PC 117, and generates a mixed reality space image according to information notified from the mixed reality space management program. The position and orientation acquisition program continues to transmit the set values of the position and orientation of the video camera 160 to the mixed reality space management program. Note that the position and orientation of the video camera 160 are initially set in advance in the mixed reality space image generation program, and the position and orientation cannot be changed dynamically.

(Printer)
A printer 170 as a printing apparatus is an apparatus for outputting a mixed reality space image, and a shutter switch 180 is, for example, a push button type apparatus used by the observer 107 to determine an arbitrary output timing of the mixed reality space image. It is. In the present embodiment, the printer 170 and the shutter switch 180 are connected to the PC 117 of the objective viewpoint image processing apparatus, but are not limited to this configuration. That is, the printer 170 or the like may be connected to the PC 110 or PC 111 that functions as an image processing apparatus for an observer, or may be connected to a plurality of PCs at the same time. Note that if the PC is running a mixed reality space image generation program, the printer 170 can be connected to output a mixed reality space image.

  A mixed reality image viewed from the video camera 160 is displayed on the monitor 147 connected to the PC 117 of the objective viewpoint image processing apparatus. The observer 107 uses the shutter switch 180 while viewing the mixed reality image. It is possible to determine a mixed reality image output from the printer 170 at an arbitrary timing. Specifically, when the shutter switch 180 is pressed by the observer 107, the mixed reality space management program operating on the PC 117 detects it, and transmits the mixed reality image drawn on the memory to the printer 170. The printer 170 outputs it to a paper medium.

  Here, the recording method of the printer 170 is not particularly limited as long as the mixed reality space image can be printed as an image on a recording medium such as a paper medium or OHP.

  Further, the shutter switch 180 may not be a mechanical manual shutter switch as in this embodiment. Specifically, it is sufficient that the mixed reality space image generation program for performing the output process can instruct the timing for generating the mixed reality image for output, and this may be realized by the program. For example, a method may be considered in which a mixed reality space management program generates a shutter signal, which is included in mixed reality space management information and transmitted to the mixed reality space image generation program.

  In this case, the timing for generating the shutter signal may be automatically generated at a preset time interval by the mixed reality space image generation program, or a specific event in the mixed reality space managed by the shutter signal may be generated. It may be automatically generated when the error occurs.

(Mixed reality space)
Next, the mixed reality space will be described. In FIG. 1, reference numeral 200 denotes a coordinate axis that is defined in advance and is used to determine a position in the real space where the virtual object is localized, and is shared between the real space and the virtual space. The method for setting the coordinate axes is not described here because it is out of the spirit of the present invention.

  Reference numeral 210 denotes a mixed reality space image generation possible area. In other words, the position and shape of a real object that exists in this area are managed by the mixed reality space image generation program and the mixed reality space management program in which all the objects are operating. It is possible to detect that position overlap has occurred. The observer enters this area and experiences the mixed reality space. Note that the position and size of this area are recorded on the memory of the PC on which the mixed reality space image generation program and the mixed reality space management program operate.

  Reference numerals 220 and 222 denote real objects, and in this embodiment, stone pillars 220 and rocks 222 are shown as examples. Reference numerals 250 and 252 denote virtual objects, which are composed of data such as polygons and textures as computer graphics (CG) objects. In this embodiment, a turtle 250 and a fish 252 are shown as examples.

  In this example, a system that generates a mixed reality space with the ocean floor as a motif is illustrated, so that real objects and virtual objects are also presented in line with that purpose. It is not limited. The virtual objects 250 and 252 are clearly shown in FIG. 1 to show the outline of the mixed reality space, but in reality, only the monitor and the HMD connected to the PC on which the mixed reality space image generation program operates are used. I can't see it.

  The outline of the system configuration and operation of the present embodiment has been described above from both hardware and software.

(HMD)
Next, the configuration of the HMD 400 will be described with reference to FIGS. It should be noted that in each figure, a component with “R” after the component number represents an observer's right eye and a component with “L” represents a left eye. If it is not particularly necessary to specify the left and right, “R” and “L” after the number are omitted.

  FIG. 2 is a diagram showing a basic configuration of the HMD, and FIGS. 3 and 4 are external views of the HMD 400. FIG. In the figure, reference numeral 320 denotes a position / direction sensor receiver, which is provided in the vicinity of the observer's viewpoint position, operates with a position / orientation sensor transmitter 310 and a main body 300, which will be described later, and is a coordinate axis defined by the position sensor transmitter 310. The observer's viewpoint position and posture are constantly measured.

  Reference numerals 410, 414, and 415 are elements constituting the display system, and an image displayed on the color liquid crystal display 414 is guided by the optical prism 415 and displayed on the display unit 410. Reference numerals 412, 416, and 417 are elements constituting the imaging system. Light input from the outside of the HMD via the input unit 412 is guided into the HMD by the optical prism 417 and received by the CCD 416.

  The output light of the optical prism 415 and the input light of the optical prism 417 coincide with the optical axis of the observer's pupil, the CCD 416 captures a real space image of the observer's viewpoint position and orientation, and the color liquid crystal display 414 Then, a mixed reality space image obtained by combining the real space image acquired by the CCD 416 and the virtual space image generated according to the viewpoint position and orientation of the observer calculated by the position and orientation sensor main body 300 is displayed.

  421 to 425 are constituent members for mounting the HMD 400 on the head. In order to mount the HMD 400 on the head, first, the length adjuster 423 is loosened by the adjuster 422 and is put on the head. Then, after the forehead mounting portion 425 is in close contact with the forehead, the length adjuster 423 is narrowed by the adjuster 422 so that the temporal mounting portion 421 and the occipital mounting portion 424 are in close contact with the temporal and occipital regions, respectively. Reference numeral 426 is a summary of power supplies and signal lines for the color liquid crystal display 414, the CCD 416, and the position / orientation sensor receiver 320.

(Position and orientation sensor)
Next, the principle of measuring the viewpoint position and orientation of the observer in the position and orientation sensor will be described. As shown in FIG. 5, the sensor system (position and orientation acquisition apparatus) includes a position and orientation sensor main body 300, a position and orientation sensor transmitter 310, and a position and orientation sensor receiver 320.

  The position and orientation sensor receiver 320 and the position and orientation sensor transmitter 310 are connected to the position and orientation sensor main body 300. A magnetic signal is transmitted from the position / orientation sensor transmitter 310, and the position / orientation sensor receiver 320 receives this magnetic signal. The position / orientation sensor body 300 calculates the position and orientation of the position / orientation sensor receiver 320 from the intensity of the magnetic signal received by the position / orientation sensor receiver 320 using a known technique.

  Since the calculated position / orientation is a value with respect to the coordinate axis of the position / orientation sensor transmitter 310, the calculated result is sent to the image processing apparatus for the observer, where the viewpoint position and line-of-sight direction of the observer 100 on the coordinate axis 200 Is converted to The detailed procedure for the conversion is not described here because it is out of the spirit of the present invention, but the parameters for conversion are measured in advance and recorded in the observer image processing apparatus in such a way that the position and orientation acquisition program can be used. It is assumed that

  In this embodiment, it is assumed that the position / orientation acquisition device uses “FASTRAK” manufactured by Polhemus, Inc. or “Flock of Birds” manufactured by US Ascension Technology, but is not limited thereto. Other commercially available ultrasonic position and orientation sensors and optical position and orientation sensors can also be used.

  By using the position sensor systems 300, 310, and 320, the observer 100 can freely change the viewpoint position and the line-of-sight direction, and observe the mixed reality space image corresponding thereto via the HMD 400.

  Next, configurations of the observer image processing device, the objective image processing device, and the mixed reality space management device will be described in detail.

(Observer image processing device)
FIG. 5 is a block diagram showing a basic configuration of an observer image processing apparatus system and an HMD 400 that generate a mixed reality space image for one observer in the present embodiment. The position sensor systems 300, 310, and 320 can be broadly classified.

  The PC 110 includes a capture card 132, a graphic card 130, a serial interface 126, an HDD 123, a CPU 121, a memory 122, a keyboard 124, a mouse 125, and a network interface 127.

  Further, the capture card 132 functions as an interface for connecting the PC 110 and the CCD 416 of the HMD 400, and images of the real space of each frame captured by the CCD 416 are transferred to the memory 122 via the capture card 132.

  Further, the graphic card 130 has a function of generating and displaying three-dimensional computer graphics (3DCG) that is an image of the virtual object in cooperation with the PC 110 and an operating program.

  Also, the HDD 123 describes a program (mixed reality space image generation program, position / orientation acquisition program, etc.) and data (mixed reality space management information, the state of the mixed reality space over time) for controlling the entire PC 110. Scenarios, virtual space object shape data, position and orientation conversion parameters, etc.), and these programs and data are loaded into the memory 122 by the CPU 121 as necessary.

  Further, the CPU 121 controls each unit of the PC 110 by executing a program loaded in the memory 122 or referring to data loaded in the memory 122. Further, the CPU 121 can generate a mixed reality space image and control the entire PC 110. The memory 122 is a readable / writable memory and includes an area for temporarily storing a program loaded from the HDD 123 to be executed by the CPU 121 and data used at that time, and when the CPU 121 performs each process. It also has a work area used for

  The memory 122 also includes an area for drawing a mixed reality space image that is an image in which a virtual space image and a real space image are superimposed. The keyboard 124 and the mouse 125 are used as pointing devices for inputting instructions to the CPU 121. For example, these pointing devices are used for inputting a virtual object arrangement position, a virtual space state setting, and the like.

  As described above, the CCD 416 is connected to the capture card 132, and the CCD 416 captures a moving image in the real space and outputs the captured image of each frame to the memory 122 via the capture card 132.

  Further, as described above, the color LCD 414 and the monitor 140 are connected to the graphic card 130, and the 3DCG generated by the graphic card 130 is displayed, and a moving image of the real space captured by the CCD 416 superimposed on the 3DCG is also displayed. indicate. That is, these display means display a mixed reality space image. Further, the color LCD 414 and the monitor 140 display not only the mixed reality space image but also information (system message, system setting screen, etc.) necessary for operating the PC 110.

  A position / orientation sensor main body 300 is connected to the serial interface 126. The position / orientation sensor main body 300 is a position obtained based on the strength of the magnetic signal obtained by the position / orientation sensor transmitter 310 and the position / orientation sensor receiver 320 incorporated in the HMD 400 attached to the head of the observer. The position and attitude of the attitude sensor receiver 320 are sent to the PC 110 via the serial interface 126.

  The PC 110 converts the sent position / orientation of the position / orientation sensor receiver 320 into a position / orientation on the coordinate axis 200 and transmits it to the PC 115 of the mixed reality space management apparatus in a predetermined format together with other data described later.

(Objective image processing device)
Next, the objective image processing apparatus will be described. FIG. 6 is a block diagram showing a basic configuration of the objective image processing apparatus system according to the present embodiment and a relationship with a device for outputting a mixed reality image from the objective image processing apparatus system on a paper medium. In FIG. 6, the same components as those shown in FIG. 5 are denoted by the same reference numerals, and the description thereof is omitted.

  The basic configuration of the PC 117 in FIG. 6 is almost the same as that of the PC 110 shown in FIG. 5 except that a printer 170 and a shutter switch 180 are connected to the serial interface 126.

  The printer 170 prints the mixed reality space image data developed on the memory 122 by the PC 117 on a paper medium at the timing when the shutter switch 180 is pressed.

(Mixed reality space management device)
Next, the mixed reality space device will be described. As shown in FIG. 1, the mixed reality space management apparatus is configured only by a PC 145 having a network interface. Note that the internal configuration of the PC 145 is substantially the same as that of the PC 117, and therefore the description thereof is omitted here.

  Further, it is performed based on a position and orientation acquisition program, a mixed reality space management program, and a mixed reality space image generation program executed by the objective image processing device, the observer image processing device, and the mixed reality space management device having the above-described configuration. The processing will be described in detail with reference to the flowcharts of FIGS. All programs are created by applying threads, which is one of parallel programming techniques, and the main program and threads created thereby operate simultaneously.

  In addition, serial numbers (device numbers) of devices in the entire system are set in advance for all the processing devices. Further, the objective image processing device and the observer image processing device include a serial number (ID number) for identifying a related real object (camera, observer), and a type (type of type) of a real image capturing camera (HMD400, video camera 160, etc.). Information such as (number) and optical characteristics (camera parameters) and physical shape information of the observer (boundary box, etc.) are preset, and the program can refer to these values.

(Position and orientation acquisition program)
First, processing performed by executing the position / orientation acquisition program will be described with reference to the flowchart of FIG.

  When the position / orientation acquisition program is started on the PC 110 in the observer image processing apparatus, a position sensor measurement value reception thread is first created. The position sensor measurement value reception thread repeatedly receives the measurement value from the position sensor system and transmits it to the main program via the serial interface (S750).

  The main program receives this measurement value (S700), converts it into a viewpoint position and orientation on the coordinate axis 200 (world coordinate system) (S702), sends it to the mixed reality space management program via the network 190 (S704), and then Processing for terminating the operation is performed (S706).

  Further, when the position / orientation acquisition program is started on the PC 117 in the objective image processing apparatus, a position sensor measurement value reception thread is created in the same manner as in the observer image processing apparatus.

  The position sensor measurement value reception thread tries to receive measurement values from the position sensor system via the serial interface, but since the position sensor system is not connected to the objective image processing apparatus, it is set in the program in advance. The transmission / reception of the video camera 160 (fixed value) is periodically transmitted to the main program (S750).

  The main program receives this measurement value (S700), converts it into a viewpoint position and orientation on the coordinate axis 200 (world coordinate system) (S702), sends it to the mixed reality space management program via the network 190 (S704), and then Processing for terminating the operation is performed (S706).

  As described above, all the position / orientation acquisition programs transmit the observer's viewpoint position / orientation or the viewpoint position / orientation of the fixed installation camera to the mixed reality space management program.

(Mixed reality space management program)
Next, processing performed by executing the mixed reality space management program will be described with reference to the flowchart of FIG.

  When the mixed reality space management program is started on the PC 115 in the mixed reality space management apparatus, a position and orientation acquisition thread is first created. The position / orientation acquisition thread receives the observer's viewpoint position / orientation or the video camera's viewpoint position / orientation (real space state information) transmitted from all the position / orientation acquisition programs, and repeats transmission to the main program (S850). .

  The main program receives the viewpoint position / posture value (S800), waits for a predetermined time dt to elapse (S802), advances the time of the managed virtual space by dt (S804), and previously According to the scenario stored in the HDD 123 and the elapsed time, the position, posture and state (virtual space state information) of the virtual space object are updated (S805).

  Subsequently, it is determined whether the virtual space object and the real space object (observer's viewpoint position and orientation) have a special relationship in the scenario (S806). If so, the position and orientation of the virtual space object are determined according to the scenario. And the state (virtual space state information) are updated again in the same manner as in S805 (S808). The mixed reality space state information (real space state information and virtual space state information) updated as described above is transmitted to all operating mixed reality space image generation programs via the network 190 (S810). Processing to end the operation is performed (S812).

  As described above, the mixed reality space management program uniformly manages and updates the mixed reality space information according to a scenario prepared in advance, and transmits it to all the mixed reality space image generation programs.

(Mixed reality space image generation program)
Next, processing performed by executing the mixed reality space image generation program will be described with reference to the flowchart of FIG.

  When the mixed reality space image generation program is started on the PC 110 in the observer image processing apparatus, first, a video capture thread and an image output thread are created.

  The video capture thread continuously acquires the real space image captured by the CCD 116 of the HMD 400 via the video capture card 132 and continues to transmit it to the main program.

  The image output thread is a program that outputs a mixed reality image from the printer 170 in accordance with the operation of the shutter switch 180. In the observer image processing apparatus according to the present embodiment, the printer 170 and the shutter switch 180 are not connected, and therefore the image output thread performs no processing.

  When the mixed reality space state notified from the virtual space management program is received (S902), a virtual space image is generated using the received position and orientation and state of the virtual space object and 3DCG data loaded from the HDD 123 to the memory 122. Generate and draw on the memory 122 (S904). This is called the first method.

  In order to present an immersive mixed reality space image to the viewer, the generated virtual space image has a quality sufficient to allow the viewer to understand that the still image of the image alone is 3DCG, even if it is different from the actual image. Even so, a series of continuously generated mixed reality images are recognized by the observer as moving images, and autonomously moving lively in the mixed reality space as if the virtual space object is a real space object. It is important to make it feel like this. Therefore, in the virtual space image generation processing here, the processing is required to be completed in a short time of about 15 to 40 msec even if the still image is not realistic. There is no particular limitation on the method as long as this is achieved.

  Subsequently, after rendering the real space image transmitted from the above-described video capture thread on the memory 122 (S906), the mixed real space image obtained by superimposing the previously generated virtual space image is similarly used. Drawing on the memory 122. The mixed reality space image is output to the HMD 400 and the monitor 140 via the graphic board 130 (S908).

  By repeating the above processing, a continuous mixed reality space image, that is, a mixed reality space image can be presented to the observer.

  Note that the processing after S910 and the processing of the image output thread are not executed in the observer image processing apparatus to which the printer 170 and the shutter switch 180 are not connected. Therefore, a detailed description thereof will be given in the description of the mixed reality space image generation program in the objective viewpoint image processing apparatus.

  Next, processing performed by the PC 112 in the objective image processing apparatus system by executing the mixed reality space image generation program will be described with reference to FIG. 9 showing a flowchart of the processing.

  When the mixed reality space image generation program is started on the PC 117 in the objective viewpoint image processing apparatus, first, a video capture thread and an image output thread are created.

  The video capture thread continuously acquires the real space image captured by the video camera 160 via the video capture card 132 and continuously transmits it to the main program (S970).

  The image output thread is a program that outputs a mixed reality image from the printer 170 in accordance with the operation of the shutter switch 180. The image output thread continues to wait until a shutter event is generated from the main program, and when this occurs, a mixed reality image is output from the printer 170 (S950).

  The processes in steps S902 to S908 and S970 are the same as the mixed reality space image generation program of the PC 110 in the observer image processing apparatus described above, and by repeating these processes, a mixed reality space image continuous to the observer, That is, a mixed reality space image can be presented. However, a printer 170 and a shutter switch 180 are connected to the PC 117 in the objective viewpoint image processing apparatus, and the following processing is executed.

  After the mixed reality space image is generated and displayed (S908), it is determined whether or not the shutter switch 180 is operated (S910). If there is an operation, a shutter event is generated (S912), and then the operation is terminated. (S914). Here, an event means a means for communicating with a thread managed in this program.

  When the shutter event occurs, the image output thread (S950) resumes operation (S952), and the printer 170 uses the received position and orientation of the virtual space object and 3DCG data loaded from the HDD 123 to the memory 122. A virtual space image to be output to is generated and rendered on the memory 122 (S954). This is called the second method.

  In step S904, the virtual space image is generated in such a way that the processing can be completed in a short time even if the sense of reality is poor as a still image. It is necessary to generate a virtual object image by a method (different from S904) that generates a virtual space image in which 3DCG is fused with a live-action image and a sense of reality is abundant.

  For example, a method for calculating in detail the effect of light reflection such as ray tracing and radiosity, a method for calculating by increasing the number of bits of numerical accuracy, and a vertical and horizontal pixel size of the virtual space image are set to the printer 170 Various methods such as a method of performing calculation with an arbitrary value within the range of the product of the resolution and the print size are conceivable.

  The virtual space image generated in this way is superimposed on the real space image acquired in S906 (S956). Here, when the vertical and horizontal pixel sizes of the virtual reality space image and the real space image differ according to the generation method described above, an image with a small pixel size is changed to the pixel size of a large image using a known interpolation technique. You should expand to. Finally, the generated mixed reality space image is transmitted to the printer 170 (S958).

  Since the image output thread and the main program operate in parallel, even if it takes time to generate a mixed reality space image for print output, the processing in steps S902 to S914 continues to be executed without interruption, and the monitor 147 The display of the mixed reality space image continues. Therefore, it is possible to obtain a high-quality mixed reality space image printout without impairing the mixed reality space experience.

(Example of mixed reality space image)
Next, an example of a mixed reality space image will be described. 10 and 11 respectively show an example of a mixed reality space image synthesized in S908 and displayed on a monitor or HMD, and an example of a mixed reality space image synthesized in S956 and output from the printer 170. All are mixed reality images at the same time, the same viewpoint, and the line-of-sight direction, but it can be seen that the mixed reality space image output from the printer 170 is a high-quality image with a sense of fusion with a real object.

  As described above, the mixed reality space image obtained by imaging the state in which the observer is experiencing the mixed reality space by the system according to the present embodiment is recorded in a high-quality recording medium such as a paper medium or an OHP without affecting the performance of the system. It is understood that it can be output to.

  In the present embodiment, high-quality print output is realized by using different generation methods for generating a virtual space image for display and a virtual space image for print output. Needless to say, the quality of the print output can be improved by using different data for print and print output.

  Specifically, model data and texture data with a large number of polygons for the same virtual object are stored in advance in the HDD 123, and the same are obtained by switching them in the processing in S906 and S954. The effect of may be obtained. Further, the quality may be improved by using both the generation method and the data together.

  In the present embodiment, a case will be described in which when a virtual space object is superimposed on an observer as viewed from the imaging apparatus, the virtual space object is removed from the mixed reality space to generate a mixed reality space image.

(Position and orientation acquisition program)
First, processing performed by executing the position / orientation acquisition program by the observer image processing apparatus and the objective image processing apparatus will be described with reference to the flowchart of FIG.

  When the position / orientation acquisition program is started on the PC 110 in the observer image processing apparatus, a position sensor measurement value reception thread is first created.

  The position sensor measurement value reception thread repeatedly receives the measurement value from the position sensor system and transmits it to the main program via the serial interface (S1750). The main program receives this measurement value (S1700), and converts it into a viewpoint position and orientation on the coordinate axis 200 (world coordinate system) (S1702). The converted value is stored as one entry of the mixed reality object table (imaging device information) created in the memory 122 together with the object type number, device number, ID number, type number, and additional information (S1706).

  FIG. 20 shows a mixed reality object table. The mixed reality object table includes a plurality of entries, and each entry includes six fields of an object type number, a device number, an ID number, a type number, a position and orientation, and additional information.

  Specific contents of the entry added in step S1706 are shown in FIG. The object type number is “0” representing a real object. As the device number and ID number, values preset in the PC 110 are read and stored. The type number is “0” representing the observer camera. Subsequently, the viewpoint position and line-of-sight direction in the right eye of the observer converted in S1702 are stored. Finally, additional information follows. The additional information is the object type number and special information corresponding to the type number, and in the case of an observer camera, a value (camera parameter) that configures a view frustum that defines the imaging range of the real image imaging camera is stored. Is done. Specifically, values such as θ, far, near, w, and h in FIG.

  Next, it is checked whether the type number is an observer camera (S1710). If the type number is an observer camera, the center position and orientation of the observer are calculated from the previously calculated viewpoint position and line-of-sight direction of the observer. (S1712). Here, the center position of the observer is a position set arbitrarily. For example, it may be near the center of the head or near the pigeon tail. When the center position of the observer is obtained, it is stored as one entry in the mixed reality object table together with other data (S1714).

  The specific contents of the entry created here are shown in FIG. The object type number is “0” representing a real object. As the device number and ID number, values preset in the PC 110 are read and stored. The type number is “2” representing the observer. Subsequently, the center position and posture of the observer calculated in S1712 are stored. Finally, additional information follows. In the case of an observer, this is observer shape information. Specifically, as shown in FIG. 18, the radius r and height h of the cylinder into which the observer 100 enters completely are stored. The center P of the cylinder is the above-described center position of the observer, and it is determined by this information whether or not the observer is in contact with the virtual object in the mixed reality space.

  A mixed reality table is created as described above. The mixed reality object table created in this way is transmitted to the mixed reality space management program via the network 190 (S1716). The above process is repeated, and then a process for terminating the operation is performed (S1718).

  When the position / orientation acquisition program is started on the PC 117 in the objective image processing apparatus, a position sensor measurement value reception thread is created in the same manner as in the observer image processing apparatus.

  The position sensor measurement value reception thread tries to receive measurement values from the position sensor system via the serial interface. However, since the position sensor system is not connected to the objective image processing apparatus, the position of the video camera 160 is determined. The posture is periodically transmitted to the main program repeatedly (S1750). This value is a fixed value set in advance by the processing device.

  Thereafter, the mixed reality object table shown in FIG. 20 is created in the same procedure as the operation of the position / orientation acquisition program in the image processing apparatus for observers.

  The concrete contents of the entry created here are shown in FIG. The object type number is “0” representing a real object. As the device number and ID number, values preset in the PC 110 are read and stored. The type number is “1” representing the objective camera. Subsequently, the viewpoint position and line-of-sight direction in the video camera 160 converted in S1702 are stored. Finally, additional information follows. As in the case of the observer camera, values constituting the viewing frustum are stored.

  The mixed reality object table created in this way is transmitted to the mixed reality space management program via the network 190 (S1716). The above process is repeated. Note that the processing in steps S1712 and S1714 is not executed because the type number is an objective camera.

  As described above, all the position / orientation acquisition programs transmit the imaging device information to the mixed reality space management program in the form of a mixed reality object table.

(Mixed reality space management program)
Next, processing performed by executing the mixed reality space management program will be described with reference to the flowchart of FIG.

  When the mixed reality space management program is started on the PC 115 in the mixed reality space management device, an imaging device information acquisition thread is first created.

  The imaging device information acquisition thread receives the mixed reality object table (imaging device information) transmitted from all the position and orientation acquisition programs, and repeatedly transmits to the main program (S1850).

  The main program receives the mixed reality object table (S1800), waits for a predetermined time dt to elapse (S1806), and then enters a real object entry (object type number) in the mixed reality object table held by the program. Is updated with the previously received mixed reality object table (S1808).

  Next, the time of the managed virtual space is advanced by dt (S1810), and the scenario stored in advance in the HDD 123 is advanced according to the elapsed time, and the positions, postures and states of all virtual space objects are updated. To do. Subsequently, the virtual object entry (entry whose object type number is 1) in the mixed reality object table held by the program is updated with the updated virtual space object information (S1812).

  The specific contents of the entry to be updated are shown in FIG. The object type number is “1” representing a virtual object. As the device number, a value preset in the PC 115 is read and stored. The ID number is a serial number of the virtual space object managed by this program. The type number is a number indicating the type of the virtual space object managed by this program. Here, as an example, shark is “0” and turtle is “1”. The position and orientation represent the center position and direction of the virtual space object. Additional information stores information for expressing the virtual space object. For example, it is assumed that the texture data number for representing the virtual space object and the data representing the status of the virtual object are set according to the scenario. Further, the radius r ′ and the height h ′ of the cylinder into which the virtual object itself for examining whether or not the virtual object has contacted in the mixed reality space are also stored. Similar to the observer described above, the center of the cylinder is the center position of the virtual object.

  Subsequently, it is determined whether the virtual space object and the real space object (observer's viewpoint position and orientation) have a special relationship in the scenario (S1814). If so, the position and orientation of the virtual space object are determined according to the scenario. And the state (virtual space state information) are updated again in the same manner as in S1812 (S1816). This special state refers to, for example, a collision between a virtual space object and a real object (observer) in the mixed reality space.

  The mixed reality space update process is realized as described above.

  Subsequently, a process for checking whether or not a virtual space object exists (overlaps) between the imaging device and the observer is executed.

  In this embodiment, the object relation table shown in FIG. 25 is used as information indicating the occurrence of superposition. If an entry exists in this table, it indicates that superimposition has occurred. As shown in FIG. 25, the entry of the object relation table includes the device number of the device to which the objective camera or the observer camera is connected, the ID number of the objective camera or the observer camera, the ID number of the observer, and the virtual object ID number. It consists of four fields.

  FIG. 26 shows an example of a specific object relation table entry. The entry in the object relation table in FIG. 26 indicates that a virtual object with ID number 6 exists between the camera with device number 1 and ID number 1 and the observer with ID number 2 in the mixed reality space. ing.

  First, the object relation table held by this program is deleted, and preparation for examining occurrence of superimposition is made (S1828).

  Next, it is detected whether an observer is included in an image captured by the imaging apparatus. Specifically, from the entry of the mixed reality object table created by the preprocessing, the imaging device entry (entry with object type number “1”, type number “0” or “1”) and the observer entry (object type) All combinations of the entries having the number “1” and the type number “2”) are extracted (S1830, S1832), and it is checked whether or not the view frustum of the imaging device in the mixed reality space and the observer intersect (S1834). ). Note that, as described above, calculation is performed by assuming that the observer is a cylinder determined from the observer center position and the additional information of the observer entry.

  If the viewing frustum of the imaging device and the observer intersect, then it is checked whether or not a virtual object exists between the imaging device and this observer in the mixed reality space (S1836). Specifically, a virtual object entry (an entry with an object type number “0”) is extracted from the mixed reality object table entry created by the preprocessing, and all the extracted virtual objects are shown in FIG. It is examined whether or not the cone E-ABCD whose bottom surface is ABCD is crossed (S1838). Here, ABCD has a straight line passing through the position E of the imaging device and the observer center P as a perpendicular line, and a radius r and a height h created from information of the observer entry with respect to a plane m in contact with the cylinder (at the contact point Q). Is a figure obtained by mapping the cylinder from the position of the imaging device.

  When there is an intersecting virtual object, the distance between the virtual object center and the plane m and the line segment PN are compared. Then, the two are compared, and when the PN is smaller, it is determined that the observer viewed from the imaging device is covered with this virtual object (overlapping occurs). If it is determined that superimposition has occurred, an object relationship entry is created from the device number of the imaging device entry, the ID number, the ID number of the observer entry, and the ID number of the virtual object entry, and is added to the object relationship table (S1840). Note that the processing in steps S1836, S1832, and S1830 is repeated, and the above-described processing is performed for all imaging devices (S1842, S1844, S1846).

  The created object relation table is transmitted to the mixed reality space management program via the network 190 as mixed reality space information together with the mixed reality object table (S1848). The above-described series of processing is repeated, and then processing for ending the operation is performed (S1849).

  In this way, the mixed reality space management program uniformly manages and updates the mixed reality space information in accordance with a prepared scenario, and transmits it to all the mixed reality space image generation programs.

(Mixed reality space image generation program)
Next, processing performed by executing the mixed reality space image generation program will be described with reference to the flowchart of FIG.

  When the mixed reality space image generation program is started on the PC 110 in the observer image processing apparatus, first, a video capture thread and an image output thread are created.

  The video capture thread continuously acquires the real space image captured by the CCD 116 of the HMD 400 via the video capture card 132 and continues to transmit it to the main program.

  The image output thread is a program that outputs a mixed reality image from the printer 170 in accordance with the operation of the shutter switch 180. In the observer image processing apparatus according to the present embodiment, since the printer 170 and the shutter switch 180 are not connected, the image output thread performs no processing.

  When the mixed reality space state notified from the virtual space management program is received (S1902), the position and orientation and the state of the virtual space object are extracted from the received mixed reality object table, and the 3DCG data loaded from the HDD 123 to the memory 122 Is used to generate a virtual space image and draw it on the memory 122 (S1904).

  In order to present an immersive mixed reality space image to the viewer, the generated virtual space image has a quality sufficient to allow the viewer to understand that the still image of the image alone is 3DCG, even if it is different from the actual image. Even so, a series of continuously generated mixed reality images are recognized by the observer as moving images, and autonomously moving lively in the mixed reality space as if the virtual space object is a real space object. It is important to make it feel like this. Therefore, in the virtual space image generation process here, it is required that the process be completed in a short time of about 15 to 40 msec even if the still image is not realistic. If this is achieved, the method is not particularly limited.

  Subsequently, after the real space image (S1970) transmitted from the video capture thread described above is rendered on the memory 122 (S1906), the mixed real space image on which the previously generated virtual space image is superimposed is also stored in the memory. Draw on 122. The mixed reality space image is output to the HMD 400 and the monitor 140 via the graphic board 130 (S1908).

  By repeating the above processing, a continuous mixed reality space image, that is, a mixed reality space image can be presented to the observer. Note that the processing after S1910 and the processing of the image output thread are not executed in the observer image processing apparatus to which the printer 170 and the shutter switch 180 are not connected. Therefore, the detailed description thereof will be made in the description of the mixed reality space image generation program in the objective viewpoint image processing apparatus.

  Next, processing performed by the PC 112 in the objective image processing apparatus system executing the mixed reality space image generation program will be described with reference to FIG. 14 showing a flowchart of the processing.

  When the mixed reality space image generation program is started on the PC 117 in the objective viewpoint image processing apparatus, first, a video capture thread and an image output thread are created.

  The video capture thread continuously acquires the real space image captured by the video camera 160 via the video capture card 132 and continuously transmits it to the main program.

  The image output thread is a program that outputs a mixed reality image from the printer 170 in accordance with the operation of the shutter switch 180. The image output thread continues to wait until a shutter event is generated from the main program, and when it is generated, a mixed reality image is output from the printer 170.

  Note that the processing in steps S1902, S1904, S1906, S1908, and S1970 is the same as steps S902, S904, S906, S908, and S970 in the mixed reality space image generation program of the PC 110 in the observer image processing apparatus in FIG. Yes, by repeating these processes, it is possible to present a continuous mixed reality space image, that is, a mixed reality space image, to the observer. However, a printer 170 and a shutter switch 180 are connected to the PC 117 in the objective viewpoint image processing apparatus, and the following processing is executed.

  After the mixed reality space image is generated and displayed (S1908), it is determined whether or not the shutter switch 180 is operated (S1910). If there is an operation, a shutter event is generated (S1912). Here, an event means a means for communicating with a thread managed in this program.

  When a shutter event occurs, the image output thread (S1950) resumes operation (S1952), and the following processing is performed for all entries in the received object relationship table.

  The device number of the processing device on which this program is operating is compared with the first field of the entry, and if they are the same (that is, the imaging device connected to the processing device on which this program is operating takes a picture) For the observer in the image, the virtual space object described in this entry is excluded from the virtual space (S1954, S1956, S1958). Specifically, the virtual space object entry having the same ID number is deleted from the received mixed reality object table.

  After processing as described above, the position and orientation and state of the virtual space object are extracted from the mixed reality object table (S1959), and the virtual object for output to the printer 170 using the 3DCG data loaded from the HDD 123 to the memory 122 is extracted. A spatial image is generated and drawn on the memory 122 (S1960). The virtual space image generated in this way is superimposed on the real space image acquired in S1906 (S1962). Here, when the vertical and horizontal pixel sizes of the virtual reality space image and the real space image are different from each other by the generation method described above, an image having a small pixel size is converted into a pixel size of a large image by using a known interpolation technique. You just have to enlarge. Then, the finally generated mixed reality space image is transmitted to the printer 170 (S1964).

  After the mixed reality space image is generated and displayed (S1908), it is determined whether or not the shutter switch 180 is operated (S1910). If there is an operation, a shutter event is generated (S1912), and then the operation ends. (S1914).

(Example of mixed reality space image)
Next, an example of the mixed reality space in the present embodiment will be described. FIGS. 15 and 16 are diagrams illustrating an example of a mixed reality space image synthesized in S1908 and displayed on a monitor or an HMD, and an example of a mixed reality space image synthesized in S1964 and output from the printer 170, respectively. Although a cylinder is shown for ease of understanding, it is not displayed in an actual output image.

  All are mixed reality images at the same time, the same viewpoint, and the line-of-sight direction. However, in the mixed reality space image output from the printer 170, there is no virtual object that covers the front of the viewer, and the viewer's face is clear. It is an image that can be confirmed.

  As described above, in the mixed reality space image in which the observer is experiencing the mixed reality space, the viewer is unexpectedly obscured by the virtual object that has crossed in front of the imaging device. It is understood that it can be prevented. In addition, it is understood that the output can be performed on a recording medium such as a paper medium or an OHP with high quality without affecting the performance of the system.

  In this embodiment, the contact determination between the observer and the virtual object is performed using a simple solid body called a cylinder. However, if the processing apparatus has high capability, it may be determined by defining a more complicated shape. . Moreover, you may operate | move so that only an observer's head (face) may not be covered with a virtual object by setting the observer's center position, the radius r, and height h of a cylinder freely.

  Since the image output thread and the main program operate in parallel, even if it takes time to generate a mixed reality space image for print output, the processes in steps S1902 to S1914 are repeatedly executed without interruption, The monitor 147 continues to display the mixed reality space image. This makes it possible to obtain a high-quality mixed reality space image printout without compromising the mixed reality space experience.

  In the second embodiment, in the second embodiment, when a virtual space object is superimposed on an observer as viewed from the imaging device, the virtual space object is removed from the mixed reality space to generate a mixed reality space image. The case where a virtual space object is moved to a position where it is not superimposed to generate a mixed reality space image is shown. The basic configuration is the same as that of the second embodiment, and the flow of processing is almost the same as that of the second embodiment except for some processes in the mixed reality space management program and the mixed reality space management program.

  The flow of processing of the mixed reality space management program in the third embodiment is shown in the flowchart of FIG. Here, the step number with “′” shown in FIG. 27 is a step for performing the same processing as the processing of the step with the same step number in FIG. 13 showing the flow of the same program processing in the second embodiment. Note that FIG. 27 differs from FIG. 13 only in the process of S1839.

  In step S1839, processing that is executed when it is determined that the observer viewed from the imaging device is covered with the virtual object (overlap occurrence) is performed. As in the second embodiment, the criterion for determining the occurrence of superposition is that the viewing frustum of the imaging device and the observer intersect, and the cone E-ABCD whose bottom surface is ABCD and the virtual object as shown in FIG. 19 intersect. In this case, the line segment PN is smaller than the distance between the virtual object center and the plane m.

  If it is determined that superimposition has occurred, the position of the virtual object at which superimposition does not occur is calculated (S1839), the device number of the imaging device entry, the ID number, the ID number of the observer entry, the ID number of the virtual object entry, and the calculated An object relation entry is created from the virtual object position and added to the object relation table (S1840 ′). In the object relation table, as shown in FIG. 29, a field for recording a virtual object position where superimposition does not occur is added from the second embodiment.

  The calculation in step S1839 will be specifically described. First, perpendicular lines n1, n2, to a plane passing through the center P ′ of the virtual object that is superimposed and on which the four side surfaces (ΔEAB, ΔEBC, ΔECD, ΔEDA) of the cone E-ABCD are placed, n3 and n4 and intersections Q1, Q2, Q3, and Q4 of these perpendicular lines and the respective surfaces are obtained. Then, among the four intersections, the one having the smallest distance from the virtual object center P ′ is obtained, and the side surface on which the intersection exists is the surface m ′ and the perpendicular is n ′. And at a point on the normal n ', the distance from the surface m' outside the cone is

The point P ″ that becomes is the center of this virtual object.

  Here, as in the second embodiment, h ′ is the height of the cylinder surrounding the virtual object, r ′ is the bottom radius of the cylinder surrounding the virtual object, and is recorded in the additional information of the entry indicating this virtual object in the mixed reality object table. ing. θ is an angle formed by the perpendicular line n ′ and a line segment in the height direction of the cylinder surrounding the virtual object. The range of the angle θ is 0 to 180 °. When this point P ″ is newly set as the center of the virtual object, the virtual object comes to be in contact with the surface m ′ outside the cone.

  As described above, when superimposition occurs, the position of the virtual space object that does not cause superimposition is calculated and transmitted to the mixed reality space image generation program as an object relation table.

  Next, the flowchart of FIG. 28 shows the flow of processing performed by the PC 112 in the objective image processing apparatus system according to the third embodiment by executing the mixed reality space image generation program. Here, the step number with “′” shown in FIG. 28 is a step for performing the same processing as the processing of the step with the same step number in FIG. 14 showing the flow of similar program processing in the second embodiment. Note that FIG. 28 differs from FIG. 14 only in the process of S1957.

  When a shutter event occurs after the generation and display of the mixed reality space image, the image output thread performs the following processing for all entries in the received object relation table.

  The device number of the processing device on which this program is operating is compared with the first field of the object relation table entry, and if they are the same (that is, the imaging connected to the processing device on which this program is operating) If superimposition has occurred for the observer in the image captured by the apparatus), the virtual space object described in the object relation table entry is found from the mixed reality object table and stored in the mixed reality object table entry. The position of the virtual object being updated is updated with the value stored in the fifth field of the object relation table entry (S1957).

  After processing as described above, the position and orientation and the state of the virtual space object are extracted from the mixed reality object table, and the virtual space image to be output to the printer 170 using the 3DCG data loaded from the HDD 123 to the memory 122 is obtained. Generate and draw on the memory 122 (S1960 ′).

(Example of mixed reality space image)
Next, an example of a mixed reality space image will be described. 30 and 31 are diagrams illustrating an example of a mixed reality space image synthesized in S1908 ′ and displayed on a monitor or HMD, and an example of a mixed reality space image synthesized in S1960 ′ and output from the printer 170, respectively. is there. Although a cylinder is shown for ease of understanding, it is not displayed in an actual output image.

  All are mixed reality images at the same time, the same viewpoint, and the line-of-sight direction, but in the mixed reality space image output from the printer 170, the virtual object covering the front of the observer has moved to another position, It can be seen that the person's face is clearly visible.

  As described above, in the mixed reality space image in which the observer is experiencing the mixed reality space, the viewer is unexpectedly obscured by the virtual object that has crossed the front of the imaging device. Can be prevented. Further, it is possible to output to a recording medium such as a paper medium or an OHP with high quality without affecting the performance of the system.

It is a block diagram of a mixed reality apparatus. It is a block diagram of a display apparatus (HMD). It is a block diagram of a display apparatus (HMD). It is a block diagram of a display apparatus (HMD). It is a block diagram of an observer mixed reality space image processing apparatus. It is a block diagram of an objective mixed reality space image processing apparatus. It is a flowchart of a position and orientation acquisition program. It is a flowchart of a mixed reality space management program. It is a flowchart of a mixed reality space image generation program. It is a figure which shows the example of the image in the display screen of a mixed reality image. It is a figure which shows the example of the image in the paper medium output of a mixed reality image. 10 is a flowchart of a position / orientation acquisition program according to the second embodiment. 10 is a flowchart of a mixed reality space management program in Embodiment 2. 12 is a flowchart of a mixed reality space image generation program according to the second embodiment. FIG. 10 is a diagram illustrating an example of a mixed reality image display screen according to the second embodiment. FIG. 10 is a diagram illustrating an example of a paper medium output image of a mixed reality image in the second embodiment. It is a figure which shows the example of a view frustum. It is a figure which shows the example of the observer shape in mixed reality space. It is a figure which shows the determination area | region. It is a figure which shows the example of a format of a mixed reality object table. It is a figure which shows the example 1 of an entry of a mixed reality object table. It is a figure which shows the example 2 of an entry of a mixed reality object table. It is a figure which shows the example 3 of an entry of a mixed reality object table. It is a figure which shows the example 4 of an entry of a mixed reality object table. FIG. 10 is a format diagram of an object relation table in Embodiment 2. It is a figure which shows the example of an entry of the object relationship table in Example 2. FIG. 10 is a flowchart of a mixed reality space management program in Embodiment 3. 12 is a flowchart of a mixed reality space image generation program according to a third embodiment. It is a format figure of the object relation table in Example 3. FIG. 10 is a diagram illustrating an image example of a mixed reality image display screen according to a third embodiment. FIG. 10 is a diagram illustrating an image example of a paper medium output of a mixed reality image according to a third embodiment.

Claims (4)

An image processing apparatus connected to a printing means for printing an image and capable of outputting an image to be printed to the printing means,
Obtaining means for obtaining a real space image from the imaging means;
Generating means for generating a virtual space image representing a virtual space based on the position and orientation of the imaging means;
A synthesis unit that synthesizes the real space image acquired by the acquisition unit and the virtual space image generated by the generation unit, and generates a mixed reality space image;
Display means for displaying the mixed reality image;
An output unit for outputting the mixed reality space image to the printing unit in order to print the mixed reality image by the printing unit;
The generating means includes
When the display means displays the mixed reality image, the virtual space image is generated by a first method,
When the output means outputs the mixed reality space image, a method of performing a detailed influence on a virtual object in the virtual space by reflection of light than the first method, a virtual space than the first method The virtual space may be any one of a method of increasing the number of bits of numerical accuracy for image generation and a method of setting the vertical and horizontal pixel size of the virtual space image within the range of the product of the resolution of the printing means and the print size. Generate an image,
The synthesis means includes
Set a cylindrical overlapping determination area for each of the observer and the virtual object,
When the output means outputs the mixed reality space image and the image is taken from the imaging means, the superimposition determination area set for the virtual object is superimposed on the superimposition determination area set for the observer. The virtual object is moved to a position not superimposed on the observer, and the mixed reality space image is generated,
An image processing apparatus , wherein when the display means displays the mixed reality space image, the mixed reality space image is generated without moving the virtual object . An acquisition step in which the acquisition means acquires a real space image from the imaging means;
A generating step of generating a virtual space image representing a virtual space based on the position and orientation of the imaging unit;
A synthesis step of synthesizing the real space image acquired by the acquisition unit and the virtual space image generated by the generation unit to generate a mixed reality space image;
Display means for displaying the mixed reality image; and
An output unit that outputs the mixed reality space image to the printing unit in order to print the mixed reality image by the printing unit;
In the generating step,
When the display means displays the mixed reality image, the virtual space image is generated by a first method,
When the output means outputs the mixed reality space image, a method of performing a detailed influence on a virtual object in the virtual space by reflection of light than the first method, a virtual space than the first method The virtual space may be any one of a method of increasing the number of bits of numerical accuracy for image generation and a method of setting the vertical and horizontal pixel size of the virtual space image within the range of the product of the resolution of the printing means and the print size. Generate an image,
In the synthesis step,
Set a cylindrical overlapping determination area for each of the observer and the virtual object,
When the output means outputs the mixed reality space image, and when the image is picked up from the imaging means, the superimposition determination area set for the virtual object is superimposed on the superimposition determination area set for the observer. The virtual object is moved to a position not superimposed on the observer, and the mixed reality space image is generated,
When the display means displays the mixed reality space image, the mixed reality space image is generated without moving the virtual object. Computer
Obtaining means for obtaining a real space image from the imaging means;
Generating means for generating a virtual space image representing a virtual space based on the position and orientation of the imaging means;
A synthesis unit that synthesizes the real space image acquired by the acquisition unit and the virtual space image generated by the generation unit, and generates a mixed reality space image;
Display means for displaying the mixed reality image;
An output unit that outputs the mixed reality space image to the printing unit in order to print the mixed reality image by a printing unit;
The generating means includes
When the display means displays the mixed reality image, the virtual space image is generated by a first method,
When the output means outputs the mixed reality space image, a method of performing a detailed influence on a virtual object in the virtual space by reflection of light than the first method, a virtual space than the first method The virtual space may be any one of a method of increasing the number of bits of numerical accuracy for image generation and a method of setting the vertical and horizontal pixel size of the virtual space image within the range of the product of the resolution of the printing means and the print size. Generate an image,
The synthesis means includes
Set a cylindrical overlapping determination area for each of the observer and the virtual object,
When the output means outputs the mixed reality space image, and when the image is picked up from the imaging means, the superimposition determination area set for the virtual object is superimposed on the superimposition determination area set for the observer. The virtual object is moved to a position not superimposed on the observer, and the mixed reality space image is generated,
When the said display means displays the said mixed reality space image, the computer program for functioning as an image processing apparatus characterized by producing | generating the said mixed reality space image, without moving the said virtual object. Computer
Obtaining means for obtaining a real space image from the imaging means;
Generating means for generating a virtual space image representing a virtual space based on the position and orientation of the imaging means;
A synthesis unit that synthesizes the real space image acquired by the acquisition unit and the virtual space image generated by the generation unit, and generates a mixed reality space image;
Display means for displaying the mixed reality image;
An output unit that outputs the mixed reality space image to the printing unit in order to print the mixed reality image by a printing unit;
The generating means includes
When the display means displays the mixed reality image, the virtual space image is generated by a first method,
When the output means outputs the mixed reality space image, a method of performing a detailed influence on a virtual object in the virtual space by reflection of light than the first method, a virtual space than the first method The virtual space may be any one of a method of increasing the number of bits of numerical accuracy for image generation and a method of setting the vertical and horizontal pixel size of the virtual space image within the range of the product of the resolution of the printing means and the print size. Generate an image,
The synthesis means includes
Set a cylindrical overlapping determination area for each of the observer and the virtual object,
When the output means outputs the mixed reality space image and the image is taken from the imaging means, the superimposition determination area set for the virtual object is superimposed on the superimposition determination area set for the observer. The virtual object is moved to a position not superimposed on the observer, and the mixed reality space image is generated,
When the display means displays the mixed reality space image, a computer storing a computer program for functioning as an image processing apparatus that generates the mixed reality space image without moving the virtual object A readable recording medium.

Images