JP3526897B2 - Image display device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is used in an apparatus for capturing an image of a moving object and adding a special effect to the image for editing. Further, it is used for a device for displaying a virtual object seen by a moving observer by projection or the like. The present invention is
For example, when shooting a player in sports or the like, this is used when a virtual object added after editing can be recognized by the player who is the subject while shooting. Also, when shooting an image edited by a special effect such as a movie, it is used when an actor of a subject acts while watching a virtual object added by editing.

[0002]

2. Description of the Related Art The applicant photographs a moving object while measuring three-dimensional spatial coordinates (hereinafter referred to as three-dimensional information), and simultaneously handles an image of the target object, three-dimensional information, and time to perform special image editing. , A technology for generating an image taken by a virtual imaging means (hereinafter referred to as a virtual camera) existing in another space-time (Japanese Patent Application No. 5-221363, Japanese Patent Application No. 5-309473, unpublished at the time of application). With this technique, it is possible to display and compare a record in another space-time and an image in the space-time.

[0003]

However, in sports practice, it is possible to compare a plurality of players by image editing when comparing with a rival player or a record player to be compared. , The athletes who are actually moving are not able to compete while the athletes to compare are not next to each other, and there is no real feeling that they are actually lining up. There's a problem.

Further, even when the pace is adjusted during the competition, the athlete who is the subject has to make a judgment while photographing.

Similarly, in the case of producing images using special effects such as television and movies, the actor is in a studio without anything,
I had to act by imagining what would be added after editing.

In the above case, in order to allow the player or actor who is the object object of the subject to recognize the virtual object added after the editing, the virtual reality technique can be used to enter the image. Had to wear a special device such as goggles for displaying the image.

An object of the present invention is to provide an image display device which allows an observer who is a subject to operate while recognizing a virtual object added after editing without mounting such a special device on the subject. To do.

Another object of the present invention is to display an actually photographed moving image of another person in a form visible to an observer, so that the observer can compete side by side by looking at the image of another person. It is an object of the present invention to provide an image display device that is capable of providing the viewer with a sense of reality.

Another object of the present invention is to display an image viewed from the observer's three-dimensional position so that the observer can enter into the virtual image and perform a natural acting or action. It is to provide a display device.

[0010]

The first aspect of the present invention is to:
The present invention relates to an image display device for displaying an image seen by an observer to an observer, and measuring means for measuring the three-dimensional information of the observer, and the observer based on the measured three-dimensional information of the observer. Image generation means for generating an image equivalent to the image seen from the viewpoint position of the observer, and the generated image can be seen from the viewpoint position of the observer when displayed by the image display means at a position different from the observer. Image conversion means for converting into an image is provided.

A second aspect of the present invention relates to an image display device for displaying a photographed image to an observer using a photographed image.
Alternatively, the image pickup means for picking up an image of a plurality of moving objects and outputting the three-dimensional information together with the three-dimensional information, and the image pickup means for an image obtained by projecting an image of the moving object on one conversion surface in the space in which the moving object moves. Image conversion means for converting into an image equivalent to an image taken by virtual imaging means existing at different spatial positions or different time axes, and image display means for displaying the image-converted image of the moving object In the image display device, the image display means is an image display means for displaying an image of the moving object to a moving observer, and includes a measuring means for measuring three-dimensional information of the observer, and the image conversion is performed. The means converts the image of the moving object displayed to the observer into an image equivalent to the image seen by the observer, based on the three-dimensional information of the observer measured by the measuring means. One conversion means and second conversion means for converting the converted image into an image in which the image projected from the projection means at a position different from the observer is equivalent to the image seen by the observer. Is characterized by.

The measuring means for measuring the three-dimensional information of the observer can be a device for taking an image of the observer and measuring the three-dimensional information of the observer.

Further, the image display means can be composed of one or a plurality of projectors provided with a platform and a screen projected by the projectors.

The image display means may be one or a plurality of display devices.

Further, the plurality of projectors can be switched according to the position of the observer.

Further, the image display means includes a movable screen and is provided with a measuring means for measuring the three-dimensional information of this screen, and the image converting means displays based on the three-dimensional information of the screen measured by this measuring means. Image conversion can be performed.

Further, it is possible to provide means for generating a sound effect sound based on the three-dimensional information of the observer.

The image displayed to the observer may be an image of one or a plurality of target objects.

Further, the image displayed to the observer may be an image generated by computer graphics.

[0020]

It is possible to measure three-dimensional information of a moving target object, output it together with the image, and convert it into an image of a virtual camera existing in an arbitrary space-time based on this three-dimensional information. If it is the eyes of an observer who moves this virtual camera, if the observer's three-dimensional information is known, the virtual camera's three-dimensional information is determined. As the reverse procedure of converting to the image of the projector, the image of the moving object that should be seen by the observer (virtual camera) is generated from the image of the target object and projected or displayed in the reverse of the shooting. You can The present invention applies the technology of the virtual camera previously proposed by the applicant to generate and display an image viewed by a moving observer from other spatiotemporal images based on the three-dimensional information of the observer. It is a thing.

First, when a moving object is imaged, a high-precision pan head and a camera with a high-precision zoom lens are used to measure the position of the target object, and the direction of the camera and the captured image are used to target the object. We are looking for 3D information about an object. By using this platform and lens, if a moving image projector (for example, a video projector) is attached instead of the camera, it is possible to project an image taken by the camera exactly opposite to that taken by the camera. Further, in order to obtain the image of the virtual camera in the editing operation, the same conversion as that of projecting the image on the virtual conversion surface and re-shooting could be performed in real time.

Using these techniques, if the position of the screen and the position of the projector head in the space and the position of the virtual object of the image to be projected in the space are known, the screen in the space can be seen from the direction of the observer. It can be converted into an image in which a virtual object can be seen and can be actually projected. The projected image is calculated with the three-dimensional information obtained in real time, converted in real time, and projected as a moving image, so that the observer sees that the virtual object is also moving.

By such an editing operation and a display such as projection, it is possible to give the observer a sense of realism.

[0024]

【Example】

(First Embodiment) FIG. 1 is a diagram for explaining the principle of the present invention, and FIG. 2 shows the positional relationship when this FIG. 1 is viewed from above.

Reference numeral 1 is an observer, and a screen 2 is arranged between virtual objects 3 viewed from the observer 1, and a projection image 5 is projected on the screen 2 from a projection center 6 of the projector. This allows the observer 1 to recognize the virtual object 3 from the projected image 4 of the virtual object projected on the screen 2.

The image to be projected is the three-dimensional information of the observer 1,
If the position of the screen 2 and the position of the projection center 6 of the projector (projector) and the three-dimensional information of the virtual object are known, the conversion parameter can be determined.

The feature of the first aspect of the present invention is to measure the three-dimensional information of the observer 1 in an image display device equipped with a screen and a projector as an image display means for displaying an image corresponding to the observer's viewpoint position. Measuring means, an image generating means for generating an image equivalent to an image seen from the observer's viewpoint position based on the measured three-dimensional information of the observer 1, and the generated image for the observer. Image conversion means for converting into an image viewed from the viewpoint of the observer when displayed by the image display means at another position.

FIG. 3 is for explaining the principle of this projected image conversion. First, when a virtual object is photographed in another space-time, the camera head (MH) 22
A virtual object (moving target object) is photographed together with dimensional information. Image conversion is performed using the viewpoint of the observer 1 as a virtual camera. This image conversion has been described in Japanese Patent Application No. 5-221363 or Japanese Patent Application No. 5-309473. Although the spatial position of the virtual camera according to this prior application could be arbitrarily set, in the case of the present embodiment,
It depends on the movement of the observer 1, and the viewpoint of the observer 1 is treated as the viewpoint of the virtual camera. In this case, the observer 1 is additionally provided with a three-dimensional information measuring device in order to determine its viewpoint. For example, the observer 1 uses a camera head that uses the same high-precision pan head and zoom lens as the one that captured the virtual object. Three-dimensional information is being measured.

When this observer 1 is a virtual camera, an image obtained by photographing the target object 21 with the camera head 22 is taken by the observer 1
Conversion is performed using the conversion surface 23 so as to obtain an image when viewed from above. This image conversion is performed in Japanese Patent Application No. 5-2213 related to the prior application.
63 or Japanese Patent Application No. 5-309473. By this conversion, an image of the target object 21 as viewed by an observer is obtained. Next, by projecting the obtained image on the screen 2 with the observer 1 as the center, an image 4 on the screen 2 is obtained. Next, this obtained image 4
Is converted into a projected image 5 using the projection center 6 of the projector. In this case, the screen 2 serves as a second conversion surface for converting the image captured by the observer 1 into the projection image 5 of the projector.

By performing image conversion in three stages in this way, a projected image 5 to be projected by the projector can be obtained. In the actual deformation, the deformation of projection from surface to surface is repeated, and the image conversion is coordinate conversion by a conversion matrix, the principle of which is described in detail in Japanese Patent Application No. 5-309473. This image conversion can be projected on the screen 2 as a moving image by performing it in units of fields.

When the virtual object 3 is generated by computer graphics, the image 4 on the screen when the virtual object 3 is viewed from the observer 1 can be calculated quite accurately by calculation. The projection image 5 and the three-dimensional data (posture and angle of view) of the projector can be converted by one conversion using the screen as the conversion surface.

On the other hand, the projector is mounted on a camera platform whose posture can be controlled with an accuracy that can be used for three-dimensional information measurement, and the angle of view can also be accurately controlled. The projected image can be updated on a field-by-field basis like a camera by using a cathode ray tube type projection system such as a video projector or a liquid crystal projector. If such a projector is used, the screen 2 can be obtained by obtaining the three-dimensional information and its conversion parameters in field units.
It can be projected as a movie on top.

The position of the observer (player) is determined by the camera head that photographs the player. In addition, even if a virtual object is composed by computer graphics or an image taken by using a camera head that measures three-dimensional information is used, the three-dimensional information is always attached to the image. Therefore, if the parameters such as the position and orientation of the screen 2 in the three-dimensional space and the position of the projector are known, the conversion calculation of the projected image can be performed. How to obtain the position of the screen 2 and the parameters of the projector will be described later.

The athlete, the observer, is also the subject.
The image obtained by photographing the player with the measurement camera head is edited in the same manner as the previously proposed moving image processing apparatus.
At this time, information on the virtual object is also added. This state is shown in FIGS. 4, 5, and 6.

In FIGS. 4 to 6, a screen 42 is provided on a straight course, a player 41 who is an observer runs, a virtual player image of a comparison target seen by the player is projected from a projector, and the player 41 is taken by a camera. Head MH1, MH
Shoot at 2. The shooting is performed as shown in FIG. Player 4
1 runs along the screen 42. The athlete is then tracked by the two measurement camera heads MH1 and MH2,
Three-dimensional position measurement and image capturing are performed. On the other hand, an image of another player, which is a virtual object, is projected on the screen 42 from the projector head PH. The projected position, size, or the image itself of this image is calculated by the position of the player 41 being measured, and when viewed from the player 41, the virtual object is projected so as to appear at the assumed position. The image of the player captured by the measurement camera head is basically an image of only the player as the target object, as shown in FIG. However, the edited image includes not only the image of the player as shown in FIG. 6, but also the image of another player that is a virtual object. Also, a background can be added if necessary. This editing operation is also performed based on the three-dimensional data of the player and the three-dimensional data of the virtual object. The image of another player on the screen, which is a virtual object, is calculated based on the player's position, so even if the virtual object does not move, it will be projected if the observer player moves. The position and shape of the image change. As a result, the position of the virtual object is not the position of the screen, even though it is projected on the screen when seen from the player,
It feels like you are at the position of the virtual object you set. Therefore, the positional relationship between the screen and the player may be anywhere as long as the player can see the projected screen image.

FIG. 7 shows a block diagram of an image display device for projecting a virtual object on the screen 42 corresponding to the movement of the player 41 shown in FIG.

In the block diagram of this image display device,
The configurations of the camera heads 71 and 72, the three-dimensional operation unit 73, the image editing unit 74, and the background database 75 having the background image given to the image editing unit 74 and the configuration of the virtual camera controller 76 giving the data of the virtual camera are described above. Japanese Patent Application 5-30947
The configuration is the same as that of 3.

Characteristic points in FIG. 7 have a virtual object database 77 having an image of the virtual object for projecting the virtual object shown in the lower part of the drawing and its three-dimensional information, and three-dimensional information of the screen. A screen database 78, an image conversion unit 80 for converting an image of a virtual object image by a conversion parameter of the projector, a projector controller 79 for controlling a projector head based on three-dimensional information of an observer and a virtual object,
The projector head 81 projects the image of the virtual object converted by the control output of the projector controller 79 onto the screen. The projector head 81 includes a projector 82, a lens 83, and a platform motor 8 for controlling a platform on which the projector 82 is mounted.
4. A lens motor 85 for controlling the lens 83 having a high-precision zoom function, a pan head motor 84, and a servo circuit 86 for servo-controlling the lens motor 85 are included.

Further, this image display device can give a sound effect, and outputs a sound database 87 and sound control parameters for controlling the sound to be outputted according to the three-dimensional information of the observer and the three-dimensional information of the virtual object. A sound controller 88, a digital signal processor (DSP) 89 that generates a sound effect to be output based on the sound control parameter, an amplifier 90 that amplifies and outputs the output of the DSP 89, and a speaker 91 are provided.

The operation will be described.

The three-dimensional position of the player 41 is calculated by the measurement camera heads 71 and 72 and the three-dimensional calculation unit 73. In this example, two measuring heads are used to measure the three-dimensional position of the player, but depending on the property of the target object, one may be used by using the constraining plane, or three or more may be used. An image is taken along with the measurement of the three-dimensional data. The captured image and three-dimensional data are edited by the image editing unit 74. Regarding this editing operation, Japanese Patent Application No. 5-3094
Since the image conversion in the image editing in the image editing unit at 73 has been described, it is omitted here.

The three-dimensional data of the player 41 is also used to calculate a virtual object. The image of the virtual object and the three-dimensional data are obtained by accessing the virtual object database 77. This data may be created with an appropriate physics model and computer graphics, but in this example we use data from another player. This virtual object database 7
7, the image can be taken out by three-dimensional data of the virtual object by designating the elapsed time from the start signal, for example. The three-dimensional data of the screen on which the virtual object is projected is measured in advance and recorded in the screen database 78 as screen data. When the screen is fixed, this data becomes a table of fixed values, but when the screen also moves, like the virtual object database, for example, three-dimensional data of the screen is obtained based on the elapsed time from the start signal. It is designed to be used.

Based on these, the projector controller 79 calculates the projection position and controls the projector head 81. The conversion parameter of the image is obtained, the image conversion unit 80 is controlled to convert the image from the virtual object database 77, and the image is projected by the projector 82. A problem with this calculation is the time lag. A player's 3D position is usually obtained a few fields behind. In addition, there is a delay of several fields in the database access of the projected image, the image conversion and the projection. There are slow movements where the time is not a problem, but even normal human movements feel a shift. Therefore, a feedforward element is required for control. First, since the virtual object database is accessed in the time from the start signal, it can be performed prior to the actual start signal. Furthermore, the player's three-dimensional data is calculated in advance by predicting the player's movement. The prediction of player movement is
It's easy to just run like the example in Figure 4, but
In the case of complicated movements, it may be necessary to devise a physical model. Also, the delay time can be reduced by using a special camera and a special video projector to increase the field period. In this case, since virtual object data is basically managed in absolute time and three-dimensional data is managed, calculation can be performed without a problem even if the field cycle is different, but the virtual object data also has a higher image point at a high-speed field cycle. Is desirable.

The projector head 81 basically replaces the camera of the measurement camera head with a video projector. Video projectors (projectors 82) that project images include those that use an ultra-high brightness CRT and those that use a liquid crystal shutter, but if the image to be projected can be updated in the field cycle and the required brightness is obtained. Basically, there are no restrictions on the method, but in order to improve the dynamic characteristics, a method that is light in weight and strong against vibration is preferable.

The image of the video projector is projected on the screen by the lens 83. This lens requires precise control of the angle of view. If the resolution of the video projector itself is sufficient and the brightness is sufficient, the lens may be fixed to the maximum required angle of view. Even in that case, the angle of view must be accurately calculated, and it is more convenient to be able to control the angle of view by remote control. When projecting a color image, a plurality of video projectors and a plurality of lenses may be combined. In that case, it is necessary to match the characteristics of the lenses well so that color misregistration does not occur. In this example, a liquid crystal projector is used to mix colors by using an optical system inside the projector and one projection lens. The lens control has a focus in addition to the angle of view, but since the three-dimensional data of the screen and the three-dimensional data of the projector head are known, the distance to the screen can be calculated. The lens 83 is driven by the lens motor 85 based on this value. Since the take an angle to the screen and the projector would distances are different in the image, it is better to have a large optical system of depth of field.

These projector 82 and lens 83 are mounted on a pan head whose angle can be precisely controlled. As for the rotation of the image, if the image conversion is performed, this platform needs only to have two degrees of freedom, and the same head as the measuring head can be used. In this example, a pan head using two direct drive motors is used in consideration of dynamic characteristics and accuracy. The platform and the projector are assembled with a servo system so as to have a designated posture in accordance with an instruction from the projector controller 79. If there is sufficient dynamic characteristics, there is no difference between the instruction from the controller 79 and the actual posture of the platform,
There is usually a gap. In order to improve the accuracy, it is necessary to consider this and use the actual posture value of the projector head 81 instead of the instruction value of the controller 79 for the calculation of the image conversion parameters and the like. Further, when the resolution of the projector 82 is sufficient, by setting the angle of view wider than the required angle of view, the movement of the image of the virtual object on the screen is not limited to the movement of the projector head, but also in the projected image. It can also be replaced by movement. By doing so, the dynamic characteristics required for the projector head 81 can be relaxed.

By using the above system, the player, who is the subject, can recognize the virtual object, but in this example, the acoustic effect is added in order to further enhance the sense of reality. Sound effects not only enhance the sense of presence, but are also important for athletes, such as the starter's voice as a starting signal or the sound of a pistol. In this example, in addition to that, a sense of presence is added. Each sound that is a source of a sound effect obtained by sampling an actual sound or electrically synthesizing the sound is stored in the sound database 87. This includes the voice of the starter and the cheers of the venue. It is also possible to include data specific to a player, such as the voice of cheering for that player. Since the player can see the 3D position of the player, he / she can control that the cheer becomes louder as he approaches the goal. The sound controller 88 performs this control. Based on the three-dimensional data of the player and the three-dimensional data of another player, which is a virtual object, it is possible to know the overtaking and so on. In that case, the voice of cheering the player is added or the volume of cheering is changed. The digital signal processor (DSP) 89 performs such an operation. A plurality of amplifiers 90 and a plurality of speakers 91 are connected to the DSP 89,
In addition to volume and sound quality control, it is possible to control the sound field using multiple speakers. By controlling the sound field, the direction of sound can be controlled independently of the position of the speaker. By controlling the direction of the sound as the player moves, it is possible to realistically reproduce the passage in front of the cheering seat and the movement of the player's breathing sound when overtaking another player. Also, although there is a screen near the player, controlling the sound field can reduce the influence of sound reflection by the screen, and can be useful for reducing the presence of the screen even slightly.

In this system, the position of the player, who is the observer, is always measured on the subject, so that the surrounding conditions can be changed according to the position. Therefore, in addition to images and sounds, stimuli that can be recognized by the observer, such as the flow of wind, tilting the floor to change the acceleration, and changing the temperature and lighting according to the observer's position,
Furthermore, the presence can be heightened.

(Second Embodiment) When there are a plurality of virtual objects, there is a method of arranging as many projector heads as there are virtual objects. However, virtual objects that are viewed in the same direction by an observer are projected by the projector head. They can be added together by image conversion processing. The image of each virtual object is converted based on the three-dimensional data. At the same time, since there is three-dimensional data of each virtual object, the front-rear relationship when viewed from the observer can be calculated. By superimposing the images converted according to the calculated context, a plurality of target objects can be taken as one projector image. When there are a plurality of virtual objects, the degree of overlap changes with the movement of the observer even if the virtual objects are stationary as shown in FIG. In FIG. 8, three virtual objects are projected.
In this figure, the screen 2 and the projector head 93 are stationary. The virtual objects 94, 95, 96 are projected on the screen 2 as having a positional relationship as shown in this figure.
Although not shown, the observer 1 measures the three-dimensional data with the measuring head. Needless to say, the projected image is synthesized based on the three-dimensional data of the observer 1. When the observer 1 is in the position A, the image projected on the screen is as shown in FIG. 9A. The virtual object 95 cannot be seen because it is hidden by the virtual object 94. In addition, the distance between the virtual object and the observer is large, so the whole is small. When the observer moves to the position B, the screen image becomes as shown in FIG. 9B. At the position A, the hidden virtual object 95 becomes visible. Further, when the observer moves to the position of C, the image is shown in FIG. 9C.
become that way. This time, the virtual object 96 is hidden, and the distance to the observer becomes shorter and the virtual object 96 becomes larger. These are plane images because they create an image of the virtual object on the screen based on the position of the observer. However, for the observer, the appearance changes when the user's position changes, giving the viewer a three-dimensional feeling. Just as you walk around in the room, you get the same feeling as the outside view seen through the window changes depending on the viewing position.

In FIG. 8, the projected image is a part of the screen, but it is possible to project it in a wide range by projecting with a projector having a sufficient resolution at a wide angle of view or by using a plurality of projectors. Become. This allows many virtual objects to be placed. Virtual objects can include both moving objects and stationary objects such as landscapes. This allows the image on the entire screen to be changed at the position of the observer. Furthermore, by enclosing the moving range of the observer with a screen, it is possible to combine all the surrounding environments seen by the observer with an image.

By making the field of view of the measuring camera head visible to the whole body of the observer and attaching markers to the joints of the hands and feet, the head, etc., not only the position of the observer but also the posture, the direction of the fingers, the head, etc. Three-dimensional data such as the orientation of can be obtained at the same time. In this case, the measurement camera head may need to shoot from a plurality of directions because it may be hidden. In this way, the position and orientation of the observer can be measured in real time, so that virtual reality can be realized without wearing goggles or a special device such as a data suit.

(Third Embodiment) As the observer moves,
When moving the image of the virtual object projected on the screen,
A shadow may be formed depending on the relationship between the observer, the screen and the projector head. In the case of the arrangement shown in FIG. 10, when the image is projected from the projector head 102, the observer 1
Will be reflected in the projected image. A shadow cannot be formed by projecting from the back surface of the screen 2 as shown in FIGS. 4 and 8, but it requires a large space. Further, if the projector head is moved in parallel as shown in FIG. 11, a relatively small space is required, but the parallel movement mechanism that can detect the position with the accuracy required for image conversion becomes large-scale and complicated. Further, the observer 1 needs to measure the position with one or a plurality of measurement camera heads 103. Therefore, in order to use the space effectively, it is necessary to arrange the measurement camera head and the projector head on the same side of the screen.

In this embodiment, a plurality of projector heads are used to project one screen image. When moving the projector, it is necessary to know its position and orientation with the accuracy required for image conversion in the field cycle. Even the parallel movement type as shown in FIG. 11 can be used if the position and orientation can be understood, but the scale of the apparatus becomes large. In this example, a polar coordinate type projector head having two rotation axes as shown in FIG. 12 is used. Video projector 1
21 is a tilt shaft motor 123 and a pan shaft motor 124.
You can change the direction with two motors. Since the video projector is heavier than the measurement camera head, it is necessary to fix the base 125 fairly firmly. In this example, the video projector itself is rotated, but the rotating portion can be lightened by using a mirror or the like. Also in this example 2
Although the optical axis of the projection lens 122 comes to the intersection of the two rotation axes, if the deviation does not change due to rotation even if it is deviated, it can be corrected by considering the deviation when calculating the image conversion.

In FIG. 10, such two polar coordinate type projector heads 101 and 102 are used. Now, when the projector head 102 is projected, a shadow of an observer is formed, but the projector head 101 can project an image without forming a shadow. In this way, by using a plurality of projector heads, it is possible to always project the image of the virtual object regardless of the position of the observer.

When a plurality of projector heads are used, switching becomes necessary. Since all three-dimensional data necessary for projection including the position of the observer have been measured, it is possible to calculate in what state the projection is obstructed.
In this example, since the observer creates a shadow, the projector head may be switched depending on the relationship between the observer's position and the projection position. Also, when the pillars of the building are in the way, they can be switched depending on the angle of the projector head. The simplest method is to switch from the image of one projector head to the image of the other projector head on a field-by-field basis when the switching condition is satisfied. Since each projector head is a video projector, there is no problem in outputting or stopping an image in field units, and switching can be easily performed. On the other hand, since it is a mechanical mechanism that determines the projection position and the angle of view, there are restrictions on the movement. Therefore, even for the projector head that is not projecting an image, the mechanical control needs to be continuously moved for a certain period of time before and after switching so that the projection position and the field angle match at the time of switching. When the switching condition is determined based on the three-dimensional data of the target object, there are inevitable effects of error and noise. When the moving speed of the target object is low, or when there is a lot of noise in the three-dimensional measurement, a plurality of image switching may be repeated and the appearance may not be good. In that case, it is advisable to add hysteresis to the switching conditions. For example, a condition for switching from an image of a certain projector head to an image of another projector head and a condition for switching the image from the other projector head are given a range to prevent switching from flapping in the middle.

When the projector heads for switching are separated from each other, the shape of the projection range greatly differs as shown in FIG. Even if the projection position is the same, the range is 131 when projected from the right direction, and 132 when projected from the left direction. When there is an image on the full screen of the projector, the shape of this projection range changes greatly at the moment of switching, which makes it unnatural. Therefore, when switching is performed, only the common area 133, which is the common area, can be used as the image of the virtual object. Note that this is not the case when a change in the image due to switching is permitted.

When there are few errors in the projection position and the angle of view, the projected image is in the common area, and there is sufficient margin in resolution and color balance is sufficient, it is possible to complete in field units. Even if you switch to, the change in the image is small and not unnatural. However, since the projection position and angle of view are mechanically determined, it is difficult to match them exactly due to servo delays, etc., and there is little margin in resolution, and the projected images differ due to color variations. At that time, if you switch completely, it may be unnatural. In that case, before completely switching, an intermediate image composed of two images is used to switch softly. For example, switching is performed with a feeling like a crossfade that is often performed in image editing. That is, images from a plurality of projectors are temporarily mixed and projected during the switching. Then, the brightness of each projector is changed to switch smoothly.

By the way, there is no problem if the switching condition and the other condition are passed in a short time. However, when the intermediate value portion takes a long time, the switching direction is considered and the spatial condition is considered. At the position, even if the switching condition is in the middle, it is necessary to finish the switching within a certain time. When an observer player runs as shown in FIG. 10, the projected image is required to have high image quality near the goal position, particularly at the start position where the player stands still, while the player moves while running. Therefore, it doesn't matter if the projected image quality is not so good. In this way, the image quality of the image of the virtual object required by the position, movement, and posture of the observer differs. In FIG. 10, therefore, the two projector heads are arranged so that the optical axes of the screen and the projector are close to vertical near the goal and start positions, respectively, so that the best image quality is obtained at the goal and start positions. In addition, the switching is arranged so that the athletes can perform it while running.

(Fourth Embodiment) In the examples so far, the screen handles only a fixed plane beside the observer, but if three-dimensional data necessary for conversion of the projected image can be obtained, the plane can be treated. Need not be, and need not be stationary. Also, if the projected image can be seen by the observer, there is no problem if it is translucent and the back can be seen through, and it need not be solid such as water, smoke, and bubbles in water.

FIG. 14 shows a case where a player runs on an elliptical track, and the screen is semitransparent and has a curved surface. Also in FIG.
Is an example where the floor is a screen. For swimming, the bottom of the pool can be used as a screen for underwater projection. In addition, if the projection does not matter the image quality as long as the pace is known, it can be projected on bubbles on the water surface. If you make a screen that surrounds the observer with a normal material as shown in Fig. 16,
The observer cannot capture the measurement camera head. However, the measurement camera head does not need to use visible light, and for example, a measurement camera head using near infrared rays can be used. If near-infrared rays are transmitted through the screen and visible light is diffusely reflected, an observer can be measured even with such a screen arrangement.

By measuring three-dimensional data such as the position and orientation of the screen, the image can be converted even if the screen is moved. Since the screen needs a certain size, it is very difficult to bring the required accuracy to the specified position when moving it. However, it is not necessary to move to that position, but the three-dimensional data of the screen, so the movement mechanism does not need to be so precise. FIG. 17 shows such an example. Although the screen is mounted on a self-propelled vehicle, the accuracy is not so great, but the three-dimensional data is measured by the measurement camera heads MH1 to MH4 together with the observer. A marker is attached to the corner of the screen to measure the position and the posture thereof. This ensures the accuracy required for image conversion. When moving such a screen, a flat screen is preferable because the resistance of air is large. At that time, the fineness of the mesh and the aperture ratio are selected according to the required image quality.

FIG. 18 shows an example in which the screen is moved together with the observer. A marker is attached to the screen so that its posture can be seen. In this figure, the orientation of the screen and the face of the observer match, but the orientation of the screen and the orientation of the face of the observer do not have to match.
Since it is sufficient to measure each independently, you may hang around without fixing the screen firmly. However, in this case, the posture of the screen and the movement of the observer's face are fairly fast, and the field cycle of the measurement camera for measuring the three-dimensional data is too slow, which is 1/60 seconds of a normal TV camera, so a special camera is required. There are cases.

(Fifth Embodiment) In the above examples, the positions of the measurement camera head and the projector head, the rotation axis, the lens parameters, etc. are known, but these parameters are used for image conversion and three-dimensional data measurement. Greatly affects the accuracy of. When the measurement camera head and the projector head are newly installed, and if their positions are displaced due to some kind of momentum, those parameters must be obtained. Even if the parameters for each head are designed values or can be measured at the factory, if the heads are installed locally, the positional relationship between the heads must be found locally. Therefore, a method for obtaining them will be briefly described. In order to obtain the parameters, the parameters are actually operated and compared with a preset reference to perform calibration.

First, the parameters of the measurement camera head are obtained. The angle of view of the lens, focus, and aperture data should be measured in advance using a jig.
As shown in FIG. 19, a calibration solid whose shape and size are known is placed, and the position and orientation of the solid on a coordinate axis determined in space considering the range of movement of the observer and the measurement range of the measurement camera head. To the measurement camera head controller. First, the operator manually operates the measurement camera head 181 to bring this solid into the field of view of the measurement camera. In that case, it is difficult to operate the focus manually, so that the autofocus control can be performed based on the image captured by the camera, for example, by an algorithm that maximizes the high-frequency component in the center of the visual field. This alleviates the burden on the operator and also provides distance information, although it is quite rough. The spatial coordinates used for calibration can be improved in accuracy by using statistical processing by increasing the number of points beyond the theoretically required number of points. Next, each axis of the measurement camera head automatically rotates while the lens parameters are fixed. The calibration volume is stationary, but it appears to move as the head rotates in the captured image.
From this, the spatial position of the rotation axis of the head itself can be obtained. Regarding the rotation angle, there is no need to calibrate because the encoder attached to the rotation axis has sufficient repeatability. These operations can be performed simultaneously when there are multiple measurement camera heads. To improve accuracy, perform the same operation by changing the angle of view of the lens, change the position of the calibration solid, and if there are multiple measurement camera heads, take a picture of each measurement camera head and rotate from the movement of the marker. Determine the axis.

Next, in the case of a fixed screen, the three-dimensional data of the screen is measured. With a fixed screen as shown in FIG. 20, the screen inevitably becomes large,
It deviates from the desired spatial solid such as a plane. In FIG. 20, it is desired to make the screen 191 flat, but distortion occurs. Therefore, many measurement points 192 are determined so that the shape can be specified, and measurement is performed by the measurement camera head. In the case of this example, 12 points are measured and it is approximated to 5 planes. It is good to attach a marker to the measurement point in order to measure it accurately. At that time, markers can be attached to all points in advance as shown in FIG. In that case, the markers can be automatically distinguished by changing the color of the markers. Also, if a human walks with a marker and touches it at a point to be measured, the automatic tracking function can be used, so that continuous three-dimensional data can be measured.

Next, the parameters of the projector head are obtained. As with the measurement camera head, the lens parameters such as the angle of view and focus are measured in advance.

As shown in FIG. 21, when the projector head itself is in a positional relationship in which the measurement camera head can photograph, the spatial position and rotation axis of the projector head are obtained from the positions and movements of the plurality of markers 201 attached to the projector head. You can also If the projector head cannot be photographed, or if it is possible that the angle is not good and the accuracy is poor, the position of the projector head is calculated by measuring the image projected from the projector head onto the screen. At this time, the focus is determined by adjusting so that the projected image is in focus, or by measuring the distance between the projector head and the screen with a rangefinder.
There is also a method of focusing with a remote control while a human observes the image, but the measurement camera head shoots the image,
It is also possible to control the focus of the projector head while detecting whether or not it is in focus by image processing. The image projected at that time has a pattern suitable for image processing.

When the focus is determined, the next known pattern is projected. For example, FIG. 21 shows an image of five circles (reference numeral 202) like a dice.
Is. These circles have different colors, and they can be automatically distinguished when taken by the measurement camera head.
For distinction, the images may be taken in synchronism with a time shift. Since the spatial position of the projected image (position on the screen surface), the angle of view of the projector head at that time, and the shape of the projected pattern are known, the position of the projector head can be calculated. However, unlike the measurement camera head, this is indirect and not accurate. Therefore, it is necessary to collect a lot of data and perform statistical processing while changing the projection pattern and changing the angle of view. Next, as with the measurement camera head,
Turn the rotary shaft and repeat the same operation. The rotation angle does not need to be measured because the encoder has sufficient accuracy. Statistical processing is indispensable in determining the parameters of the projector head in order to improve accuracy. Therefore, the operation of the operator is reduced by automating the cooperative operation with the measuring camera head.

(Sixth Embodiment) By preparing a plurality of this systems and connecting them by a communication line as shown in FIG. 22, it is possible to compete just like a video conference even if a player does not move.

At this time, the most serious problem is a time delay. It is necessary to minimize the delay of communication lines. Especially for satellite lines, terrestrial lines are desirable because the delay will inevitably increase. Depending on the competition, when the player's movement speed is relatively slow, such as swimming, the player's position information is not noticeable, but the movement is not slow in the image, so it is noticeable.

Since the observer who is actually running does not notice the details, even if there is a slight delay, he / she cannot understand it. However, when the image of the virtual camera is edited, it is necessary to record it once and take into account the delay. is there.

(Seventh Embodiment) As shown in FIG. 23, when shooting a special effect of a movie or a television, it was necessary to act against an actor who had no actor. , It becomes possible to perform more accurate acting because it can recognize virtual objects.

Since the image projected at this time is basically the target of the actor's performance, the screen may be the back or the floor. In addition, it is necessary to perform projection so that it does not get in the way of extracting the actor from the video with a chroma key when editing the video. For example, in the case of a chroma key, it is necessary to project a monochrome image in which the hue and the saturation are similar to those of a blue background when viewed with a camera for shooting, and only the luminance is changed. Further, in addition to projecting the virtual object as it is in the image, the influence on the image can be reduced by reducing only the contour or the range in which the necessary points are known.

When a plurality of actors perform, it is possible to switch to which actor the image should be taken according to the scene. Except when there is a virtual object at the position of the screen, basically it is necessary to consider which actor the virtual object is for because it changes the projection location and shape according to the observer's position. If the image to be projected is simplified and if it really matches the target level, the position of the observer can be displaced only by the position of the projection, and by carefully devising the screen or a set instead of the screen It can be projected.

If multiple actors are distant and need to perform more accurately, then as many projection images as actors are needed regardless of the number of virtual objects. When changing the color does not affect the captured image, the actors distinguish by using different colors to distinguish virtual objects that act against them. If the colors cannot be used, a pattern is used as a mark on the projected image so that they can be distinguished.

(Eighth Embodiment) In the above examples, the projected image appears to the observer as a planar image. If the observer can wear special glasses or goggles,
By making the projected image a stereoscopic image as shown in FIG. 24, the sense of reality can be further enhanced. Further, as shown in FIG. 25, if a system for showing different images to the left and right eyes can be used and different images can be provided to a plurality of observers, a system for a plurality of observers can be used. This goggle is a device that allows different images to be displayed to both eyes by, for example, using a liquid crystal shutter to temporally divide the image and further divide the image by color. As a result, a virtual stereoscopic image can be viewed while freely moving with the observer.

Similarly, different images can be shown to a plurality of observers. Of course, in the case of multiple observers, the position of each observer must be measured, but if multiple observers can obtain their respective positions by mechanical coupling, such as riding on a vehicle, the object to be measured. Can reduce the number of.

In the examples of FIGS. 24 and 25, the measurement camera head for measuring the observer wearing the stereoscopic glasses is not shown, but the position and orientation of the observer are shown as in the above-described embodiments. The three-dimensional image of the virtual object that is being measured is converted based on the measured three-dimensional information.

(Ninth Embodiment) When the size of the image provided to the observer may be relatively small, it is clear by displaying it on a CRT, a liquid crystal, a plasma display or the like instead of projecting it on the screen. A beautiful image can be obtained. Also, by combining a plurality of display devices, it is possible to provide a large image with a joint. For example, in FIG. 26, a display device in which six CRTs are combined is configured, and an image viewed from the position of the observer is displayed on each CRT. In FIG. 26, it is displayed that a jar is on the table. The urn is a virtual object, and the table has the same size and positional relationship as the table on which the display is actually mounted.
The 3D information of the observer is measured by another measuring device, and is displayed on the CRT converted into an image that can be seen from the observer's observation position. Therefore, the observer moves around within the observer's position measurement range. Can be displayed as seen from that position, so if you turn to the back side, you can see the back of the jar. If you dive under the table, you can see the jar seen from below. CRT
If you handle the joints well, you can get the same feeling as when you look at the jar in the display case.

FIG. 27 shows a case where a movable flat display is used. The orientation of the display automatically changes according to the position of the observer so that the display has the highest image quality. Also, the displayed image is naturally converted as seen from the observer. In this figure, the front suspension of an automobile is displayed as a virtual object. In such a complicated three-dimensional object, it is difficult to see the whole image unless it is seen from various angles, but the observer can observe while observing the desired portion of the virtual object. If there is a movable range of the display device and an observer's position measurement range, it is possible to go around behind or dive down. Further, the magnification of the virtual object can be changed as required, and since it is a moving image display, a moving image like this case can display a moved image, so that an expression that cannot be realized by a hologram is possible.

Alternatively, in the case where the observer moves in front of a multi-display device in which a large number of CRTs are arranged, three-dimensional information such as the observer's position and posture is measured, so that the displayed virtual image is It can be converted into a virtual image that can be viewed by an observer and displayed on the entire CRT device.

Further, when the observer can wear goggles or stereoscopic glasses, or if a display capable of stereoscopic display is used, the display itself can be stereoscopic by binocular vision.

A system similar to the one shown in FIG. 28 in which a marker is attached so that the posture and position of the display device can be recognized even if the display device does not move automatically, and a marker is attached so that the observer can also understand the position and posture. Can be configured. This can be used for output of three-dimensional CAD, for example. When you want to see the side or the back of a solid object, you can see the side or the back by turning the display device by hand with the observer. In conventional CAD, when changing the posture of the displayed object, it was difficult to see it at the desired angle unless it was necessary to give instructions using a keyboard, etc. However, if you do this, you will be in the direction you want to see. If the CRT is turned so that the screen can be moved and the screen can be seen, the image from that angle can be seen, so the system does not require any skill.

[0084]

Since the present invention is constructed as described above, it has the following effects.

(1) Since it is possible to operate while seeing the movement of the target object seen from itself in response to the movement of the observer, in the case of sports, the movement should be natural while recognizing the relationship with the competitor. You can

(2) When the observer as the subject is a performer, the observer can move while looking at the combined images, so that a natural performance can be performed.

(3) The observer can have a sense of reality because he / she can feel as if he / she was in the spot and operated while looking at the image of the actual photographed object.

(4) The sense of presence can be further enhanced by the acoustic effect according to the position, posture, etc. of the observer.

(5) It is possible to give a player the feeling that he or she is playing at the same time alongside the competitor even in a competition that is conducted simultaneously in a remote place.

(6) It is possible to give a player a feeling of competing with a competitor to be compared, which is performed at different time and space.

(7) Since the virtual image can be projected in accordance with the observer's motion, it is possible to generate an image according to the degree that the observer can recognize.

(8) Since the viewing position of the observer can be measured and the image viewed from the viewing position can be displayed on the display device, even if one object is displayed, the display according to the viewing direction is performed. be able to.

(9) Since the projected screen can also be moved, even if the observer makes a complicated movement, an image corresponding thereto can be displayed to the observer.

(10) The observer can wear the device for performing stereoscopic vision and project or display the stereoscopic image so that the observer can operate while recognizing the stereoscopic image.

(11) Since the optimum image can be switched and displayed according to the position viewed by the observer, a natural image for the observer can be displayed.

(12) By using computer graphics to generate a virtual image to be displayed to the observer,
Since the image to be displayed is accurately obtained by calculation, the image generation calculation and the image conversion for converting into the image projected by the projector can be performed only by calculation.

[Brief description of drawings]

FIG. 1 is a diagram showing a relationship between an observer, a projector, and a projected image, which is a principle of the present invention.

FIG. 2 is a view of FIG. 1 seen from above.

[Figure 3] Convert the captured image into an image viewed by an observer,
The figure explaining the principle which projects it on a screen.

FIG. 4 is a diagram illustrating a first embodiment in which a virtual object is displayed on a screen along with a running observer.

FIG. 5 is a diagram showing an image of a viewer.

FIG. 6 is a diagram showing an image in which a virtual moving object and a background are displayed alongside an observer.

FIG. 7 is a diagram showing a block diagram of an image display device according to an embodiment of the present invention.

FIG. 8 is a diagram illustrating a state in which a plurality of virtual objects according to the second embodiment are overlapped and viewed by an observer.

FIG. 9 is a diagram showing an example of a projected image seen by an observer of the second embodiment.

FIG. 10 is a diagram showing an example of projecting a virtual object using a plurality of projectors according to the movement of the observer in the third embodiment.

FIG. 11 is a diagram showing an example of projecting a virtual object by moving the projector according to the movement of the observer in the third embodiment.

FIG. 12 is a diagram showing an outer appearance of a projector head used in an embodiment of the present invention.

FIG. 13 is a diagram showing a projection area generated when two projectors project images in the third embodiment.

FIG. 14 is a diagram showing an example of projecting a virtual object by providing a screen along a course of an ellipse that the observer moves in the fourth embodiment.

FIG. 15 is a diagram showing an example of projecting a virtual object with the plane on which the observer moves according to the fourth embodiment as a screen.

FIG. 16 is a diagram showing an example in which a screen is provided so as to surround the observer of the fourth embodiment, and a virtual object is projected on the screen while measuring the observer through the screen.

FIG. 17 is a diagram showing an example in which a moving mechanism is provided on the screen of the fourth embodiment and a virtual object is projected on the moving screen.

FIG. 18 is a diagram showing an example in which the screen moves with the observer of the fourth embodiment and a virtual object is projected on the moving screen.

FIG. 19 is a diagram for explaining the calibration of the measurement camera head of the fifth embodiment.

FIG. 20 is a diagram showing an example in which three-dimensional information of a fixed screen is measured by the measurement camera head of the fifth embodiment.

FIG. 21 is a diagram showing how the projector head of the fifth embodiment is adjusted.

FIG. 22 is a diagram showing a system for simultaneously displaying viewers at spatially remote locations according to a sixth embodiment.

FIG. 23 is a diagram showing an example of projecting a virtual object in the vicinity of the performer of the seventh embodiment to photograph the performer.

FIG. 24 is a diagram showing an example in which the observer of the eighth embodiment wears stereoscopic glasses.

FIG. 25 is a diagram showing an example in which a plurality of observers of the eighth embodiment wear goggles.

FIG. 26 is a diagram showing an example in which a virtual object is displayed by combining a plurality of CRT devices of the ninth embodiment.

FIG. 27 is a view showing an example in which a virtual object is displayed on the movable flat display according to the ninth embodiment.

FIG. 28 is a diagram showing an example in which a virtual object is displayed on the rotary CRT device of the ninth embodiment.

[Explanation of symbols]

1 observer 2 screen 3 virtual objects Image of 4 virtual objects 5 projected image 6 Projection center 21 target object 23 Conversion surface 41 players 42 screen 71, 72 Measurement camera head 73 Three-dimensional operation unit 74 Image editing unit 75 background database 76 Virtual camera controller 77 Virtual Object Database 78 screen database 79 Projector controller 80 Image conversion unit 81 Projector head 82 Projector 83 lens 84 Pan head motor 85 lens motor 86 Servo circuit 87 Sound Database 88 sound controller 89 Digital Signal Processor 90 amp 91 speaker 93 Projector head 94-96 virtual objects 101, 102 Projector head 103 Measuring camera head 121 video projector 122 Projection lens 123 tilt axis motor 124 Pan axis motor 125 base 131, 132, 133 Projection area 181 Measuring camera head 191 screen 192 measurement points 201 marker

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI H04N 7/18 H04N 7/18 U (56) Reference JP-A-6-28452 (JP, A) JP-A-5-225313 ( JP, JP-A-63-121388 (JP, A) JP-A-63-276092 (JP, A) JP-A-1-120596 (JP, A) JP-A-5-165921 (JP, A) Flat 5-241539 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/262-5/28 G06T 1/00 G06T 17/40 H04N 1/38-1/393 H04N 13/00-17/06

Claims (9)

(57) [Claims] 1. An image of one or a plurality of moving objects is captured.
An image pickup means for outputting together with the three-dimensional information of the position, and an image of this moving object in a space where the moving object moves.
The image projected on one conversion surface is different from the image pickup means.
Virtual imaging hands existing in different spatial positions or different time axes
An image converter that converts an image equivalent to the image captured by the column
An image display that displays the image of the moving object that has been converted into a step
And a means for moving the observer to the moving observer.
Image display means for displaying an image of an object, and a measuring means for measuring three-dimensional information of the position of the observer is provided.
Well, the image conversion means is for the observer measured by the measuring means.
Displayed to the observer based on the three-dimensional position information
And the image of the moving object that the observer can see
First conversion means for converting the image into an equivalent image and a projection hand at a position different from the observer for the converted image.
The image projected from the step is the same as the image seen by the observer, etc.
An image display device , comprising: a second conversion means for converting the image into a valuable image. 2. A meter for measuring three-dimensional information of the position of an observer.
The measuring means captures an image of the observer and three-dimensionally displays the position of the observer.
The image display device according to claim 1, which is a device for measuring information . 3. The image display means is equipped with a pan head 1 or
Projectors with multiple projectors
The image display device according to claim 1, wherein the image display device comprises a screen . 4. The image display means comprises one or a plurality of disks.
The image display apparatus according to claim 1 or 2 wherein a play apparatus. 5. A plurality of projectors are placed at a position of an observer.
The image display device according to claim 3 , wherein the image display device is switched accordingly . 6. The image display means is a movable screen.
Including, and measures the three-dimensional information of the position of this screen
The measuring means is provided, and the image converting means measures with this measuring means.
Image displayed based on the 3D information of the screen
Any of claims 1 to 3 or 5 including means for performing a conversion
The image display apparatus according to any Re. 7. Based on three-dimensional information of the position of the observer,
Sound that produces sound effects that an observer would hear at that location
7. The image display device according to claim 1, further comprising means for synthesizing a field . 8. The image displayed to the observer is 1
The image display device according to any one of claims 1 to 7, which is an image obtained by capturing a plurality of target objects . 9. An image to be displayed to the observer, co
The image display device according to claim 1, wherein the image display device is an image generated by computer graphics .