JP2006127004A - Cg presenting device, its method and its program and cg display system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a CG presenting device for presenting an image viewed from an arbitrary direction desired by a viewer on a display device for computer graphics having the format of three-dimensional information. <P>SOLUTION: This CG presenting device 10 is provided with a viewer visual point position measuring means 12 for measuring the visual point position of a viewer, a replica position measuring means 13a for measuring the position of a replica simulating the shape of the object to be viewed, a replica posture measuring means 13b for measuring the direction of a replica, a CG area calculating means 14a for calculating the region of CG of the object to be viewed to be presented to a display device 30 based on the measured position and direction of the replica and the predetermined size of the replica and a CG image generating means 14b for generating a CG image from CG data stored in the CG data storing means 11 based on the region of the CG calculated by the CG area calculating means 14a, and for outputting the CG image to the display device 30. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention presents a CG image of the appreciation object on a display device in accordance with the movement of a replica (model, doll) imitating the shape of the appreciation object created as computer graphics (hereinafter abbreviated as CG) data. The present invention relates to a CG presentation device, a method and a program thereof, and a CG display system.

  Conventionally, when information such as digital images and text data processed by a computer is visualized and presented (exhibited) to general viewers in museums, libraries, etc., the viewers are computers such as mice, keyboards, joysticks, touch panels, etc. A general method is to operate a device for viewing the visualized information.

  In recent years, virtual reality technology has been used to allow viewers to wear sensors, head-mounted displays, etc. to build a virtual world in the computer. There is also a method of appreciating the visualized information by a device (virtual reality system) that can interact with the computer.

However, in order to view information by operating a computer device, the viewer needs to know in advance how to operate the computer device as knowledge. In addition, a viewer who does not know the operation method cannot appreciate the information.
In addition, when viewing information using virtual reality technology, the viewer must wear a sensor, head-mounted display, etc., and an assistant who needs to wear it properly or wearing glasses There are problems such as narrowing the field of view of the person. Furthermore, there is a problem that information cannot be browsed easily or casually for the viewer.

Therefore, recently, an exhibition system that presents information (images) using a magnifying glass type interface device (virtual scope) has been proposed (see, for example, Patent Document 1 and Non-Patent Document 1).
In this virtual scope display system, the viewer moves the virtual scope over the panel that shows the entire image, enlarges the image corresponding to the position of the virtual scope, and presents it on the display device. is there. Thus, the image can be enlarged and presented by an intuitive and easy operation of the viewer.
JP 2002-152620 A (paragraphs 0022 to 0061, FIGS. 1 to 3) Fukaya, Mitsumine, Inoue, “Virtual Scope”, Proceedings of the 7th Image Sensing Symposium, pp. 211-214, 2001

However, in conventional exhibition systems using virtual scopes, it is assumed that the image to be viewed is two-dimensional information, so basically, when computer graphics that represent the shape of three-dimensional information are presented. There is a problem that only images viewed from one direction can be handled.
To solve this problem, in a conventional display system using a virtual scope, the panel showing the entire image is a three-dimensional model, and the viewer holds the virtual scope over the model, so that computer graphics of three-dimensional information is displayed. It is possible to enlarge and present images from multiple directions.

However, in this case, when looking at the back side of the model (the surface on the display device side), the viewer is positioned between the model and the display device, and the viewer operates the virtual scope and the image on the display device. There is a problem that operability is extremely difficult because viewing must be performed in the direction of the line of sight.
It is also possible to operate the virtual scope and view the image on the display device in the same line of sight by rotating the three-dimensional model, but the operation of rotating the model can be performed by a device such as a joystick. It is very difficult to do. Furthermore, since the viewer needs to perform the rotation of the model and the operation of the virtual scope together, there is a problem that the viewer cannot concentrate on the image displayed on the display device.

  The present invention has been made in order to solve the above-described problems. The shape of the three-dimensional information can be obtained without imposing a burden of wearing a sensor or the like on the viewer, and without using a virtual scope. To provide a CG presentation device, a method and a program thereof, and a CG display system capable of presenting on a display device an image viewed from an arbitrary direction desired by a viewer with respect to computer graphics. With the goal.

  The present invention has been developed to achieve the above-mentioned object. First, the CG presentation device according to claim 1 is a viewer of a replica imitating the shape of an object to be viewed created as CG data. A CG presentation device that changes the CG position, size, and orientation of the object to be viewed in association with the position and orientation relative to the display object, and presents the display device with CG data storage means and viewer viewpoint position measurement means And a replica position measuring means, a replica attitude measuring means, a CG region calculating means, and a CG image generating means.

  According to such a configuration, the CG presentation device measures the viewer's viewpoint position with respect to a predetermined reference position by the viewer's viewpoint position measurement unit. Here, the reference position is a position serving as a reference for specifying the three-dimensional space, and for example, a specific position of the display device installed at the fixed position is set as the reference position. In addition, in order to measure the viewer's viewpoint position, for example, the viewer can wear a head-mounted display or a spectacle equipped with a magnetic sensor, etc. Although the viewpoint position may be measured from the image by a stereo matching method, a method of measuring the viewpoint position from an image obtained by photographing the viewer is preferable in order not to burden the viewer.

  Further, the CG presentation device measures the position of the replica with respect to the reference position by the replica position measuring unit. For example, the replica position measuring unit may measure the position of the replica by an ultrasonic sensor, or may measure the position of the replica by a stereo matching method from a plurality of images obtained by photographing the replica.

Further, the CG presentation device measures the orientation of the replica with respect to the display surface of the display device by the replica attitude measuring means. For example, the replica attitude measuring means may measure the orientation of the replica by a gyro sensor, or affix a pattern such as a barcode on a plurality of replicas and detect the pattern from an image obtained by photographing the replica, thereby replicas May be measured.
As a result, the CG presentation device has a positional relationship between the display device installed at the fixed position, the viewer's viewpoint position, and the replica position in the three-dimensional space based on the reference position. Yes, it is possible to specify which direction the replica is facing in the three-dimensional space.

Therefore, the CG presentation device uses the CG region calculation means to display an object to be displayed on the display device when viewed from the viewpoint position of the viewer based on the position and orientation of the replica and a predetermined replica size. The CG area is calculated.
Thus, the CG area in which the replica is projected onto the display area of the display device is specified from the viewpoint position of the viewer.

The CG presentation device generates a CG image from the CG data stored in the CG data storage unit based on the CG area calculated by the CG region calculation unit by the CG image generation unit and outputs the CG image to the display device. To do.
As a result, a CG image in accordance with the direction of the replica is displayed on the display device in the line-of-sight direction where the viewer is looking at the replica.

  The CG presentation device according to claim 2 is the CG presentation device according to claim 1, wherein the CG region calculation unit directs the viewer's viewpoint position to the center of the display region of the display device. The virtual camera position is calculated, and the CG area of the viewing object is calculated in a quadrangular pyramid including the four corners of the shooting angle of view of the virtual camera including the display area and the virtual camera position.

  According to such a configuration, the CG presentation device assumes that the viewer's viewpoint position is the virtual camera position by the CG region calculation means, and the virtual camera always faces the center in the display region of the display device. . As a result, the viewer's actual line-of-sight direction is not affected. Then, the CG area of the object to be viewed is specified by the quadrangular pyramid including the four corners of the shooting angle of view of the virtual camera including the display area and the virtual camera position, that is, the viewer's viewpoint position.

  The CG presentation device according to claim 3 is the CG presentation device according to claim 1 or 2, wherein the viewer viewpoint position measurement means includes a standing position measurement means, a parietal part detection means, and a face. The detection unit and the viewpoint height measurement unit are provided.

  According to such a configuration, the CG presentation device includes a foot image captured by the first camera fixed at a predetermined position by the standing position measuring unit and a foot image of the viewer's foot captured in advance. A difference image obtained by taking a difference from the background image corresponding to the image is generated. This difference image is an image of only the foot of the viewer that is different from the background image. Therefore, the standing position measuring means sets a predetermined position of the viewer's foot in the difference image, for example, the tip (toe) as the viewer's standing position.

Furthermore, the CG presentation device corresponds to the viewer image captured by the viewer and the viewer image captured in advance by the second camera fixed at a predetermined position by the top detection unit. A difference image with the background image is generated. This difference image is an image of only the viewer who is different from the background image. Thus, the head top detection means detects the top of the viewer from the image of the viewer alone. Then, the CG presentation device detects the face of the viewer within a predetermined range from the top of the head by the face detection means. In this way, the calculation speed can be increased by narrowing (limiting) the area for detecting the face. Then, the CG presentation device corrects the position of the viewer's eyes by using a predetermined offset value from the position of the center of gravity in the face area detected by the face detection unit by the viewpoint height measurement unit, thereby viewing the reference position relative to the reference position. Measure the height of the person's eyes.
Thereby, the CG presentation device can specify the position of the viewer's eyes, that is, the viewpoint position.

  Furthermore, the CG presentation device according to claim 4 is the CG presentation device according to claim 1 or 2, further comprising a standing position measurement unit, wherein the CG image generation unit is measured by the standing position measurement unit. A CG image having different display contents is generated based on the standing position of the viewer.

  According to such a configuration, the CG presentation device is captured in advance by the standing position measurement unit and the foot image captured by the first camera fixed at a predetermined position and the foot of the viewer. The viewer's standing position with respect to the reference position is measured based on a difference image with a background image corresponding to the foot image. Accordingly, the CG presentation device can sequentially measure the standing position even when the viewer moves back and forth or right and left with respect to the display device and changes the standing position.

  Then, the CG presentation device generates a CG image with different display contents every time the standing position of the viewer measured by the standing position measuring unit moves to a predetermined area by the CG image generating unit. For example, the CG image generation means can change various colors, change the transparency, change the illumination direction for the viewing object, etc. in the CG image of the same viewing object depending on the standing position of the viewer. Switch.

  The CG presentation device according to claim 5 is the CG presentation device according to any one of claims 1 to 4, wherein the CG region calculation means is measured by the viewer viewpoint position measurement means. The CG region of the viewing object is calculated by adding an offset in a predetermined direction to the viewpoint position or the replica position measured by the replica position measuring means.

  According to such a configuration, the CG presenting device has a CG region calculating means in a predetermined direction at the viewpoint position measured by the viewer viewpoint position measuring means or the replica position measured by the replica position measuring means. A predetermined amount (offset) is added to obtain a new viewpoint position or a new replica position. For this reason, the CG area of the viewing object calculated by the CG area calculating means is shifted in the direction in which the offset is added. As a result, the displayed CG image is displayed so as to be shifted upward, for example, from the extension of the line of sight where the viewer is looking at the replica. For this reason, the replica and the CG image seen by the viewer do not completely overlap each other, and the CG image can be visually recognized even when the viewer is looking at the replica.

  Even if this offset is not added, it is possible to visually recognize the CG image by the parallax of the viewer's eyes, but in order to visually recognize the CG image more clearly, in this way, It is preferable to add an offset to the replica position.

  Further, the CG presenting method according to claim 6 is configured such that the position of the CG of the viewing object is linked to the position and orientation of the replica imitating the shape of the viewing object created as CG data with respect to the viewer, A CG presentation method in which the size and orientation are changed and presented on a display device, and includes a measurement step, a CG region calculation step, and a CG image generation step.

According to such a procedure, the CG presentation method measures the viewer's viewpoint position, replica position and orientation with respect to a predetermined reference position in the measurement step.
As a result, the CG presentation method is based on the reference position, and the positional relationship between the display device installed at the fixed position, the viewer's viewpoint position, and the replica position in the three-dimensional space. Yes, it is possible to specify which direction the replica is facing in the three-dimensional space.

Then, the CG presentation method is a CG region calculation step in which a viewing object to be presented on the display device when viewed from the viewpoint position of the viewer based on the position and orientation of the replica and a predetermined replica size. The CG area is calculated.
Thus, the CG area in which the replica is projected onto the display area of the display device is specified from the viewpoint position of the viewer.

In the CG presentation method, in the CG image generation step, a CG image is generated from the CG data stored in the CG data storage means based on the CG area calculated by the CG area calculation means and output to the display device. To do.
As a result, a CG image in accordance with the direction of the replica is displayed on the display device in the line-of-sight direction where the viewer is looking at the replica.

  Further, the CG presentation program according to claim 7, the CG position of the viewing object in association with the position and orientation of the replica imitating the shape of the viewing object created as CG data with respect to the viewer, In order to change the size and orientation and present the image on the display device, the computer functions as a viewer viewpoint position measurement unit, a replica position measurement unit, a replica attitude measurement unit, a CG region calculation unit, and a CG image generation unit. It is characterized by.

According to such a configuration, the CG presentation program measures the viewer's viewpoint position with respect to a predetermined reference position by the viewer's viewpoint position measurement unit. Further, the CG presentation program measures the replica position with respect to the reference position by the replica position measuring means.
Further, the CG presentation program measures the orientation of the replica with respect to the display surface of the display device by the replica attitude measuring means.
As a result, the CG presentation program determines the positional relationship between the display device installed at the fixed position, the viewer's viewpoint position, and the replica position in the three-dimensional space based on the reference position. Yes, it is possible to specify which direction the replica is facing in the three-dimensional space.

Therefore, the CG presentation program uses the CG area calculation means to display an object to be displayed on the display device when viewed from the viewer's viewpoint position based on the position and orientation of the replica and a predetermined replica size. The CG area is calculated.
Thus, the CG area in which the replica is projected onto the display area of the display device is specified from the viewpoint position of the viewer.

The CG presentation program generates a CG image from the CG data stored in the CG data storage means based on the CG area calculated by the CG area calculation means by the CG image generation means and outputs the CG image to the display device. To do.
As a result, a CG image in accordance with the direction of the replica is displayed on the display device in the line-of-sight direction where the viewer is looking at the replica.

  Furthermore, the CG display system according to claim 8 is configured such that the position of the CG of the viewing object is linked to the position and orientation of the replica imitating the shape of the viewing object created as CG data with respect to the viewer. A CG display system that changes the size and orientation and presents it on a display device, comprising a CG data storage means, a replica, an ultrasonic transmitter, a camera, a viewer viewpoint position measuring means, and a replica position measuring means And a replica attitude measuring means, a CG region calculating means, a CG image generating means, and a display means.

  According to such a configuration, the CG display system measures the viewer's viewpoint position with respect to a predetermined reference position from the viewer's image captured by the camera by the viewer viewpoint position measuring unit.

  Then, the CG display system measures the position of the replica with respect to the reference position by the replica position measuring unit. That is, by receiving ultrasonic waves transmitted by a plurality of ultrasonic transmitters arranged around the replica with an ultrasonic receiver provided in the replica, and detecting a phase difference between the plurality of ultrasonic waves, The replica position measuring means measures the position of the replica in the three-dimensional space.

Further, the CG display system measures the orientation of the replica with respect to the display surface of the display device by the replica attitude measuring means. That is, the replica attitude measuring means measures the orientation of the replica with respect to the display surface of the display device based on attitude information such as roll, pitch, yaw detected by the gyro sensor provided in the replica.
As a result, the CG display system has a positional relationship between the display device installed at the fixed position, the viewer's viewpoint position, and the replica position in the three-dimensional space based on the reference position. Yes, it is possible to specify which direction the replica is facing in the three-dimensional space.

Then, the CG display system uses the CG region calculation means to display an object to be displayed on the display device when viewed from the viewer's viewpoint position based on the position and orientation of the replica and a predetermined replica size. The CG area is calculated.
Then, the CG display system generates a CG image from the CG data stored in the CG data storage unit based on the CG region calculated by the CG region calculation unit by the CG image generation unit, and the display device The CG image is displayed.

According to the invention of claim 1, claim 6, claim 7 or claim 8, it is possible to change the direction of the replica by simply changing the direction of the replica without placing a burden of wearing a sensor or the like on the viewer. CG of three-dimensional information in the same direction can be presented on the display device as an image viewed from an arbitrary direction that the viewer wants to see. In addition, for the viewer, the CG image corresponding to the shape of the viewer looking at the replica can be visually recognized on the display device by a simple operation of changing the orientation of the replica. You can view the moving image.
Further, according to the present invention, since the CG image is presented on the extension of the line of sight when the viewer looks at the replica, the viewer can view the CG image with a natural line of sight.

  According to the second aspect of the present invention, a CG image corresponding to the shape of the viewer looking at the replica can be presented on the display device. Also, since the viewing direction of the viewer looking at the display device always faces the center of the display area, the movement of the CG image is presented in a stable state regardless of the viewer's actual viewing direction. can do.

  According to the third aspect of the present invention, when measuring the position (viewpoint position) of the viewer's eyes, the face is not detected directly from the image captured by the camera, but by the background subtraction method. The area for detection is limited, the face is detected in the area, and the position of the eyes is specified. Therefore, the amount of calculation for detecting the face is reduced, and the viewpoint position of the viewer can be measured at high speed. it can.

  According to the fourth aspect of the present invention, CG images having different display contents can be generated depending on the position of the viewer, so that the display mode can be easily switched. Further, even for the same viewing object, CG images can be presented in various display modes depending on the position of the viewer, so that more information can be presented to the viewer. .

  According to the fifth aspect of the present invention, since the CG image is presented at a position shifted from the extension of the line of sight where the viewer is looking at the replica, the overlap between the replica seen by the viewer and the CG image is displayed. Therefore, the CG image becomes easier to see.

Embodiments of the present invention will be described below with reference to the drawings.
[Configuration of CG display system]
First, the configuration of a CG display system used in a museum or the like will be described with reference to FIG. FIG. 1 is a schematic diagram showing the overall configuration of a CG display system according to the present invention. The CG display system 1 presents a CG image corresponding to the shape of the replica 20 viewed by the viewer M on the display device 30 by moving the replica 20 held by the viewer M. Here, the CG display system 1 includes a CG presentation device 10, a replica 20, a display device 30, an ultrasonic transmitter 40, and a camera 50.

  The CG presentation device 10 measures the position and orientation of the replica 20, measures the viewpoint position of the viewer M, and displays the CG image of the replica 20 corresponding to the real image of the replica 20 viewed from the viewpoint position. To be presented. The configuration of the CG presentation device 10 will be described in detail later.

  The replica 20 is a model imitating the shape of an object to be viewed (appreciation target) created as CG data. Here, the replica 20 includes an ultrasonic receiver 21 and a gyro sensor 22.

  The ultrasonic receiver 21 detects a phase difference in the frequency of ultrasonic waves transmitted from the plurality of ultrasonic transmitters 40 and a deviation in arrival timing. The phase difference of the frequency detected by this ultrasonic receiver 21 is transmitted to the CG presentation device 10 via a cable (not shown) or the like. As a result, the CG presentation device 10 can measure the position of the replica 20 in the virtual three-dimensional space.

  The gyro sensor 22 detects the direction of the replica 20. For example, a triaxial gyro sensor is used as the gyro sensor 22 to detect the triaxial inclination of roll, pitch and yaw. The direction (for example, roll, pitch, and yaw) detected by the gyro sensor 22 is transmitted to the CG presentation device 10 via a cable (not shown) or the like. As a result, the CG presentation device 10 can detect the attitude of the replica 20 in the virtual three-dimensional space.

  The display device 30 is a general display device for displaying a CG image, and is, for example, a plasma display panel (PDP), a liquid crystal display (LCD), or the like.

  The ultrasonic transmitter 40 transmits ultrasonic waves. The ultrasonic transmitters 40 are installed at a plurality of positions around the display device 30, the back side of the viewer M, pillars supporting the ceiling, and the like, and the frequencies of ultrasonic waves transmitted by each are different.

The camera 50 photographs the viewer M. An image captured by the camera 50 is transmitted to the CG presentation device 10 in order to measure the position (viewpoint position) of the viewer M. Here, the cameras 50 are respectively installed at the upper center (camera 50 _U ), left side (camera 50 _L ), and right side (camera 50 _R ), which are predetermined positions of the display device 30.

The camera 50 _U (first camera) captures an image for measuring the standing position of the viewer M. Specifically, the camera 50 _U photographs the foot of the viewer M. In this way, in the CG presentation device 10, the difference between the image (foot image) captured by the camera 50 _U and the background image captured in advance is taken, and the standing position of the viewer M is determined from the display device 30. It is possible to measure whether the display device 30 is far away from the display device 30 or not.

The camera 50 _L and the camera 50 _R (second camera) take an image for measuring the height of the viewpoint position of the viewer M. Specifically, the camera 50 _L and the camera 50 _R photograph at least the upper body including the face of the viewer M. As a result, the CG presentation device 10 takes the difference between the image (viewer image) captured by the camera 50 _L (or the camera 50 _R ) and the background image captured in advance, thereby reducing the region of the viewer M. A face is detected from the extracted image based on a statistical feature amount of the face, and the height of the viewpoint position (eye position) is measured.

Here, the reason why the two cameras 50 _L and 50 _R are used to capture an image for measuring the height of the viewpoint position is to increase the movement range of the viewer M. For example, when the CG presentation device 10 determines that the standing position of the viewer M is shifted to the left from the image captured by the camera 50 _U , the height of the viewpoint position is determined from the image captured by the camera 50 _L. measured, if it is determined that the shift to the right, the camera 50 _R is to measure the height of the viewpoint position from the captured image. If it is not necessary to widen the range of movement of the viewer M, one camera for measuring the height of the viewpoint position may be used, for example, installed at the lower center of the display device 30.
In this way, by using the image captured by the camera 50 _U and the image captured by the camera 50 _L or the camera 50 _R , the CG presentation device 10 can view the viewer M in the virtual three-dimensional space. The position can be measured.

[Configuration of CG presentation device]
Next, the configuration of the CG presentation device will be described with reference to FIG. 2 (refer to FIG. 1 as appropriate). FIG. 2 is a block diagram showing the configuration of the CG presentation device according to the present invention. As shown in FIG. 2, the CG presentation device 10 includes a CG data storage unit 11, a viewer viewpoint position measurement unit 12 as a control unit CO, a replica position / posture measurement unit 13, and a CG generation unit 14. ing.

  The CG data storage unit 11 stores CG data of an object to be viewed in advance, and is a general storage device such as a hard disk. The CG data stored in the CG data storage unit 11 is, for example, volume data in a general volume rendering method. This volume data is data in which a three-dimensional space in which an object to be viewed exists is divided into cubes called voxels (volume elements), and information such as color information, transparency, and reflectance of each voxel is made into a three-dimensional array. is there.

The viewer viewpoint position measuring unit 12 measures the current viewpoint position of the viewer in the virtual three-dimensional space. The viewpoint position measured by the viewer viewpoint position measuring unit 12 is output to the CG generating unit 14. Here, the viewer viewpoint position measuring means 12 uses, as shown in FIG. 3, the intersection point between the perpendicular line drawn from the center of the display screen of the display device 30 to the floor surface and the floor surface as the origin O (reference position). Virtual three-dimensional coordinates passing through the origin O and having the X axis parallel to the horizontal direction of the display screen, the Y axis parallel to the vertical direction of the display screen, and the Z axis orthogonal to the XY plane consisting of the X axis and the Y axis The position in space shall be measured.
Here, the viewer viewpoint position measuring unit 12 includes a standing position measuring unit 12a, a crown detecting unit 12b, a face detecting unit 12c, and a viewpoint height measuring unit 12d.

The standing position measuring means 12a detects the standing position of the viewer standing on the floor. Here, standing position measuring means 12a inputs the image captured by the camera 50 _U, that the photographed image (feet image), by taking the difference between the background image taken by the absence previously viewer The position of the viewer (for example, toe) is detected.

  Here, with reference to FIG. 4 (refer to FIG. 2 as appropriate), a method for detecting the standing position of the viewer will be described. 4A and 4B are explanatory diagrams for explaining a method of detecting the standing position of the viewer. FIG. 4A shows a background image, FIG. 4B shows a foot image, and FIG. 4C shows a difference image.

First, the standing position measuring unit 12a stores a background image as shown in FIG. 4A in advance in a storage unit (not shown). Here, the coordinates on the XZ plane shown in FIG. 3 are associated in advance with the four points (La, Lb, Lc, Ld) of the background image. The standing position measuring unit 12a includes a background image shown in FIG. 4 (a), by taking the difference between the captured feet image by the camera 50 _U shown in FIG. 4 (b), in FIG. 4 (c) The difference image shown is generated. This difference image is an image of only the viewer's foot. Therefore, the standing position measuring means 12a sets the viewer's standing position at the tip of the foot or the middle position between the tips of both feet. Note that the coordinates of the standing position can be relatively calculated from the coordinates of the four points (La, Lb, Lc, Ld) associated with the background image. As a result, it is possible to obtain (Cx, Cz) which are the X and Z coordinates of the standing position of the viewer on the XZ plane shown in FIG.
Returning to FIG. 2, the description of the configuration of the CG presentation device 10 is continued.

  Note that the viewer's standing position measured by the standing position measuring means 12a is not only for measuring the viewer's viewpoint position, but also for the CG image generating means 14b described later, the display mode of the CG image to be displayed. It can also be used as a reference for switching. The display mode switching is performed when the CG image generation unit 14b is described.

Parietal detecting means 12b from the camera 50 _L or the camera 50 _R Pictures taken at (viewer image), it detects the viewer, is to identify (detect) the position of the top of the viewer. Here, top portion detecting unit 12b, the standing position of the viewer which is measured at the standing position measuring unit 12a is, if the image is shifted left relative to the display device 30, from the image the camera 50 _L have taken detecting a viewer, if the image is shifted right with respect to the display device 30, a camera 50 _R detects the viewer from the captured image. When the viewer's standing position is at the center with respect to the display device 30 (X coordinate is “0”), the viewer is selected from one of the images taken by the camera 50 _L or the camera 50 _R. Shall be detected.

Here, top portion detecting unit 12b, the camera 50 _L or camera 50 inputs the image captured by the _R, and the photographed image (viewer images) of the background image captured in the absence previously viewer The viewer is detected by taking the difference, and the highest pixel position is set as the top of the viewer.

  The face detection unit 12c detects the region of the viewer's face within a predetermined range from the top of the viewer detected by the top detection unit 12b. Here, a general detection technique can be used to detect the face of the viewer from the image. For example, the face detection unit 12c holds face feature quantities (feature quantities such as eyes and mouth) in advance as learning data, and determines an area most similar to the feature quantity as a face. In this way, if the face detection range is made small, the detection speed can be increased.

  The viewpoint height measuring means 12d is for specifying eyes from the face area detected by the face detecting means 12c and measuring the height of the viewer's eyes with respect to the reference position. Here, the viewpoint height measuring unit 12d calculates the centroid position of the face area detected by the face detecting unit 12c, and corrects the position based on a predetermined offset value from the centroid position. Position. The coordinates of the eye position are obtained by associating the background image with the height at a certain Z coordinate (see FIG. 3) in advance by associating several points (for example, four points) with each other. Calculate relatively. However, since the viewer moves in the Z direction (the optical axis direction of the camera), the height of the viewer's eyes is corrected based on the viewer's standing position measured by the standing position measuring unit 12a.

Specifically, the Z coordinate of the viewer's standing position measured by the standing position measuring means 12a at the eye height (= y ₁ ) at a certain Z coordinate value (= z ₁ ) associated with the background image. If the value is z _2, the real eye height y _2, it is calculated by the following equation (1).
y ₂ = (z ₂ / z ₁ ) y ₁ (1)

  As described above, the viewer viewpoint position measuring unit 12 is configured such that, in the three-dimensional space (X, Y, Z) shown in FIG. 3, the viewer's standing position measured by the standing position measuring unit 12a and the X coordinate and Z The position specified by the coordinates and the Y coordinate which is the eye height measured by the viewpoint height measuring means 12d is set as the viewer's viewpoint position.

Here, the viewer viewpoint position measuring means 12 measures the standing position of the viewer from the image (foot image) taken by the camera 50 _U , and the image (viewing) taken by the camera 50 _L or the camera 50 _R. The viewer's viewpoint position is measured by measuring the height of the viewpoint position of the viewer M from the viewer image), but the viewpoint position is measured by a stereo matching method from images obtained by photographing the viewer with a plurality of cameras. It is good as well. Alternatively, the viewer may attach a head-mounted display or spectacles to a magnetic sensor, ultrasonic sensor, gyro sensor, etc. to measure the viewpoint position, but the viewer must wear the sensor, etc. It can be said that the embodiment is preferable.

The replica position / orientation measuring means 13 measures the position and orientation of the replica in a virtually set three-dimensional space. The position and orientation of the replica measured by the replica position / posture measuring unit 13 are output to the CG generating unit 14. Note that the replica position / orientation measuring means 13 measures the position and orientation of the replica in the virtual three-dimensional space (X, Y, Z) shown in FIG.
Here, the replica position / posture measuring unit 13 includes a replica position measuring unit 13a and a replica posture measuring unit 13b.

  The replica position measuring means 13a measures the position of the replica 20. Specifically, the replica position measuring unit 13a outputs the phase difference and arrival timing of the ultrasonic waves output from the ultrasonic receivers 40 output from the ultrasonic receiver 21 provided in the replica 20. On the basis of the deviation or the like, the position of the replica 20 (corresponding to the position of the ultrasonic receiver 21) is measured in the three-dimensional space (X, Y, Z) shown in FIG.

  The replica attitude measuring means 13b measures the orientation of the replica 20. Specifically, the replica attitude measurement means 13b outputs attitude information (for example, roll, pitch, and yaw) detected by the gyro sensor 22 output from the gyro sensor 22 provided in the replica 20 to the replica 20. Measure as orientation.

  Here, the replica position measuring means 13 a measures the position of the replica 20 by the ultrasonic receiver 21, and the replica attitude measuring means 13 b measures the direction of the replica 20 by the gyro sensor 22. A pattern such as a code may be pasted, and the position and orientation of the replica 20 may be measured from images captured by a plurality of cameras by a stereo matching method.

  The CG generation unit 14 generates a CG image of the replica 20 and presents it on the display device 30. Here, the CG generation unit 14 includes a CG region calculation unit 14a and a CG image generation unit 14b.

  The CG area calculating unit 14 a is a replica position / posture measuring unit 13 in a virtual area of a quadrangular pyramid composed of the viewer viewpoint position measured by the viewer viewpoint position measuring unit 12 and the four corners of the display area of the display device 30. Based on the position and orientation of the replica 20 measured in step 1, the CG area of the viewing object presented on the display device 30 is calculated. The CG area calculated by the CG area calculating unit 14a is output to the CG image generating unit 14b.

  Note that the CG region calculating unit 14 a may add an offset in a predetermined direction to the position of the replica 20 measured by the replica position / posture measuring unit 13. As a result, the replica 20 and the CG image displayed on the display device 30 are shifted in the viewing direction of the viewer, so that the viewer can easily see the CG image.

  Here, with reference to FIGS. 5 and 6 (refer to FIG. 2 as appropriate), a method in which the CG region calculation unit 14a calculates the CG region of the viewing object to be presented (displayed) on the display device 30 will be described. FIG. 5 is an explanatory diagram for explaining the relationship between the viewpoint position, the replica, and the display device in the virtually set three-dimensional space. FIG. 6 is an explanatory diagram for explaining the relationship between the four corners of the shooting angle of view of the virtual camera included in the display area and the viewpoint position when the viewpoint position is the virtual camera position.

  In FIG. 5, it is assumed that the display area (display screen) of the display device 30 is preset as a virtual display area Dv in the three-dimensional space described with reference to FIG. Here, the width of the virtual display area Dv is 2w and the height is 2t. The coordinates of the center Po of the virtual display area Dv are Po = (0, h, 0), and the coordinates Pa, Pb, Pc, Pd of the four corners (each vertex) of the virtual display area Dv are Pa = (− w , H + t, 0), Pb = (− w, ht, 0), Pc = (w, ht, 0), and Pd = (w, h + t, 0).

Here, it is assumed that the viewpoint position C is C = (Cx, Cy, Cz) and the position R of the replica 20 is R = (Rx, Ry, Rz).
Thereby, the viewpoint position C, the replica 20, and the virtual display area Dv of the display device are specified in the virtual three-dimensional coordinate space.

At this time, as shown in FIG. 6, the CG area calculation unit 14a sets the viewpoint position as the virtual camera (viewpoint) position C, and shoots the virtual camera including (inscribed in) the virtual display area Dv that is the display area of the display device. A quadrangular pyramid (C, Pa ′, Pb ′, Pc ′, Pd ′) composed of four corners (Pa ′, Pb ′, Pc ′, Pd ′) of the field angle Dc and the virtual camera position C is virtually set. . It is assumed that the line-of-sight direction from the virtual camera position C always faces the center Po of the virtual display area Dv.
Then, the CG area calculating unit 14a calculates the CG area of the viewing object included in the quadrangular pyramids (C, Pa ′, Pb ′, Pc ′, Pd ′).
As a result, as shown in the virtual camera video I, the CG area corresponding to the replica 20 is calculated as shown in FIG.

In FIG. 5, when adding an offset to the position of the replica 20, for example, for the position R = (Rx, Ry, Rz) of the replica 20, R = (Rx, Ry + Offset, Rz) and the amount of Offset, By increasing the Y coordinate, a CG area shifted upward in correspondence with this offset is calculated in the virtual display area Dv. This offset may be added to the viewpoint position.
Returning to FIG. 2, the description of the configuration of the CG presentation device 10 is continued.

The CG image generation unit 14 b generates a CG image from the CG data stored in the CG data storage unit 11 based on the CG region calculated by the CG region calculation unit 14 a and outputs the CG image to the display device 30. is there.
The CG image generation means 14b reads CG data corresponding to the CG area calculated by the CG area calculation means 14a from the CG data storage means 11, and, for example, color information, transparency, reflection, set as CG data. A CG image is generated based on the rate or the like.

  Here, it is assumed that the CG image generation unit 14b switches the method of displaying the CG image based on the viewer's standing position measured by the standing position measurement unit 12a. For example, the CG image generation unit 14b switches the display method for the same viewing object depending on the left / right and front / rear standing positions of the viewer with respect to the display device 30.

  Here, with reference to FIG. 7, the display mode switching performed by the CG image generation unit 14b will be described. FIG. 7 is a schematic diagram showing a state in which the display mode is switched depending on the viewer's standing position. FIG. 7A is a case where the viewer's standing position is at the front center with respect to the display device, and FIG. (C) shows the case where the viewer's standing position is on the right side of the display device relative to the display device.

  As shown in FIG. 7A, when the viewer's standing position is the front center with respect to the display device, the CG image generation unit 14b displays an image of only “bone” corresponding to the replica 20 (here, a doll). Is generated as a CG image. Further, as shown in FIG. 7B, when the viewer's standing position is on the left side of the center with respect to the display device, the CG image generation unit 14 b displays the “built-in” image corresponding to the replica 20 CG. Generate as an image. Further, as shown in FIG. 7C, when the viewer's standing position is on the right side of the center with respect to the display device, the CG image generation unit 14b displays the image of only “skin” corresponding to the replica 20 as CG. Generate as an image.

In this example, the viewer's standing position is divided into three regions and the display mode is switched. However, this division is not limited to this.
The switching of the display mode shown in FIG. 7 is realized by changing the transparency of the voxel by the CG image generating unit 14b depending on the position of the viewer when the CG data is volume data in the volume rendering method. be able to.

As described above, the CG presentation device 10 displays the CG image of the replica 20 on the display device 30 ahead of the viewer's line of sight in accordance with the position and orientation of the replica 20 that the viewer has in hand. Can do. Further, since the display mode can be switched depending on the standing position of the viewer, the display method can be changed even for the same viewing object.
The CG presentation device 10 operates according to a CG presentation program that causes a computer to function as each unit of the control unit CO.

  The configuration of the CG presentation device 10 according to the present invention has been described above, but the present invention is not limited to this. For example, the viewer viewpoint position measurement unit 12, the replica position / posture measurement unit 13, and the CG generation unit 14 that are the control unit CO of the CG presentation device 10 can be caused to function by different computers. By distributing the functions in this way, a CG image can be displayed at a high speed in conjunction with the movement of the replica 20.

[Operation of CG presentation device]
Next, the operation of the CG presentation device will be described with reference to FIG. 8 (refer to FIG. 2 as appropriate). FIG. 8 is a flowchart showing the operation of the CG presentation device according to the present invention.

(Measurement step: Viewer viewpoint position measurement step)
First, the CG presentation device 10 inputs an image photographed by the standing position detecting camera 50 _U by the standing position measuring means 12a of the viewer viewpoint position measuring means 12 (step S1). Then, the standing position measuring means 12a detects the toe of the viewer standing on the floor from the image taken in step S1 based on the background difference, and calculates the standing position of the viewer (step S2).

The CG presentation device 10 then captures either one of the images photographed by the left and right cameras (50 _L or 50 _R ) depending on the position of the viewer by the top detection unit 12b of the viewer viewpoint position measurement unit 12. Select and input (step S3). Furthermore, the top of the viewer is detected by taking the difference between the image captured by the selected camera and the background image captured in advance by the top detecting means 12b, and the face detecting means 12c A face is detected by limiting the range in which the face is detected with reference to the top of the head (step S4). Then, the viewpoint height measuring unit 12d specifies the position of the eye from the position of the center of gravity in the face region by using a predetermined offset value, and calculates the eye height (step S5).
As described above, the viewer's viewpoint position in the virtual three-dimensional space is measured based on the standing position of the viewer calculated in step S2 and the eye height calculated in step S5.

(Measurement step: replica position measurement step)
In addition, the CG presentation device 10 uses the ultrasonic wave transmitted from the ultrasonic wave transmitter 40 output from the ultrasonic wave receiver 21 provided in the replica 20 by the replica position measuring unit 13a of the replica position / attitude measuring unit 13. The position of the replica 20 in the virtual three-dimensional space is measured on the basis of the phase difference of the frequency, the arrival timing shift, etc. (step S6).

(Measurement step: replica attitude measurement step)
Further, the CG presentation device 10 measures the direction detected by the gyro sensor 22 output from the gyro sensor 22 provided in the replica 20 as the direction of the replica 20 by the replica attitude measurement unit 13b (step S7). ).
At this stage, the viewer's viewpoint position and the position and orientation of the replica 20 are measured.

(CG region calculation step)
Then, the CG presentation device 10 uses the CG region calculation unit 14a of the CG generation unit 14 to create a replica in a virtual region of a quadrangular pyramid including the viewer's viewpoint position and the four corners of the display region of the display device 30 set in advance. Based on the position and orientation of 20, the CG area of the viewing object presented on the display device 30 is calculated (step S8).

(CG image generation step)
Here, the display mode of the CG image to be displayed is changed based on the X-axis value of the viewer's standing position on the XZ plane shown in FIG.
Therefore, the CG presentation device 10 determines the standing position of the viewer calculated in step S2 by the CG image generation unit 14b (step S9).
When the X coordinate of the viewer's standing position is within a range of two predetermined threshold values (x1, x2) (x1 ≦ X ≦ x2), the CG image generating unit 14b sets the first display mode as the first display mode. For example, an image of only “bone” corresponding to the replica 20 (here, a doll) is generated as a CG image (step S10).

Further, when the X coordinate of the viewer's standing position is smaller than the threshold value x1 (X <x1), the CG image generation unit 14b uses, for example, an “internal” image corresponding to the replica 20 as the second display mode. Is generated as a CG image (step S11).
Furthermore, when the X coordinate of the viewer's standing position is larger than the threshold value x2 (x2 <X), the CG image generation unit 14b uses, for example, an image of only “skin” corresponding to the replica 20 as the third display mode. Is generated as a CG image (step S12).

Then, the CG image generation unit 14b outputs the CG image generated in step S10, S11, or S12 to the display device 30, and presents the CG image to the viewer (step S13).
By executing the above operations continuously, a CG image is presented in real time in conjunction with the operation of the replica 20 of the viewer.
Note that the viewer viewpoint position measurement step (steps S1 to S5), the replica position measurement step (step S6), and the replica attitude measurement step (step S7) do not necessarily have to be performed in this order.

  Through the above operation, the CG presentation device 10 measures the viewer's viewpoint position and the position and orientation of the replica 20, and the display device 30 arranged in the line-of-sight direction in which the viewer looks at the replica 20, A CG image linked to the movement of the replica 20 can be presented.

[Usage example of CG presentation device]
In the above description, the form in which the CG presentation device according to the present invention is used as a CG display system used in a museum or the like has been described. Here, referring to FIG. 9 (refer to FIG. 2 as appropriate), the present invention is described. Another usage pattern of the CG presentation device according to the present invention will be described.
FIG. 9 is a conceptual diagram showing an example in which the replica is a remote controller for operating the television device. In FIG. 9, the viewer pushes the remote control (replica) 20 for operating the television device Tv in front of the television device Tv, so that the operation method and buttons of the buttons of the remote control 20 are displayed on the screen of the television device Tv. This shows an example of displaying the functions and the like.
In this case, the CG presentation device 10 measures the viewer's viewpoint position and the remote control (replica) 20 position with reference to the preset position of the television device Tv.

  The CG presentation device 10 changes the image to be presented to the television device Tv according to the channel being viewed and the operation mode (for example, operation of the video device). That is, in the CG image generation unit 14b, the image to be presented to the television device Tv is changed according to the current operation mode or the like. For example, when the operation mode is “video device operation”, the function of each button of the remote control (“rewind”, “fast forward”, “stop”, “take out”, etc.) is displayed on the screen of the television device Tv. Is displayed together with the remote control image. As a result, the buttons of the remote controller can be gathered to a minimum, and the currently operable operations can be visualized, so that a simple and intuitive operation is possible.

It is a schematic diagram which shows the whole structure of the CG display system which concerns on this invention. It is a block diagram which shows the structure of the CG presentation apparatus which concerns on this invention. It is a conceptual diagram which shows the virtual three-dimensional coordinate space on the basis of a display apparatus. It is explanatory drawing for demonstrating the method of detecting a viewer's standing position, Comprising: (a) is a background image, (b) is a foot image, (c) has shown the difference image. It is explanatory drawing for demonstrating the relationship between a viewpoint position, a replica, and a display apparatus. It is explanatory drawing for demonstrating the relationship between the four corners of the display area contained in the imaging | photography angle of view of a virtual camera, and a viewpoint position when a viewpoint position is made into a virtual camera position. FIG. 5 is a schematic diagram showing a state in which the display mode is switched depending on the position of the viewer, where (a) is the center of the viewer with respect to the display device, and (b) is the position of the viewer. (C) shows the case where the viewer's standing position is on the right side of the center with respect to the display device. It is a flowchart which shows operation | movement of the CG presentation apparatus based on this invention. It is a conceptual diagram which shows the example which used the replica as the remote control which operates a television apparatus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 CG display system 10 CG presentation apparatus 11 CG data storage means 12 Viewer viewpoint position measurement means 12a Standing position measurement means 12b Head top detection means 12c Face detection means 12d View height measurement means 13 Replica position / posture measurement means 13a Replica position measurement Means 13b Replica attitude measurement means 14 CG generation means 14a CG region calculation means 14b CG image generation means 20 Replica 21 Ultrasonic receiver 22 Gyro sensor 30 Display device 40 Ultrasonic transmitter 50 Camera 50 _U camera (first camera)
50 _L , 50 _R camera (second camera)

Claims (8)

The position, size, and orientation of the CG of the viewing object are changed in association with the position and orientation of the replica of the viewing object created as CG data in relation to the viewer and presented to the display device. A CG presentation device,
CG data storage means for storing CG data of the viewing object;
Viewer viewpoint position measuring means for measuring the viewer's viewpoint position with respect to a predetermined reference position;
Replica position measuring means for measuring the position of the replica with respect to the reference position;
Replica attitude measurement means for measuring the orientation of the replica relative to the display surface of the display device;
Based on the measured position and orientation of the replica and a predetermined size of the replica, a CG region of the viewing object to be presented to the display device when viewed from the viewpoint position of the viewer CG region calculating means for calculating,
CG image generation means for generating a CG image from the CG data stored in the CG data storage means based on the CG area calculated by the CG area calculation means, and outputting the CG image to the display device;
A CG presentation device comprising:   The CG area calculation means sets the viewer's viewpoint position as a virtual camera position facing the center of the display area of the display device, and includes four corners of the shooting angle of view of the virtual camera including the display area, The CG presentation apparatus according to claim 1, wherein a CG area of the object to be viewed is calculated in a quadrangular pyramid including a camera position. The viewer viewpoint position measuring means includes:
Based on a difference image between a foot image taken by the first camera fixed at a predetermined position and taken from the viewer's foot and a background image corresponding to the foot image taken beforehand, Standing position measuring means for measuring the position of the viewer relative to a reference position;
Based on a difference image between a viewer image captured by the viewer and a background image corresponding to the viewer image captured in advance, which is captured by a second camera fixed at a predetermined position. A top detection means for detecting the top of the viewer;
A face detection means for detecting a region of the viewer's face within a predetermined range from the top of the head detected by the top of the head detection means;
Viewpoint height measuring means for identifying the position of the viewer's eyes from the position of the center of gravity in the face area detected by the face detecting means, and measuring the height of the eyes of the viewer with respect to the reference position;
The CG presentation device according to claim 1, wherein the CG presentation device is provided. Based on a difference image between a foot image taken by the first camera fixed at a predetermined position and taken from the viewer's foot and a background image corresponding to the foot image taken beforehand, Comprising standing position measuring means for measuring the position of the viewer with respect to a reference position;
3. The CG according to claim 1, wherein the CG image generating unit generates a CG image having different display contents based on a viewer's standing position measured by the standing position measuring unit. Presentation device.   The CG area calculating means adds an offset in a predetermined direction to the viewpoint position measured by the viewer viewpoint position measuring means or the replica position measured by the replica position measuring means, and The CG presentation device according to claim 1, wherein the CG region of the object is calculated. The position, size, and orientation of the CG of the viewing object are changed in association with the position and orientation of the replica of the viewing object created as CG data in relation to the viewer and presented to the display device. CG presentation method,
A measuring step for measuring the viewer's viewpoint position relative to a predetermined reference position, the position and orientation of the replica;
Based on the position and orientation of the replica measured in this measurement step and the predetermined size of the replica, the viewing object to be presented on the display device when viewed from the viewer's viewpoint position A CG region calculating step for calculating a CG region;
A CG image generation step of generating a CG image from the CG data stored in the CG data storage means based on the CG region calculated in the CG region calculation step and outputting the CG image to the display device;
CG presentation method characterized by including. The position, size, and orientation of the CG of the viewing object are changed in association with the position and orientation of the replica of the viewing object created as CG data in relation to the viewer and presented to the display device. For the computer,
Viewer viewpoint position measuring means for measuring the viewer's viewpoint position with respect to a predetermined reference position;
Replica position measuring means for measuring the position of the replica with respect to the reference position;
Replica attitude measuring means for measuring the orientation of the replica with respect to the display surface of the display device;
CG area calculation means for calculating a CG area of the viewing object to be presented to the display device based on the measured position and orientation of the replica and a predetermined size of the replica;
CG image generation means for generating a CG image from the CG data stored in the CG data storage means based on the CG area calculated by the CG area calculation means, and outputting the CG image to the display device;
A CG presentation program that functions as a computer program. The position, size, and orientation of the CG of the viewing object are changed in association with the position and orientation of the replica of the viewing object created as CG data in relation to the viewer and presented to the display device. A CG display system,
CG data storage means for storing CG data of the viewing object;
The replica including an ultrasonic receiver and a gyro sensor;
A plurality of ultrasonic transmitters arranged around the replica;
A camera for photographing the viewer;
Viewer viewpoint position measuring means for measuring the viewer's viewpoint position with respect to a predetermined reference position based on an image photographed by the camera;
Replica position measuring means for measuring the position of the replica with respect to the reference position, based on the phase of the ultrasonic wave transmitted by the ultrasonic transmitter, received by the ultrasonic receiver;
Replica attitude measuring means for measuring the orientation of the replica relative to the display surface of the display device based on attitude information output from the gyro sensor;
Based on the measured position and orientation of the replica and a predetermined size of the replica, a CG region of the viewing object to be presented to the display device when viewed from the viewpoint position of the viewer CG region calculating means for calculating,
CG image generation means for generating a CG image from CG data stored in the CG data storage means based on the CG area calculated by the CG area calculation means;
The display means for displaying the CG image generated by the CG image generating means;
A CG display system comprising:

Images