JP2002245490A - Device and method for image reproduction - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image reproducing device which enables an appreciator to securely grasp a movable area and to provide an image reproducing device or the like which enables the appreciator oneself to freely move in a virtual space without feeling that something is wrong. SOLUTION: At a reproduced image display part 33 of the image reproducing device, an area where an image can be reproduced is clearly shown and when the user tries to exit from the area, a caution is displayed. A position detecting device 36 which detects the position of the appreciator putting on the reproduced image display part 33 detects the appreciator's walking and estimates the movement quantity of the appreciator through the detection. At this time, the position detecting device 36 gathers components of low frequency bands conducted in the body during the walk through a microphone and detects the walk from the sound.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image reproducing apparatus and an image reproducing method which can be used for reproducing, for example, an image taken by a compound eye.

[0002]

2. Description of the Related Art In recent years, with the development of video shooting technology and reproduction technology such as image synthesis, a plurality of video cameras have been used.
It is possible to perform panoramic photography in which images (videos) are photographed in a plurality of directions from one point, and reproduction as if a video camera moves three-dimensionally around a subject during reproduction by photographing one point from a plurality of directions. By using such a so-called compound-eye imaging technique and moving while performing imaging in a plurality of directions from one point in a specific area, a virtual space of the specific area can be generated based on the captured image (data). Then, by using the image data of the virtual space, it is possible to provide a so-called virtual experience as if a viewer moves in a specific area shown by an image in a computer game, an amusement machine, an attraction or the like.

For example, in the case of a computer game or an amusement machine, the viewer can freely move the viewer's position in the virtual space by operating a joystick or a controller.
Then, an image from the viewpoint of the position of the viewer (in the virtual space) determined based on the operation information of the controller or the like is displayed on the monitor. The viewer can freely operate the joystick and controller in the virtual space and watch the displayed image to experience as if they were actually moving in that space. You can. As another method, for example, Glasstron (trade name of Sony Corporation)
The viewer wears a head mounted display such as
It is also conceivable that an image corresponding to the position is displayed on the head-mounted display while the viewer actually moves.

[0004]

However, the above-described conventional techniques have the following problems. First, when a viewer operates a joystick or controller in a computer game or an amusement machine, the viewer can freely move in a virtual space by operating the joystick or controller as desired. Specifically, an image can be displayed only in a specified area that holds data of a captured image. For this reason, if the viewer tries to go out of the specified area with the joystick or controller, the displayed image will remain stopped or return to the start point even if the operation is continued. I have. However, there is a problem that the viewer does not know that he has gone out of the prescribed area until the image stops or returns to the start point. This problem is also common when a viewer actually moves using a head-mounted display. That is, when the viewer goes out of the prescribed area, the viewer does not know that he has gone out of the prescribed area until the displayed image stops or returns to the start point in the same manner as described above. Moreover, in this case, since the viewer is actually moving, there is a problem that if the movement is not stopped, the viewer feels a gap between the image and the actual movement sensation, and feels strange. In addition, in such a form in which the viewer actually moves, it is conceivable that the viewer actually performs a virtual experience while moving in a space such as a gymnasium. In such a case, especially when the actual (gymnasium) space is smaller than the specified area holding the image data, the viewer may hit the wall of the gymnasium even within the specified area. It is possible.

In a computer game or an amusement machine, the position of a viewer in a virtual space can be accurately grasped from information obtained by operating a joystick or a controller. On the other hand, when the viewer actually moves using the head-mounted display, the position of the viewer in the virtual space is determined by moving the viewer, more specifically, the position and orientation of the viewer. Must be detected accurately.
To detect the viewer's position, GPS (Global
It is possible to use a positioning system (global positioning system) or the like, but this is not possible in indoors where radio waves from GPS satellites cannot be received. For this reason, for example, the position of the viewer can only be detected by providing a pressure-sensitive sensor or the like on the floor surface. This is extremely costly and is not practical in many cases. In addition, it is conceivable to use an acceleration sensor like an autonomous navigation mechanism used in a car navigation device. However, in a case where the viewer moves on a vehicle, a certain degree of acceleration is generated. Even if the user moves on foot, it is difficult to accurately detect the acceleration. On the other hand, the direction of the viewer can be detected relatively easily by using a gyro sensor or the like. Therefore, in the conventional method, in a form in which the viewer actually moves,
It is difficult to detect the viewer's movement unless it meets certain conditions, such as when the viewer moves on a vehicle. Otherwise, it is only possible for the viewer to turn his head. At present, it has to be said that it could not be detected.

[0006] The present invention has been made based on such a technical problem, and it is a main object of the present invention to provide an image reproducing apparatus capable of surely grasping a movable area for a viewer. Another object of the present invention is to provide an image reproducing apparatus and an image reproducing method that can reliably detect the movement of a viewer and allow the viewer to freely move in a virtual space without feeling uncomfortable.

[0007]

With this object in mind, in an image reproducing apparatus according to the present invention, a viewer of an image is provided with a display means for displaying an image based on image data stored in a data storage means, such as a head. And displays an image according to the movement of the viewer. For example, at each of a plurality of locations in a predetermined area, shooting is performed in a plurality of directions from one shooting position, and image data of a compound eye image such as a panoramic image is stored in the data storage unit. Then, based on the position of the viewer detected by the position detection means, an image is selected from the images in which the image data is stored in the data storage means, and the selected image is displayed on the display means. In this way, the images displayed on the display means are sequentially switched in accordance with the movement of the viewer's position and the change in the direction. Of course, the present invention can also be applied to a case where an image obtained by performing shooting in one direction from one shooting position, instead of a compound-eye image, is reproduced. Then, when the position detected by the position detecting means deviates from a predetermined range determined by a photographing viewpoint (viewing point at the time of photographing) of the image in which the image data is stored in the data storing means, a caution is output to the viewer. . For example,
From the viewpoint at the time of shooting, there may be a part that cannot be photographed behind the subject, and naturally, even if the viewer tries to look at the back side of the subject during playback, there is no image data, so the display means The image of that part cannot be displayed. In addition, it is not always possible to take an entire image in a predetermined area at the time of shooting. At the time of playback, an image is enlarged or reduced according to the position of the viewer (the position of the viewpoint at the time of playback). In some cases, image synthesis processing or the like is performed. Even in the case of performing such processing, in order to display the processed image without causing the viewer to feel uncomfortable, the position (range of the viewpoint at the time of reproduction with respect to the image obtained at the viewpoint at the time of shooting) ) Must be restricted. In such a case, a predetermined range is determined based on the image capturing viewpoint, and when the position of the viewer during reproduction deviates from the predetermined range (including a time point at which it is predicted to deviate), the viewer is notified of the position. It outputs caution. By receiving this caution, the viewer can avoid deviating from a predetermined range. As the output caution, a mark, a message, or the like may be displayed on the display means, and in addition, a voice message, vibration, or the like may be used. By the way, as the position detecting means, the walking of the viewer is detected by a walking detecting device set for the viewer, and the moving amount of the viewer is determined based on the detected number of steps, walking pitch, walking information such as the stride. It can also be detected.

The image reproducing apparatus according to the present invention is characterized in that an area in which an image can be displayed on the display means is displayed based on the image data stored in the data storage means. Note that this image reproducing device is not only used when the viewer wears the display means and looks at the image while moving,
For example, in a computer game, an amusement machine, or the like, the present invention can also be applied to a case where a viewer operates a joystick or a controller to have an experience of moving in a virtual space formed by displayed images. In particular, when a joystick or a controller is used, the viewer does not actually move, so the position information of the viewer required to display an image includes:
It is not the position of the viewer but the position of the viewpoint of the viewer.

[0009] The image reproducing apparatus of the present invention may also be characterized in that the position of the viewer is detected by detecting the amount of walking movement of the viewer, and an image selected based on the detected position is displayed. good. Here, the position detecting means
Vibration generated when the viewer walks with a microphone is collected, and a change in a signal corresponding to a predetermined frequency or less is analyzed to detect the walker of the viewer, and to detect the walking movement amount of the viewer. good.

The present invention detects the position of a viewer wearing an image viewer, displays the selected image on the viewer based on the detected position of the viewer, and displays the selected image when there is no selected image. Can be regarded as an image reproducing method of outputting a caution for the image. In addition, among vibrations transmitted through the body of the viewer when the viewer walks, 10
A component in a frequency band of 0 Hz or less is detected, the detected component is analyzed, a walk of the viewer is detected, a walk distance of the viewer is estimated, and a position of the viewer is detected based on the walk distance. You may make it.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below in detail based on an embodiment shown in the accompanying drawings. FIG. 1 is a diagram for describing a configuration of a compound-eye image capturing system for capturing an image to be reproduced by the image reproducing system according to the present embodiment. As shown in FIG. 1, the compound-eye image capturing system includes an image capturing unit 10 that captures an image with a compound eye, a dynamic detecting unit 11 that detects dynamic data such as a capturing position and a capturing direction when capturing an image with the image capturing unit 10. An image data storage 12 for storing image data captured by the imaging unit 10 and dynamic data detected by the dynamic detection unit 11 in association with each other, and a control device 13 for controlling the whole.

The image pickup section 10 has a plurality of cameras 15, each of which is a CCD (Charge Coupled Device).
e: charge-coupled device) and outputs a captured image as a digital signal. Note that this camera 15
Captures an image every minute predetermined time similarly to a so-called video camera or the like. FIG. 2 shows a specific example of such an imaging unit 10, for example, a helmet-shaped base 16 set on the head of a photographer, and a plurality of cameras 15 attached to the base 16. And a viewfinder 17 for determining an imaging target. Here, five cameras 15 are provided, and in a state in which the base 16 is set on the head of the photographer, a total of five directions of the front, rear, left, and right of the photographer and upward are photographed simultaneously. It is attached so that it can be done. Note that the imaging unit 10 shown in FIG. 2 is merely an example, and the number of cameras 15, the shooting direction, and the like are not limited thereto. Instead of setting the imaging unit 10 on the head of the photographer, the imaging unit 10 may be set on a movable cart or the like. Further, as shown in FIG. 1, the movement detection unit 11 detects the position of the photographer, that is, the GPS 18,
It has an acceleration sensor 19 and a terrestrial magnetism sensor 20, and also detects the direction of the photographer, that is, the photographing direction,
A gyro sensor 21 is provided. This dynamic detector 11
Although not shown in FIG. 2, the base 16
At a predetermined position.

When photographing is performed by the compound-eye image photographing system having such a configuration, data of an image photographed by each camera 15 of the photographing unit 10 is associated with photographing position and photographing direction movement data obtained by the movement detecting unit 11. And stored in the image data storage 12.

FIG. 3 is a diagram showing a configuration of a compound-eye image reproducing system (image reproducing apparatus) for reproducing an image photographed by the compound-eye image photographing system as described above. As shown in FIG. 3, the compound-eye image reproduction system controls the entire operation of an image data storage (data storage unit) 31 for storing image data transferred from the image data storage 12 shown in FIG. A control device (control means, caution output means) 32 for performing an output process of the image data stored in the image data storage 31 and a synthesizing process of the output image data, and an image based on the image data output from the image data storage 31 And a reproduced image display unit (display means) 33 for reproducing and displaying.

Here, as shown in FIG. 4, a head-mounted display (viewer) 34 used by being set on the viewer's head is used for the reproduced image display section 33. For example, glasstron (trade name, manufactured by Sony Corporation) or the like is suitable for the head-mounted display 34, and a liquid crystal is used for the display element of the reproduced image display unit 33, for example. In addition, this compound eye image reproduction system includes:
A movement detection unit 35 for detecting movement data such as the position and orientation of the viewer is provided. This dynamic detection unit 35
A position detecting device (position detecting means, viewpoint position detecting means, walking detecting device) 36 which is provided integrally with the head mounted display 34 and detects the position of the viewer, and a gyro sensor 37 which detects the direction of the viewer. And a line-of-sight sensor 38 for detecting the direction of the line of sight based on the direction of the viewer. Here, an acceleration sensor or a terrestrial magnetism sensor may be used as the position detection device 36 in the same manner as in the compound-eye image capturing system shown in FIG. 1, but in the present embodiment, the walking of the viewer is detected and the number of steps , The position is detected by detecting the moving amount of the viewer.

FIG. 5 is a diagram for explaining a basic configuration of the position detecting device 36 in the present embodiment. As shown in FIG. 5, the position detecting device 36 includes a microphone 40 that collects surrounding sound and converts it into an electric signal, a low-pass filter 41 that passes only a signal having a frequency lower than a predetermined frequency, and a microphone 40. A converter 42 that A / D converts a sound that has been sounded and has passed through the low-pass filter 41 to convert the sound into a digital waveform, and an analysis unit (described later) that detects walking by analyzing the waveform converted by the converter 42 ( (Detection unit) 43, a database 44 storing data used for analysis in the analysis unit 43, and an output unit 45 for outputting data of a detection result in the analysis unit 43 to the control device 32 shown in FIG. I have.

In the microphone 40, vibration (including sound) in all frequency bands that can be collected by the microphone 40
Collect. Here, the surrounding sounds are collected by the microphone 40, so that vibrations generated during walking and transmitted to the viewer's body and sounds transmitted in the air are collected.
Usually, in a microphone of a portable telephone terminal,
Although a signal in a frequency band outside the audible range (for example, 20 Hz or less and 20000 Hz or more) is cut by the band-pass filter, in the present embodiment, the signal output from the microphone 40 is determined by the low-pass filter 41. Only signals having a frequency of, for example, 200 Hz or less are passed, and separated from components in a frequency band higher than 200 Hz, so-called audio signals. Then, the analysis unit 43
Then, analysis is performed on a signal having a component equal to or lower than the determined frequency.

The analyzing section 43 includes a spectrogram processing section 51 for generating a spectrogram based on the signal converted by the converter 42, a walking detecting section 52 for detecting a viewer's walking, and a stride estimating section 53 for estimating a stride. A moving distance estimating unit 54 for estimating the moving distance of the viewer based on the stride obtained by the stride estimating unit 53; a walking mode detecting unit 55 for detecting the walking mode of the viewer; And a walking mode determination unit 56 that determines the walking mode of the viewer based on the detection result.

Next, the analysis section 43 uses the microphone 40
The basic concept of performing the analysis based on the sound collected in the section will be described. That is, vibration (sound) when the viewer walks and the left foot and the right foot are alternately landed is collected by the microphone 40, and the vibration is A / D-converted by the converter 42 into a digital signal. When a signal (voltage) is subjected to wavelet transform, a spectrogram which is a frequency intensity spectrum pattern as shown in FIG. 6 is obtained.

FIG. 6 shows a spectrogram when the same viewer walks normally. Here, “normal” means a case where the viewer walks on a flat ground without being fast or slow, and particularly walking without being aware of the pace. As shown in FIG. 6, when a human walks, the impact resulting from the landing of the foot or the ground motion of the foot, the conduction to each joint, skeleton, and muscles of the foot, such as the hips, hips, knees, ankles, and toes, is observed. Audible area (about 20 to 2000)
A characteristic time / frequency / intensity pattern is generated in the vicinity of a low frequency band of about 100 Hz or less, which is close to the lower limit of 0 Hz), more specifically, for example, in the vicinity of 30 to 40 Hz or less. In the present embodiment, walking is detected by analyzing the pattern of the low frequency band component.

In the walking motion of a human, the heel lands (Heel-Strik
e) Ground movement from toe release (Toe-Off)
It can be divided into two phases of kick-out movement from Toe-Off to heel landing. The landing on the sole of the foot at the time of the ground motion starts from the heel, passes through the outer part of the foot (external longitudinal foot arch), changes the direction from the base of the little finger (fifth metatarsal floor), and The procedure is sequentially performed up to the base (the first metatarsal floor of the first finger). The ground impact accompanying this ground motion is absorbed by each joint, muscle, and fat layer, but a part of the low-frequency component resonates with these and conducts through the human body, as shown in the spectrogram of FIG. It emits vibration in a low frequency band having a complicated time / frequency / intensity pattern.

Using the example of the viewer shown in FIG. 6, the components of the low frequency band generated during walking will be examined in detail. FIG.
In (a), clear peaks appear in the portion (a), and each peak corresponds to each step of walking. Examining the spectrogram shown in FIG. 6 in more detail, the range indicated by (a) in the horizontal axis direction is the right foot ground exercise time,
The area shown in (c) is the ground exercise time of the left foot, the area shown in (d) is the gap time when the foot touching the ground changes from the right foot to the left foot, and the area shown in (o) is the foot touching the ground from the left foot to the right foot Is the gap time when changing to. The temporal characteristics of these (a), (c), (d), and (e) time lengths can characterize the manner of walking of the viewer. Here, in the spectrogram, the darker the color, the higher the frequency intensity, and the lighter the color, the lower the frequency intensity.

FIG. 7 shows spectrograms when the same viewer walks in various walking modes. FIG.
(A), when stepping on the spot, (b) walking slowly, (c) walking normally, (d) walking fast, (e) jogging, (f) climbing stairs, (g) Is a spectrogram detected when going down the stairs. Comparing the spectrograms of these walking modes, (b), (c), and (d) of FIG.
Have different ground exercise times (the faster you walk, the shorter the walk). In addition, in the jogging shown in FIG. 7E, the range where the frequency intensity is high extends to the region where the frequency is high, and the peak frequency (a) of the signal is high as compared with the other cases. In the case of climbing the stairs in FIG. 7F, the frequency intensities (f) and (g) when the feet are grounded are weak. This is because the foot is lifted and touches the stair surface, so that the impact when touching the ground is weaker than when walking on a flat surface. In addition, when climbing the stairs, the toe is mainly grounded, and the frequency intensity particularly when the heel of (f) is grounded is low.
Also, comparing the up and down steps of FIGS. 7 (f) and 7 (g), the band extends to a significantly higher frequency band especially in the case of down steps. This means that an energy spectrum is also generated in a high frequency band according to the impact received by the foot. On the other hand, when descending the stairs in FIG. 7 (g), the toes touch the ground before the heels.
The frequency intensity when the toe is grounded is strong.

By the way, as can be seen from the spectrogram of FIG. 6, with the above-described series of ground motions, in normal walking, a band-like pattern over a wide band from low frequencies outside the audible range to the audible range is approximately 1 per step. Continue for about a second. Further, in this band-shaped time / frequency / intensity pattern, intensity spectra of different frequency ranges corresponding to the ground motion of each part of the sole can be seen.

FIG. 8 shows the processing in the walking detecting section 52. In this processing, the ground motion continuation time of each step of the sole obtained from the time, frequency, and intensity in this spectrogram ((a) in FIG. 6). , (C)) to determine the number of steps per unit time. At this time, as shown in FIG. 6, the frequency intensity continues for a certain time or more according to the landing times of the left and right soles, and the spectrogram of time, frequency, and intensity draws a pattern unique to walking ( Utilizing the fact that these data are stored in the database 44 in advance), the number of steps per unit time and the walking cycle are detected, and the characteristics of the time, frequency, and intensity are used to detect the hand of the viewer, for example. Impacts other than walking, such as collision with a vehicle, and external noise caused by a vehicle or the like are distinguished from sounds caused by walking.

First, a signal w obtained by A / D-converting a signal from the microphone 40 by the converter 42 is obtained.
From (t), a signal having a frequency equal to or higher than a predetermined frequency, that is, a normal audio signal is removed by the low-pass filter 41, and the signal w '
(T) is obtained (step S201). And the analysis unit 4
At 3, the signal w '(t) is converted into a function expression p (t, f) of the intensity of time and frequency (step S202).
Subsequently, the function expression p (t, f) is set to a predetermined frequency band B.
And the status of the viewer is determined from the relationship with the frequency intensity α. Here, if the integrated value L _B (t) is larger than the frequency intensity α, the status of the viewer: st
atus (t) is set to “landing state”, and the integrated value L _B (t)
Is less than or equal to the frequency intensity α, the status of the viewer is set to a “normal state”, that is, a state other than the “landing state” (step S203).

Thereafter, every time a new signal is input every minute time, the following step S204 is executed. In this step S204, the signal at the time t at that time is not a signal due to walking, that is, a signal having a characteristic point (pattern, intensity distribution) unique to walking as shown in FIG. It is determined whether or not is determined to be "landing state" (step S204). As a result, if this condition is satisfied, it is determined that an erroneous determination has been made, and the viewer status is changed to “landing state” (step S205). On the other hand, if the condition of step S204 is not satisfied, the status of the viewer is not changed. In this way, each time a signal is input, the status (t) of the viewer at that time is determined, and the status (t-1) immediately before is changed from the "normal state" to the "landing state". If it has changed, the number of landings is counted once (step S206). By calculating the number of landings per unit time from the count value of the number of landings,
The number of steps s per unit time can be obtained, and the walking period can be obtained by dividing the unit time by the number of steps s (step S207).

Next, in the walking mode detecting section 55, FIG.
5 shows a process of recognizing the walking mode of the viewer based on the sounds collected by the microphone 40 based on the characteristics of various walking modes as shown in FIG. In this case, the data of feature points of various walking modes are stored in the database 44 in advance.
As shown in FIG. 9, the analysis unit 43 performs step S in FIG.
The detected walking characteristic pattern W is obtained from the function expression p (t, f) of the intensity of time and frequency obtained in the same manner as in steps 201 to S202 (step S301). Then, the obtained feature pattern W, refers to the database 44, the various features patterns W _k which is stored in the database 44, for example, slow feature pattern W ₁ of the walk, the ordinary walk characteristic patterns W _2, brisk walking the feature pattern W _3, a pattern recognition processing and ... etc. These include, for example, by performing a characteristic pattern W _k, the arithmetic operations of the Mahalanobis distances between the feature pattern W _k stored in the database 44, obtains a difference d _k between the feature pattern W (step S30
2). Then, of the differences d _k from each feature pattern W _k ,
The smallest one is searched (step S303), and the walking mode of the viewer is estimated from the feature pattern W _k with the smallest difference d _k (step 304).

In order to estimate a stride by the stride estimating unit 53, processing based on the detection results of the walking detecting unit 52 and the walking mode detecting unit 55 is performed. As one method of the processing here, there is a method in which a walking frequency is detected by the walking detecting unit 52 and a stride is estimated from the walking frequency and the height of the viewer. Generally, as shown in FIG. 10, the walking frequency (step)
It is known that the height of the viewer is correlated with the stride. FIG. 10 plots the relationship between walking frequency, height, and stride for a large number of subjects, and it is clear that these have a correlation as a whole.

In this case, the database 44 includes
The data of the relational expression (coefficient of) of the walking frequency, the height of the viewer, and the stride obtained from the correlation shown in FIG. Then, prior to detection of walking, the height (and gender) of the viewer is input to the setting memory 57. To estimate the stride, as shown in step S401 in FIG. 11, the stride is calculated using the walking model M based on the walking cycle detected by the walking detecting unit 52 and the height of the viewer set in the setting memory 57. Predict l. The walking model M is represented by, for example, the following relational expression. Step length l (cm) = x1 x walking cycle (step / min) + x2 x
Height (cm) + C In the above relational expression, x1, x2, and C are coefficients and initial values stored in advance in the database 44 and set for each walking mode.

Then, the moving distance estimating unit 54 integrates the step length l obtained in this way and the number of steps s per unit time detected by the walking detecting unit 52 to obtain the moving distance per unit time. The distance can be estimated (step S402).

Further, in step S401, an example of the relational expression of the walking model M has been described, but the present invention is not limited to this. For example, the relational expression of the walking model M can be determined based on the ratio of the energy when the heel lands and the energy when the toe kicks out. Those shown in FIG. 12, which represents the change in energy when walking, wherein the energy E ₁ kicking the energy when kicking the toe is the peak value of the walk signal. Also, the heel landing energy E ₂ at the time of landing the heel is the maximum value immediately before the peak value corresponding to the energy E ₁ out the kick. Then, divided by the energy E ₁ kick the heel landing energy E ₂ is the energy ratio kick landing-. Based on the landing / kickout energy ratio obtained in this manner, the walking cycle detected by the walking detecting unit 52, and the height of the viewer set in the setting memory 57, the stride l is calculated using the walking model M '. Can be predicted. The walking model M ′ is represented by, for example, the following relational expression. Step length l (cm) = x1 x walking cycle (step / min) + x2 x
Height (cm) + x3 × landing / kicking energy ratio In the above relational expression, x1, x2, and x3 are coefficients for each walking mode stored in the database 44 in advance.

As another method of estimating a stride by the stride estimating unit 53, there is a method that utilizes a change in frequency intensity resulting from an impact when a heel or a toe lands. In the human walking mode, walking at a fast pace increases both the walking frequency and the stride. However, when walking consciously, the walking frequency itself does not change, and only the stride changes. In addition, when moving long distances, the stride is a substantially constant steady value determined by the height and the pace, but the stride easily changes depending on the short moving distance and the mood at that time. In order to avoid such an effect, as shown in FIG. 6, the frequency intensity at the time of normal walking is stored in advance in the database 44 as a reference value, and the heel and the toe are landed by comparison with this reference value. The increase or decrease in the amount of impact at the time of performing is detected, and the stride is corrected.

FIG. 13 shows a specific flow of the processing as described above. First, at step S401 in FIG.
Similarly to the above, the stride 1 is predicted using the walking model M or M ′ based on the walking cycle detected by the walking detecting unit 52 and the height of the viewer set in the setting memory 57 (step S501). Next, steps S201 to S20 in FIG.
Relational expression p between time and frequency intensity obtained in the same manner as
(T, f) is integrated with respect to the frequency band B 'in which the landing impact can be characteristically extracted, to obtain the landing impact L _B ' (step S502). Here, what is obtained by integrating the relational expression p (t, f) is the occupancy (area) of the signal of the spectrogram in the frequency band B ′. Based on the table or relational expression stored in the database 44 in advance, a correction value Δl of the stride is calculated from the impact L _B ′ (step S503). Then, the correction value Δl of the stride is
Is used to obtain the moving distance (1 × s + Δl) (step S
504). In this way, the stride can be corrected from the frequency intensity of the spectrogram.

Further, as another method of estimating the stride by the stride estimating unit 53, there is a method that utilizes the frequency intensity when the toe is kicked out and when the toe lands. That is, even when walking at the same pace, the stride length varies depending on the kicking strength of the toes. For this reason, according to the intensity of the frequency band of, for example, about 10 to 16 Hz generated according to the landing and kicking of the toe, the reference value of the frequency intensity of the reinforcing person stored in the database 44 in advance, or the conventional walking The step length is corrected by comparing the history data.

FIG. 14 shows the flow of such processing.
Therefore, similar to steps S501 to S503 in FIG.
The walking cycle detected by the walking detecting unit 52 and the setting memory 5
7 based on the height of the viewer set in Step 7.
Or the step length l is predicted using M ′ (step S60).
1) The relational expression p (t, f) between time and frequency
For the frequency band B 'where the ground impact can be extracted characteristically
Integrate and land impact L _B(Step S60)
2). Further, data stored in the database 44 in advance is used.
This impact L_B’
The width correction value Δl is calculated (step S603). this
Thereafter, the analysis unit 43 determines a predetermined low frequency band f
(For example, 0 to 40 Hz), the relational expression p (t, f)
Are extracted (step S604), and the data
The characteristic pattern data of the frequency intensity stored in the base 44
Data, the reference value of the viewer, and walking history data,
The correction value h is obtained (Step S605). After a while
Uses the correction value h to calculate the moving distance (l × s + Δl +
h) can be obtained (step S606).

According to the position detecting device 36, the microphone 40 collects low-frequency band components transmitted through the body during walking, and detects walking based on the collected low-frequency components. Can be estimated.
Thus, the amount of walking movement of the viewer (from the detection start position such as a start point), that is, the position of the viewer can be detected.

In the compound-eye image reproducing system having the above-described configuration, the control device 3 is controlled based on dynamic data such as the position, orientation, and line of sight of the viewer obtained by the dynamic detector 35.
2 reads the optimal image data from the image data storage 31, and causes the reproduced image display unit 33 to reproduce and display an image based on the image data. In addition, based on the height of the viewer set in the setting memory 57, the dynamic detection unit 35 determines not only the position of the viewer but also the position coordinates of the viewpoint of the viewer (in a coordinate system in two horizontal directions and one vertical direction). Position coordinates)
Can be detected. At this time, the position of the viewer does not always match the shooting position in the compound-eye image shooting system. Therefore, based on the image data stored in the image data storage 31, the control device 32 corrects the image obtained at the shooting position at the time of shooting according to the line of sight from the viewer's position.

FIG. 15 is a diagram for explaining this correction principle. It is assumed that an image photographed at the viewpoint R at the time of photographing is projected inside a virtual spherical surface K having a radius r around the viewpoint R at the time of photographing. The radius r is determined by the distance to the closest one of the subjects in the image. That is,
At the time of photographing, it is preferable to record data of the distance to the closest subject together with the image data. The coordinates of a point P on the virtual spherical surface K are represented by φ, the azimuth angle with respect to the x-axis in the horizontal plane, and the vertical Is the angle of elevation in θ, P = (x, y, z) = (rcosθcosφ, rcosθsinφ, rsinθ). On the other hand, when the coordinates of the viewpoint V at the time of reproduction are V = (x ₀ , y ₀ , z ₀ ), when the point P is viewed from the viewpoint V at the time of reproduction,
It can be projected onto a point P 'on a virtual spherical surface K' having a radius r 'based on the viewpoint V at the time of reproduction. The point P 'is, P' = (x, y , z) = β (rcosθcosφ-x 0, rcosθsinφ-y 0, rsinθ-z 0) is expressed as. Here, β is a variable. Needless to say, since the relationship x ² + y ² + z ² = r ² holds, if the equation for the point P ′ is solved for the variable β, the variable β becomes the size of the image at the time of reproduction.

Here, in order to reproduce an image at the time of reproduction, the size of the image needs to be 0 or more. Therefore, the condition of β> 0 is determined by changing the viewpoint V = (x ₀ , y ₀ , z ₀ ) must be satisfied. If this condition cannot be satisfied, it means that the reproduction limit has been exceeded. Specifically, FIG.
As shown in FIG. 5, outside the virtual spherical surface K determined at the time of shooting, β does not satisfy the above condition, so that the image cannot be reproduced.

As described above, the viewpoint V at the time of reproduction is
However, if the above conditions cannot be satisfied, the compound-eye image reproduction system cannot reproduce the image. When the subject Q as shown in FIG. 16A is photographed by the compound-eye image photographing system, as shown in FIG. 16B, the image pickup unit 10 of the compound-eye image photographing system is moved as indicated by an arrow (c). In addition, there is a blind spot (S) in which photographing cannot be performed, that is, image data cannot be obtained on the back side of the subject Q or the like. In addition to this, for example, a wall or a window, which cannot be physically moved at the time of photographing, cannot be photographed on the other side.

Because of such restrictions, in the control device 32, as shown in FIG. 17, in the image displayed on the reproduced image display section 33, the area where the image can be reproduced, that is, the viewer can move. Area F is specified. Further, the control device 32 checks whether or not the position of the viewer's viewpoint V detected by the dynamic detection unit 35 satisfies a predetermined condition for reproducing an image. If playback is not possible, a caution is output to the viewer. More specifically, when the position of the viewpoint V of the viewer is going to go from the inside to the outside of the virtual spherical surface K which is a predetermined range determined based on the viewpoint at the time of shooting, the back side of the subject, the wall or the window For example, when trying to move to the other side, which cannot be physically exceeded during shooting, caution information CI as shown in FIGS. 18A and 18B is displayed on the reproduced image display unit 33. .

For example, when the viewer walks forward as indicated by an arrow J1 while an image as shown in FIG. 17A is displayed, the reproduced image display unit 33 displays the image shown in FIG. As shown, an image obtained at the advanced position is displayed.
Then, as shown by an arrow J2 in FIG. 17B, when the viewer changes the direction, the dynamic state detection unit 35 detects this, and
As shown in FIG. 19A, a window D and an image of a scenery seen through the window D are displayed on the reproduced image display unit 33.
Here, when the viewer approaches the window D as shown by an arrow J3 in FIG. 17 (b), as shown in FIG. The image is displayed. When the viewer tries to approach the window D further, the dynamic detection unit 35 determines that the position of the viewer is in FIG.
It is detected that the user is about to go out of the area F shown in (b) (or has gone out), and at this time, for example, the caution information CI as shown in FIG. 33 is displayed. In this case, the area F is limited for physical reasons because it is not possible to pass through the window D during photographing. According to the caution information CI, it is possible to prevent the viewer from going out of the area F. Also, as shown by an arrow J4 in FIG. 17B, the caution information CI as shown in FIG. 18B is displayed on the reproduced image display unit 33 even when the viewer moves forward. In this case, the area F is limited because the image data does not exist because no image is taken ahead of the area F.
It is designed to prevent you from going outside.

Next, a series of flows in the case of displaying an image in the compound-eye image reproducing system having the above configuration will be described with reference to the flowchart shown in FIG. First, the viewer wears the reproduced image display unit 33 including the head-mounted display 34 shown in FIG.
Register with. And GPS or GI (Geographic
al Information) The start position of the viewer is set by the viewer operating a localization means for positioning the absolute position such as a stone (electronic geographical information mark) or a predetermined switch at a predetermined start point. (Step S
701).

Thereafter, the viewer starts walking and moves freely. Then, the movement detecting unit 35 detects the walking distance of the viewer from the start point by the method shown in FIG. 11, FIG. 13, or FIG. 14 at predetermined small time intervals (step S702). And the detected walking distance,
From the direction of the viewer detected by the gyro sensor 37,
The relative position of the viewer with respect to the start point is calculated. At this time, from the height of the viewer set in the setting memory 57, the coordinates of the viewpoint V of the viewer are actually obtained (step S703: position detection step). Subsequently, the control device 32 extracts the shooting position corresponding to the coordinates of the viewpoint V, that is, the position coordinates at the time of reproduction (step S704). For this purpose, a predetermined number (one or more) of the viewpoints R close to the coordinates of the viewpoint V may be extracted from the viewpoints R at the time of capturing a plurality of images stored in the image data storage 31.

Next, a study will be made on images (the number of which is equal to the number of cameras 15 constituting the image pickup section 10 of the compound-eye image photographing system) photographed at each of the extracted photographing positions (viewpoints R). To do this, it is determined whether each image satisfies the relationship | R−V | <r. Thus, as described above, it is determined whether or not the coordinates of the viewpoint V at the time of reproduction are inside the virtual sphere K having a radius r around the viewpoint R at the time of capturing each image data. As a result, if the above relationship is satisfied, the image is extracted as a reproduction candidate image. Such a process is performed for all images photographed at the photographing position extracted in step S704. As a result, an image satisfying the above relationship is extracted as a reproduction candidate image (step S705).

Subsequently, it is determined whether or not there is a reproduction candidate image extracted in step S705 (step S70).
6). As a result, if a reproduction candidate image exists, an image to be actually reproduced is selected in the next step S707 (image selection stage). For this purpose, an image that minimizes | R−V | is selected from the images extracted in step S705. Then, the selected image is deformed or combined with an adjacent image by using various image processing techniques, if necessary, so as to be an image viewed from the viewpoint V at the time of reproduction (step). S708). Then, the control device 3
No. 2 displays the image on the reproduced image display section 33 (step S709: display stage). Thus, the viewer can view the reproduced image at the viewpoint V after the movement according to his / her own movement.

If the result of the determination in step S706 is that there is no playback candidate image extracted in step S705, the viewpoint V at the time of playback is outside the specified area F. The caution information CI as shown in FIG. 18A or 18B is displayed on the reproduced image display unit 33 (step S710: caution output stage). The processes of steps S701 to S710 are repeatedly performed at predetermined small time intervals. As a result, the image and the caution information CI corresponding to the position of the viewer are sequentially (at any time) displayed on the reproduced image display unit 33. Thus, the viewer gets an experience as if he / she is moving in the virtual space constituted by these images.

With the above-described configuration, the reproduced image display section 33 clearly shows the area F in which the image can be reproduced, and further displays the caution information CI when the user tries to leave the area F. did. Thus, it is possible to prevent the viewer from getting out of the area F.
If the viewer follows the caution information CI, the state in which the image is always displayed on the reproduced image display unit 33 in the area F is maintained, and the image is suddenly interrupted or returns to the start point as in the related art. It is possible to have a virtual experience without any discomfort.

The position detector 36 of the dynamic detector 35
Then, the walking of the viewer is detected, and the walking movement amount of the viewer is estimated based on the detected walking. With this, GPS
Can be used to determine the position of the viewer without using
Not only will there be no problems even indoors, but it is also possible to reliably locate the viewer in the virtual space at low cost. In addition, the position detecting device 3
Reference numeral 6 denotes a microphone 40, which collects low-frequency band components transmitted through the body during walking and detects walking from this sound, so that it is possible to distinguish it from noise (vibration or sound) other than during walking. This can be performed reliably, and a highly accurate detection result can be obtained. In this way, in the compound-eye image reproduction system, the viewer can freely walk around the area F, and can experience a virtual space that has never existed before.

In the above-described embodiment, the reproduced image display section 33 displays the area F to which the viewer can move. This is because, as shown in FIG. 21 (a), when the viewer views an image reproduced by the compound-eye image reproduction system in an actual space RA such as a gymnasium, for example, the reproducible area F becomes an actual space RA. If it is completely within the RA, it can be applied as it is. By the way, as shown in FIG. 21B, the reproducible area F may not completely fit in the actual space RA ′. In such a case, it is possible to display the area and output the caution information CI by adding a restriction not only to the area F that can be played back but also to the actual space RA ′ in the playback image display unit 33. . In this way, it is possible to prevent the viewer from hitting a wall or the like of the gymnasium constituting the actual space RA ', even though the viewer is in the area F.

Further, in the above-described embodiment, the head mounted display 34 as shown in FIG. 4 is used as the reproduced image display section 33, but other forms may be employed. is there. Further, the compound-eye image reproduction system may have all of the components mounted on such a head-mounted display 34,
Some components such as the image data storage 31 and the control device 32 may be separated from the head-mounted display 34, and may be connected to each other by communication means such as a cable or wireless communication. In addition, the position detection device 3 of the dynamic detection unit 35
In 6, the low frequency band component transmitted through the body during walking is collected, and based on this, the viewer's walking is detected. The method for estimating the walking distance is shown in FIG.
The method is not limited to those shown in FIGS. 13 and 14, and other methods may be applied. Alternatively, walking may be detected using an acceleration sensor or the like. Further, in the position detecting device 36, the position of the viewer is detected from the detected amount of walking movement. However, if it is used outdoors or the like, a GPS can be used as the position detecting device 36. In addition, in addition to this GPS, walking may be detected in the same manner as in the above-described embodiment, and the position of the viewer may be detected in consideration of information on the amount of walking movement.

In the above embodiment, the method of photographing an image and the process of selecting an image to be displayed at the time of reproduction have been described. However, any other photographing method (such as An apparatus) or a processing method may be used. Also, the conditions for outputting caution information CI are not intended to be limited to the conditions described in the above embodiment, and the caution information CI may be output in accordance with appropriately set conditions. Further, the caution information CI to be output is not limited to the display of the mark or the message. For example, a message by voice, a warning sound, vibration, or the like may be used, or these may be appropriately combined.

[0054]

As described above, according to the present invention,
It is possible for the viewer to surely grasp the movable area. Further, according to the present invention, the movement of the viewer can be reliably detected, and the viewer himself can freely move in the virtual space without a sense of incongruity.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of a compound-eye image capturing system according to the present embodiment.

FIG. 2 is a three-view drawing showing a specific example of an imaging unit.

FIG. 3 is a diagram illustrating a configuration of a compound-eye image reproduction system.

FIG. 4 is a diagram illustrating an example of a head mounted display.

FIG. 5 is a diagram showing a configuration of a position detection device.

FIG. 6 is an example of a spectrogram obtained from signals continuously obtained when a viewer walks.

FIG. 7 is a diagram showing spectrograms of the same viewer in various walking modes.

FIG. 8 is a diagram showing a flow of processing when detecting the number of steps.

FIG. 9 is a diagram showing a flow of processing when estimating a walking mode.

FIG. 10 is a data distribution diagram showing a correlation between a walking frequency, a stride, and a height.

FIG. 11 is a diagram showing a flow of processing when estimating a moving distance.

FIG. 12 is a diagram showing a change in energy during walking.

FIG. 13 is a diagram showing a flow of another process when estimating a moving distance.

FIG. 14 is a diagram illustrating a flow of still another process when estimating a moving distance.

FIG. 15 is a diagram for explaining the principle of correcting an image obtained at a shooting position at the time of shooting according to a line of sight from a viewer's position.

FIGS. 16A and 16B are diagrams illustrating an example of a subject in which a blind spot occurs during photographing, where FIG. 16A is a front view and FIG. 16B is a plan view of FIG.

FIG. 17 is a diagram illustrating a display example of an image on a reproduced image display unit.

FIG. 18 is a diagram illustrating an example of a caution to be output.

FIG. 19 is a diagram showing a display example of an image when the viewer changes the direction from the display state of FIG. 17 (b).

FIG. 20 is a diagram showing a series of flows during reproduction.

FIG. 21 is a diagram illustrating an example of a relationship between an area that can be displayed based on image data and a place (space) where playback is actually performed;

[Explanation of symbols]

31 image data storage (data storage means), 32
... Control device (control means, caution output means), 33 ...
Reproduction image display unit (display means), 34: head mounted display (viewer), 35: dynamic detection unit, 36 ...
Position detecting device (position detecting means, viewpoint position detecting means, walking detecting device), 40: microphone, 41: low-pass filter, 43: analyzing unit (detecting unit), 53: step estimating unit,
54: Moving distance estimation unit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/225 H04N 7/18 U 5C054 G10L 3/00 531N 5D015 5/76 551G 5/93 7/08 C 7/18 H04N 5/93 EF term (reference) 5B050 BA09 BA10 DA07 EA03 EA13 EA27 FA02 FA10 5B069 AA16 BA10 DD09 DD15 JA04 5C022 AB68 AC01 AC13 AC18 AC21 AC42 AC77 5C052 AA01 AB04 CC06 DD04 5C053 FA23 HA40 A05A05 A05 A05 LA05 CC02 CE02 EA05 EA07 EH01 FA02 FD02 FE12 FE28 GB06 HA16 5D015 AA06 CC03 KK01

Claims (9)

[Claims] 1. A data storage means for storing image data, a display means set in a viewer and displaying an image based on the image data stored in the data storage means, and a position detection for detecting a position of the viewer. Means, based on a position detected by the position detecting means, a control means for selecting an image from among images in which image data is stored in the data storage means, and displaying the selected image on the display means; A position output unit that outputs a caution to a viewer when a position detected by the position detection unit deviates from a predetermined range determined by a shooting viewpoint of an image in which image data is stored in the data storage unit; An image reproducing apparatus comprising: 2. The image data of a compound eye image obtained by performing photographing in a plurality of directions from one photographing position is stored in the data storage means. The display means displays an image based on the image data of the compound eye image. The image reproducing apparatus according to claim 1, wherein the image is displayed. 3. The position detecting means detects a walk of the viewer by a walk detecting device set to the viewer, and detects a moving amount of the viewer based on information of the detected walk. The image reproducing apparatus according to claim 1, wherein: 4. A data storage means for storing image data, a display means for displaying an image based on the image data stored in the data storage means to a viewer, and a viewpoint for detecting a position of a viewpoint of the viewer. Position detecting means, selecting an image to be displayed based on the position of the viewer's viewpoint detected by the viewpoint position detecting means, extracting image data of the selected image from the data storage means, Control means for displaying an image based on the image data on the display means, and area display means for displaying on the display means an area where an image can be displayed based on the image data stored in the data storage means. An image reproducing apparatus characterized by the above-mentioned. 5. A caution output means for outputting a caution to a viewer when a relation between the position of the viewer's viewpoint detected by the viewpoint position detecting means and the area satisfies a predetermined condition. The image reproducing apparatus according to claim 4, further comprising: 6. A data storage means for storing image data, a display means set in the viewer and displaying an image based on the image data stored in the data storage means, and detecting a walking movement amount of the viewer. Position detecting means for detecting the position of the viewer by means of, based on the position detected by the position detecting means, an image is selected from the image data stored in the data storage means, the selected An image reproducing apparatus, comprising: control means for displaying an image on the display means. 7. The microphone according to claim 1, wherein the position detection unit collects vibrations generated when the viewer walks and converts the vibrations into electric signals, and based on the electric signals transferred from the microphone,
7. The image reproducing apparatus according to claim 6, further comprising: a detecting unit that analyzes a change in a signal corresponding to a predetermined frequency or less to detect a walker of the viewer and detects a walking movement amount of the viewer. . 8. A method for reproducing an image captured at a plurality of positions in a predetermined area, comprising: a position detecting step of detecting a position of a viewer wearing a viewer of the image; An image selection step of selecting an image to be displayed on the viewer; a display step of displaying the selected image on the viewer; and a caution for the viewer when the image selected in the image selection step does not exist. Outputting a caution output step. 9. In the position detecting step, a component in a frequency band of 100 Hz or less is detected from vibration transmitted through the body of the viewer when the viewer walks, and the detected component is converted into a signal. 9. The image reproducing method according to claim 8, further comprising: analyzing the information to detect the walking of the viewer, and estimating a walking movement amount of the viewer.