JP2003199074A - Video image presentation system, video image transmission device, video image presentation device and video image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video image presentation system capable of presenting the movement in space to a user by reproducing the video image where a state of walk-through is recorded and enabling the user to move his/her sight lines freely and smoothly to change the visual field. <P>SOLUTION: In the video image presentation system, a video image is transmitted from a video image transmission device 3 to a video image presentation device 1, and the video image is presented with the device 1. In this system, the device 3 calculates a segmented range according to the direction of user's sight line during video reproducing on the basis of the designation, notified from the device 1, of the direction of the user's sight line who watches the video image and the device 3 segments the video image into the segment range to transmits it as a new video image to the device 1, and the device 1 receives the transmitted video image and present the image. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video image presenting system, and more particularly to a video image presenting system, a video image transmitting device, which can respond to a user's request for changing the visual field even when presenting accumulated video images. The present invention relates to a video image presentation device and a video image processing program.

[0002]

2. Description of the Related Art Conventionally, a movement route is set in a street or a building, and the movement on the route is observed in the same way as when a person walks or observes while moving in a car. It is continuously recorded in a video image, and when playing it, a video image corresponding to the user's position is reproduced from the recorded video image, as if a person walks and observes, or moves in a car. There is known a video image presentation device that provides an image similar to the case.

Hereinafter, a video image recording a state of moving in a street or a building is referred to as a walkthrough image.

As an example of a video image presentation device that provides walk-through images, "NTT R & D, Vol.49, No.1 PP.2"
5-30, published in 2000 ", an example of a video image presentation device that provides a walk-through image moving in an actual street or building is reported, and JP-A-10-21406.
No. 9 discloses an example of a video image presentation device that provides a walk-through image moving in a virtual space constructed by using three-dimensional computer graphics technology.

Further, these video image presentation devices are VO
By using a D (Video On Demand) system, it is possible to provide a user with a walk-through video over a network.

In the VOD system, in an environment in which a server machine and a client machine are connected by a network, a video image is stored in the server machine, and the video image stored in the server machine is networked according to a request from a client machine. It is a system for delivering via video and presenting a video image on a client machine, and is disclosed in, for example, "NEC Technical Report, Vol. 51, No. 8, (August 1998)".

[0007]

However, the above-mentioned conventional technique has the following problems.

First, there is a problem that the user cannot change the visual field by freely moving the line-of-sight direction up, down, left and right while receiving a walk-through image moving on the route. .

This is a problem due to the characteristic of the system that a video image is recorded in advance and the walk-through image is presented to the user by reproducing it.

In order to give a degree of freedom to the line-of-sight direction, video images obtained by observing various directions while moving on the same route are recorded, and when presenting a walk-through image, a user's request is made. A method of switching the video image to be reproduced according to

However, for example, when moving the visual axis in the horizontal direction by 180 degrees, 180 video images must be prepared in order to realize the visual axis movement in steps of 1 degree, so the video images are recorded. It can be said that it is practically difficult to implement considering the man-hours and the storage location of video images.

Secondly, for example, a state of moving on a plurality of routes is divided for each route and stored as a video image, and the video image is switched according to a route selection operation by a user,
When providing a service to users by combining multiple video images, such as presenting a walk-through image, when switching the video images, the image is temporarily stopped at the joint, and There was a problem that it gave an unnatural impression.

This is because it is necessary to first confirm the location of the video image, read the video image, and then start the reproduction before starting the display of the video image. In particular, it becomes remarkable when the video image stored in the server machine is acquired and displayed via the network in the form of streaming video.

This is because in the streaming video system, it is generally necessary to establish one virtual connection between the server and the client for one video reproduction. Therefore, when switching the video image, the connection is discarded, In addition to the establishment overhead, in the streaming video system, a buffering process of about 5 seconds is performed to maintain smooth playback even when the network communication status changes, so new video is added when switching video images. This is because even if the reproduction is started, it is necessary to wait until the buffering time elapses before the actual reproduction is started.

A first object of the present invention is to solve the above-mentioned drawbacks of the prior art and to freely and smoothly move the line of sight, a video image presentation system, a video image transmission device, a video image presentation device, It is to provide a video image processing program.

A second object of the present invention is to solve the above-mentioned drawbacks of the prior art, particularly to solve these problems which become remarkable when it is connected via a network, and to provide a user when switching a plurality of video images. An object of the present invention is to provide a video image presenting system, a video image presenting device, a video image presenting device, and a video image processing program capable of smoothly displaying video images without being aware of the joints between the video images.

[0017]

In order to achieve the above object, the present invention provides a video image presentation system for presenting captured video images, accepting designation of a line-of-sight direction of a user who views the video images, A cutout range corresponding to the line-of-sight direction is calculated, and the video image is cut out based on the calculated cutout range and presented as a new video image.

The video image presentation system of the present invention according to claim 2 is a video image presentation device for presenting the video image, and a video image for cutting out the video image and sending a new video image to the image presentation device. A delivery device is provided.

In the video image presentation system of the present invention according to claim 3, the video image transmission device calculates the cut-out range in the video image corresponding to the direction of the user's line of sight notified from the image presentation device. Then, the video image is cut out based on the calculated cut-out range and is sent to the image presentation device as a new video image.

In the video image presentation system of the present invention according to claim 4, the image presentation device calculates the cut-out range in the video image corresponding to the direction of the line of sight of the user,
It is characterized in that a new video image is cut out and presented based on the cutout range from the video image from the video image sending device.

In the video image presentation system of the present invention according to claim 5, the video image transmission device calculates a cutout range in the video image corresponding to the user's line-of-sight direction notified from the image presentation device. , Of each block obtained by dividing the screen of the video image into predetermined sizes,
It is characterized in that an area of each block in the range including the cutout range in the video image is cut out as a new video image and sent to the image presentation device.

In the video image presentation system of the present invention according to claim 6, the image presentation device calculates a first cut-out range in the video image corresponding to the direction of the line of sight of the user, and the image transmission device Calculating a second clipping range that is larger than the first clipping range based on the first clipping range, clipping the video image based on the second clipping range, and creating a new video image Is sent to the video presenting device, and the video presenting device receiving the new video image cuts out the new video image based on the first cutout range and presents it.

In the video image presentation system according to the present invention of claim 7, the image presentation device receives the second cutout range from the image sending device, and receives the first cutout range and the second cutout range. By comparing, when the first cutout range is included in the second cutout range, the new video image is cut out based on the first cutout range, and the first cutout range is the first cutout range. If it is not included in the second cutout range, the second
The first cutout range is changed so as to be included in the cutout range, and the new video image is cut out based on the changed first cutout range.

In the video image presentation system of the present invention according to claim 8, the video image transmission device transmits the video image to the image presentation device based on the number of the image presentation devices to which the video image is transmitted. By changing the size of the image to a predetermined enlargement ratio, the communication amount of the video image is suppressed below the communication band between the video image transmission device and the image presentation device.

In the video image presentation system according to the present invention of claim 9, the size of the image of the video image transmitted by the video image transmission device to the image presentation device is predetermined based on an instruction from the image presentation device. It is characterized in that the communication amount of the video image is suppressed below the communication band between the video image transmitting device and the image presenting device by changing to the enlargement ratio.

According to a tenth aspect of the present invention, there is provided the video image presentation system, wherein the image presentation device sets the size of the image of the video image transmitted by the video image transmission device to the video image transmission device. It is characterized in that the instruction is given based on the reception state of.

In the video image presentation system according to the present invention of claim 11, the image presentation device receives the image size of the video image transmitted by the video image transmission device from the video image transmission device. The feature is that it is specified based on the processing status of the video image.

The video image presentation system of the present invention according to claim 12 is characterized in that the video image is transmitted from the image presentation device to the image presentation device via a network.

In the video image presentation system according to the present invention of claim 13, the video presentation device compresses the video image to be sent and then sends the video image, and the video presentation device receives the compressed image from the video presentation device. The present invention is characterized by presenting the decoded video image after decoding.

The video image presentation system according to the present invention of claim 14 receives a plurality of the extracted video images from the image transmission device at the same time, and selectively switches each of the video images to the image presentation device. It is characterized by comprising a switching device for transmitting without changing the connection with the video presentation device.

The video image presentation system of the present invention according to claim 15 is characterized in that the video image is a walk-through image recording a state of moving in a street or a building.

The video image presentation system of the present invention according to claim 16 is characterized in that the video image is a wide-field video image including an image in a range in which the user's line of sight moves.

The present invention of claim 17 is a video image transmitting apparatus for transmitting a video image to an image presenting apparatus, wherein said video image transmitting apparatus corresponds to a line-of-sight direction of a user who sees said video image notified from said image presenting apparatus. It is characterized by comprising: means for calculating a cutout range in the video image; and cutout means for cutting out the cutout range of the video image as a new video image and sending it to the image presentation device.

The video image transmitting apparatus according to the present invention of claim 18 corresponds to the line-of-sight direction of the user in the video image within each block obtained by dividing the screen of the video image into predetermined sizes. It is characterized by further comprising means for cutting out an area of each of the blocks in the range including the cut-out range calculated as a new video image and transmitting the new video image to the image presentation device.

According to a nineteenth aspect of the present invention, in the video image sending apparatus according to the present invention, the size of the image of the video image sent to the video presenting apparatus is predetermined based on the number of the video presenting apparatuses to which the video image is sent. By providing a unit for changing to the enlargement ratio, the communication amount of the video image is suppressed below the communication band with the image presentation device.

The video image sending device of the present invention according to claim 20 is a means for changing the size of the image of the video image sent to the video presenting device to a predetermined enlargement ratio based on an instruction from the video presenting device. By including the above, the communication amount of the video image is suppressed below the communication band with the image presentation device.

According to a twenty-first aspect of the present invention, there is provided a video image transmitting apparatus including means for transmitting the video image after compressing the video image.

According to a twenty-second aspect of the present invention, there is provided a video image transmitting device for transmitting the video image to the image presenting device via a network.

The present invention of claim 23 is, in an image presentation device for presenting a video image sent from a video image sending device, instructing the video image sending device to specify a line-of-sight direction of a user who views the video image. It is characterized by including a means, and receiving and presenting a new video image clipped from the video image in the clipping range corresponding to the line-of-sight direction by the video image sending device.

According to a twenty-fourth aspect of the present invention, there is provided the first video image in the video image corresponding to the direction of the line of sight of the user.
And a new video image clipped in a second clipping range that is larger than the first clipping range calculated by the video sending device based on the first clipping range. Is included, and the new video image is cut out based on the first cut-out range and is presented.

In the video presenting apparatus of the twenty-fifth aspect of the present invention, the clipping means receives the first clipping range from the image sending device and compares the first clipping range with the second clipping range. Thus, when the first cutout range is included in the second cutout range, the new video image is cut out based on the first cutout range, and the first cutout range is the second cutout range.
Is not included in the cut-out range, the first cut-out range is changed so as to be included in the second cut-out range, and the new video image is cut out based on the changed first cut-out range. It is characterized by

According to a twenty-sixth aspect of the present invention, the video presenting device instructs the video image sending device to change the size of the image of the video image to a predetermined enlargement ratio based on the reception state of the video image. It is characterized by comprising means for performing.

According to the twenty-seventh aspect of the present invention, in the video presentation device, the size of the image of the video image is changed to a predetermined enlargement ratio based on the processing status of the received video image for the video image transmission device. It is characterized by comprising means for giving an instruction.

According to a twenty-eighth aspect of the present invention, there is provided the image presenting apparatus, which receives the video image from the video image transmitting apparatus via a network.

According to a twenty-ninth aspect of the present invention, there is provided a video presenting device comprising means for receiving a plurality of the video images simultaneously from the video image sending device and selectively switching each of the video images for presentation. To do.

The video presenting apparatus according to the present invention of claim 30 is characterized in that the video image compressed and transmitted from the video image transmitting apparatus is presented after being decoded.

According to a thirty-first aspect of the present invention, in a video processing program for transmitting a video image from a video transmission device to a video presentation device by controlling a computer, the video image notified from the video presentation device is displayed. A function of calculating a cutout range in the video image corresponding to the viewing direction of the viewing user, and a function of cutting out the video image based on the calculated cutout range and presenting it to the image presentation device as a new video image. It is characterized by being provided.

According to a thirty-second aspect of the present invention, in a video processing program in which a video is sent from a video sending device to a video presenting device and presented by controlling a computer, a line of sight of a user who sees the video image. A function of instructing the video image sending device to specify a direction, and a function of receiving and presenting a new video image cut out from the video image in the cutting range corresponding to the line-of-sight direction in the video image sending device. It is characterized by including.

A video processing program according to a thirty-third aspect of the present invention is based on the function of calculating a first cutout range in the video image corresponding to the direction of the user's line of sight, and based on the first cutout range. Receives a new video image clipped in a second clipping range that is larger than the first clipping range calculated by the video sending device, and clips the new video image based on the first clipping range. It is characterized by having a function of presenting as.

In the video processing program of the present invention according to claim 34, the cutout function receives the first cutout range from the image sending device, and compares the first cutout range with the second cutout range. Thus, when the first cutout range is included in the second cutout range, the new video image is cut out based on the first cutout range, and the first cutout range is set to the second cutout range. If the new video image is not included in the cutout range, the first cutout range is changed to be included in the second cutout range, and the new video image is cut out based on the changed first cutout range. Is characterized by.

According to the present invention, an area corresponding to the line-of-sight direction is cut out from the video image and presented based on the designation of the line-of-sight direction of the user who views the video image.
It is possible to continuously change the line-of-sight direction up, down, left and right, and switch the video image according to the change.

Further, according to the present invention, in the video image sending device, an area corresponding to the line-of-sight direction is roughly cut out in block units from the video image, and the video cutting device receives the roughly cut out video image, A range corresponding to the precise line-of-sight direction can be cut out and displayed.

Further, according to the present invention, the size of the video image can be controlled and distributed based on the network band between the video image transmitting device and the image presentation device.

Further, according to the present invention, when a delay in transmission of a video image caused by a change in the network condition or a frame loss thereof is detected, the video image transmitted by the video image transmitting device is relieved in order to mitigate the situation. By controlling the size of the image, QoS control can be realized.

Further, in the present invention, smooth video switching can be realized by switching video images in one connection.

[0056]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing the configuration of a video image presentation system according to the first embodiment of the present invention.

Referring to FIG. 1, the video image presentation system according to the present embodiment moves along a street or a building with a walk-through image presentation device 1 (walk-through image presentation client) that presents a walk-through image to a user. The video image storage device 2 that stores the video images that continuously record the situation, and the video images stored in the video image storage device 2 are processed to obtain the video images that should be displayed by the walk-through image presentation device 1. A video image sending device 3 (video image sending server) for generating and sending to the walk-through image presenting device 1 is provided.

The walk-through image presentation device 1 and the video image transmission device 3 are connected to each other via a network 10 as shown in FIG. 2, and each of the walk-through images transmitted from the video image transmission device 3 is walk-through. The video presentation device 1 receives and reproduces.

The walk-through image presentation device 1 of the present embodiment is realized by a program-controlled CPU or the like, and a video image selection unit 11 for selecting a video image to be reproduced according to an input from the user, and a user. Line-of-sight input unit 12 that determines the line-of-sight direction to be observed according to the input of
And a video image decoding unit 13 that decodes the video image received from the video image transmission device 3 in accordance with the encoding method, and a video image display unit that displays the video image decoded by the video image decoding unit 13 to the user. 14 is provided.

The video image transmitting apparatus 3 of this embodiment is
A video image reading unit 31, which is realized by a program-controlled CPU, holds the video image name received from the walkthrough image presentation device 1 and continuously reads out the corresponding video image from the video image storage device 2, and a walkthrough. A cut-out part calculation unit 32 that calculates which part in the video image is cut out and displayed according to the line-of-sight direction received from the image presentation device 1, and the cut-out part calculation unit 32.
The cut-out part storage unit 33 that stores the cut-out part information S3 calculated in step 3, and the part to be displayed on the walk-through image presentation device 1 from the video image read by the video image reading unit 31 according to the cut-out part stored in the cut-out part storage unit 33. Block cutout unit 34 for cutting out
Equipped with.

Next, the operation of the video image presentation system according to this embodiment will be described.

In the video image storage device 2, a video image recording a state of walking through a virtual space constructed by using a three-dimensional computer graphics technique, or a video image recording a state of walking through an actual street. Is stored. The video image stored in the video image storage device 2 has a wide field of view including the vertical and horizontal ranges in which the user moves the line of sight.

First, the flow of the walkthrough image display will be described with reference to the flowchart of FIG. 3 showing the processing of the video image transmitting apparatus 3.

The video image selection section 11 determines a video image (video image name) to be reproduced according to an instruction input from the user. When deciding the video image,
For example, a method of presenting a list of video image names and selecting a video image name from the list, or a URL of the video image name
Examples such as a determination method for directly designating
Any determination method may be used. The video image name selected by the video image selection unit 11 is transmitted to the video image transmission device 3 as the video image information S1.

In the video image sending device 3, the sent video image information S1 is transferred to the video image reading section 31.
The video images stored in the video image storage device 2 are sequentially read out frame by frame in accordance with the video image information S1 (step 301), and are sent to the block cutout unit 34.

When the user's line-of-sight direction is input from the line-of-sight method input unit 12 of the walk-through video presenting device 1 (step 302), the input line-of-sight direction information S2 (that is, the direction in which the user's line of sight is facing) ) And the display area information S5 indicating the shape and size of the video image display area acquired from the video image display unit 14, the cutout portion calculation unit 32 cuts out which portion of the video image and displays it as a walkthrough image. Cutout portion information S3 that specifies whether or not to perform is calculated (step 303).

In the line-of-sight direction input unit 12, the direction in which the user's line of sight is facing (line-of-sight direction) is input. The line-of-sight direction is a value that indicates which part of the video image the user is gazing at, and can be specified by any method, but as a difference value (number of pixels in the horizontal and vertical directions) from the center of the video image. A method of specifying is given as an example. The line-of-sight direction is sent from the line-of-sight direction input unit 12 to the cut-out site calculation unit 32 as line-of-sight direction information S2.

The line-of-sight direction input unit 12 is, for example, the video image display unit 1 of the walk-through image presentation device 1.
It is also possible to display it on the display screen of No. 4 by displaying a cross direction instruction key or the like which the user operates to input the line-of-sight direction.

On the other hand, the cut-out region storing unit 33 stores the cut-out region information S3 calculated by the cut-out region calculating unit 32.
Is stored (step 304).

Further, when the line-of-sight direction is not input from the line-of-sight method input unit 12 (in the initial state or when the line-of-sight direction is not changed) (step 302), the cut-out portion calculation unit 32
The cut-out site information is not calculated in step S1, but the cut-out site information stored in advance in the cut-out site storage unit 33 as an initial value, or the cut-out site storage unit 3 previously calculated.
The cut-out site information stored in 3 is sent to the block cut-out unit 34 described later.

The cut-out portion storage unit 3 is used as an initial value.
The cut-out part information stored in 3 is, for example, information indicating a predetermined range (entire or part) of the video image.

As the cut-out part information S3 stored in the cut-out part storage unit 33, the coordinate value (pixel value) in the video image of the part to be cut out, the ratio of the part to be cut out in the video image, and the like can be considered. Other information may be used as long as it is information that can specify the cutout portion.

In the block cutout unit 34, for each frame of the video image sent from the video image reading unit 31, the video image to be displayed is displayed according to the cutout region information S3 stored in the cutout region storage unit 33. A cutout process is performed (step 305). The cut-out video image is sent to the walk-through image presentation device 1 in the form of streaming video as a walk-through image (step 306).

In the walk-through image presentation device 1, the video image display unit 14 appropriately decodes the received walk-through image (video image) according to the encoding method.
Send to. The video image display unit 14 presents the video image to the user.

Referring to FIGS. 4 and 5, the video image storage device 2 is shown.
The relationship between the video image stored in the and the walk-through image actually presented by the user is shown.

The video image 101 shown in FIG. 4 is one scene of the video image stored in the video image storage device 2. The video image sending device 3 determines the cut-out range 102 (cut-out part information) according to the direction of the user's line of sight, cuts out the video image 103 as shown in FIG. 5, and sends it to the walk-through video presenting device 1. The walk-through image presentation device 1 presents the video image 103 to the user as a walk-through image.

Here, an example of a video image cutout method in the block cutout unit 34 will be described for each type of video image stored in the video image storage device 2.

(1) When the video image stored in the video image storage device 2 is in an uncompressed format, the following method can be considered.

The data of each frame sent from the video image reading unit 31 to the block cutout unit 34 is
It is a sequence of uncompressed bitmap images. In this case, the part stored in the cut-out part storage unit 33 is cut out from the transmitted bitmap image of each frame.

(2) When the video image stored in the video image storage device 2 is an MPEG4 video image, the following method can be considered.

The MPEG4 video image is encoded in units of 16 × 16 pixel blocks (macroblocks). Therefore, if the cutout unit is to be cut out in units of macroblocks, decoding for each frame,
It is possible to perform the clipping process without clipping and re-encoding. In order to reduce the amount of calculation at the time of clipping, it is recommended that the clipping process be performed in macroblock units in the case of MPEG4 video images. In addition, M
As with the PEG4 format, the video image divided into macro blocks and encoded includes MPEG2 and H.264. 263, etc. In the case of these video images, the clipping process is performed in the same method as the MPEG4 video image.

(3) If the video image stored in the video image storage device 2 does not meet the above (1) and (2), the block cutout unit 34 decodes the sent video image. The uncompressed image is cut out, and the encoding process is performed again.

Next, the procedure for changing the cut-out portion will be described.

When the line-of-sight direction input unit 12 changes the line-of-sight direction in accordance with an instruction input from the user,
The changed line-of-sight direction information S2 is the cut-out portion calculation unit 3
Sent to 2. In the cut-out part calculation unit 32, the cut-out part information S is sent according to the sent change information of the line-of-sight direction.
3 is calculated, and the information about the cutout portion stored in the cutout portion storage unit 33 is updated (step 3 in FIG. 3).
03, 304). When the cutout portion information S3 stored in the cutout portion storage unit 33 is changed, the range of the video image cut out from the video image by the block cutout unit 34 is changed.

As a result, the image part cut out from the video image as a walk-through image is changed.
It is possible to obtain the same change in the visual field as when the line-of-sight direction is changed, and it is possible to change the video image according to the change of the line-of-sight direction of the user.

FIGS. 6 to 8 are schematic diagrams showing how the walk-through image changes when the cutout portion is changed. Here, for simplification of description, it is illustrated that the same video image is always sent from the video image reading unit 31 to the block cutout unit 34.

In FIG. 6, a video image 111 is a video image sent from the video image reading section 31. Now, it is assumed that the cutout part storage unit 33 stores the cutout part information S3 indicated by the part 112. At this time, the walk-through image cut out by the block cut-out unit 34 and displayed on the walk-through image presentation device 1 becomes the video image 113 shown in FIG. 7.

Next, it is assumed that the cutout portion stored in the cutout portion storage unit 33 is changed to the cutout portion indicated by the portion 114. At this time, the walk-through image cut out by the block cut-out unit 34 and displayed on the walk-through image presentation device 1 is the video image 11 shown in FIG.
It becomes 5.

By changing the walk-through image presented by the walk-through image presenting apparatus 1 from the video image 113 to the video image 115, the user can obtain a display as if the direction of the line of sight was moved to the right. .

Here, when changing the cut-out region information S3 stored in the cut-out region storing unit 33 in the cut-out region calculating unit 32, the cut-out region before the change is not suddenly changed to the cut-out region after the change. It is also possible to consider an embodiment in which the cut-out region calculation unit 32 has a function of changing the cut-out region stepwise so that the walk-through image presented to the user can be smoothly moved.

FIG. 9 is a schematic diagram for explaining the change of the cutout portion when the cutout portion calculating unit 32 having this function is used. In FIG. 9, it is assumed that the cut-out part 116 is set for the video image 111 in the initial state.

At this time, when the cut-out region stored in the cut-out region storing unit 33 is changed to the region 117 by the signal from the line-of-sight direction input unit 12, the cut-out region calculating unit 32 changes the cut-out region to the regions 118 and 119. 1
It is sequentially changed to 20, 117 and 4 steps. That is, by the time the video image of the part 117 is presented, the parts 118,
Like 119 and 120, video images whose positions are gradually moved are sequentially presented. As a result, the walk-through image displayed by the walk-through image presentation device 1 can be smoothly scrolled.

In this example, the case of changing the number of steps to four has been described, but the number of steps may be any number.

As described above, in the video image presentation system of the present embodiment, the video image stored in the video image storage device 2 is a wide-field video image including the vertical and horizontal ranges in which the user moves his or her line of sight. In the video image sending device 3, a part of the wide-view video image is cut out as a walk-through image and sent out, and when the user performs an operation of changing the line-of-sight direction, a new line-of-sight direction is displayed each time. By changing the image cut-out position in the area centered on, it is possible for the user to walk through and simultaneously change the line-of-sight direction vertically and horizontally.

Next, a second embodiment of the present invention will be described. FIG. 10 is a block diagram showing the configuration of the video image presentation system according to the second embodiment of the present invention.

Referring to FIG. 10, the video image presentation system of the present embodiment is a walkthrough image presentation device (client) 1 for presenting a walkthrough image to a user.
a, a video image storage device 2 that stores a video image that continuously records a state of moving in a street or a building, and a walkthrough image presentation by processing the video image stored in the video image storage device 2. A video image sending device (server) 3 for generating a video image to be displayed by the device 1a and sending it to the walkthrough image presentation device 1a is provided. Also in this embodiment, the video image transmitting device 3 and the walk-through image presenting device 1a are connected to each other via the network 10 as shown in FIG.

The walk-through image presentation device 1a of this embodiment determines a video image selection unit 11 for selecting a video image to be reproduced according to an input from the user, and a line-of-sight direction to be observed according to an input from the user. Line-of-sight direction input unit 12, and display part calculation unit 1 for calculating a part to be displayed according to the line-of-sight direction determined by line-of-sight direction input unit 12
5, a display region storage unit 16 that stores the cut-out region information S3 calculated by the display region calculation unit 15, a video image decoding unit 13 that decodes the video image received from the video image transmission device 3 according to the encoding format, A display portion cutout unit 17 that cuts out a portion to be displayed according to the display portion stored in the display portion storage unit 16 from the video image decoded by the video image decoding unit 13.
And a video image display unit 14 for displaying the video image cut out by the display portion cutout unit 17 to the user.

The video image transmitting apparatus 3 of this embodiment is
A video image reading unit 31 that holds the video image information S1 (video image name) received from the walkthrough image presentation device 1a and continuously reads out the corresponding video image from the video image storage device 2, and a walkthrough image presentation. A cutout portion calculation unit 32 that calculates a portion to be sent to the walk-through image presentation device 1a according to the display portion calculated by the device 1a, a cutout portion storage unit 33 that stores the cutout portion calculated by the cutout portion calculation unit 32, and a cutout portion The block cutout unit 34 cuts out a region to be transmitted to the walk-through image presentation device 1a from the video image read out by the video image reading unit 31 according to the cutout region stored in the storage unit 33.

Next, the operation of the video image presentation system according to this embodiment will be described.

In the video image storage device 2, a video image recording a state of walking through a virtual space constructed by using three-dimensional computer graphics technology, or a video image recording a state of walking through an actual street. Is stored. The video image stored in the video image storage device 2 has a wide field of view including the vertical and horizontal ranges in which the user moves the line of sight.

First, the flow of the walkthrough image display will be described with reference to FIG. 11 showing the process of the video image transmitting device 3 and the flowchart of FIG. 12 showing the process of the walkthrough image presenting device 1a.

The video image selection section 11 determines a video image (video image name) to be reproduced according to an instruction input from the user. When deciding, for example, the first
As in the embodiment described above, the method of determining the video image name from the list of video image names and the UR of the video image names are determined.
A determination method of directly designating L can be considered. The video image name selected by the video image selection unit 11 is transmitted to the video image transmission device 3 as the video image information S1.

In the video image sending device 3, the sent video image information S1 is stored in the video image reading section 31.
The video images stored in the video image storage device 2 according to the video image information S1 are sequentially read out frame by frame (step 1101), and are sent to the block cutout unit 34.

On the other hand, in the walk-through image presenting apparatus 1a, the direction of the user's line of sight (line of sight) is input by the line-of-sight direction input unit 12 (step 120).
1). The line-of-sight direction is a value that indicates which part of the video image the user is gazing at, and can be specified by any method, but as a difference value (number of pixels in the horizontal and vertical directions) from the center of the video image. The method of expressing is mentioned as an example.
The line-of-sight direction is the line-of-sight direction input unit 1 as line-of-sight direction information S2.
2 to the display part calculation unit 15.

In the display part calculation unit 15, based on the line-of-sight direction information S2 received from the line-of-sight direction input unit 12, and the display region information S5 indicating the shape and size of the video image display region acquired from the video image display unit 14. , Display part information S
4 is calculated (step 1202), and the display part storage unit 16 is calculated.
(Step 1203). Display site information S4
Is represented by a pair of the viewing direction and the shape and size of the video display area.

At the same time, the display portion calculation unit 15 calculates the circumscribed rectangle of the display portion information as the display portion circumscribed rectangle information (step 1202), and the cut portion of the video image transmitting apparatus 3 as the display portion circumscribed rectangle information S6. It is sent to the calculation unit 32 (step 1203). The display site circumscribed rectangle information S6 is composed of the line-of-sight direction and the vertical and horizontal sizes of the circumscribed rectangle.

In the cut-out part calculating section 32 of the video image sending device 3, when the display part circumscribing rectangle information S6 is sent from the display part calculating part 15 (step 1102), the sent display part circumscribing rectangle information S6 is sent. In accordance with a predetermined enlargement ratio, calculation is performed to enlarge by α times in the horizontal direction and β times in the vertical direction (step 1103), and stores it as cutout portion information S3 in the cutout portion storage unit 33 (step 1104). Here, the enlargement ratios α and β can be set to arbitrary values of 1 or more, but values of about 1.1 to 1.5 are desirable.

If the display area circumscribing rectangle information S6 is not input from the display area calculating unit 15 (in the initial state or when the line-of-sight direction is not changed) (step 1102), the cut-out area calculating unit 32 calculates the cut-out area information S3. However, the cutout portion information stored in advance in the cutout portion storage unit 33 as the initial value, or the cutout portion information S3 previously calculated and stored in the cutout portion storage unit 33 is sent to the block cutout unit 34 described later. To be

The cut-out portion storage unit 3 is used as an initial value.
The cut-out part information stored in 3 is, for example, information indicating a predetermined range (entire or part) of the video image.

FIG. 13 is a schematic diagram showing the relationship among the line-of-sight direction, display part information, display part circumscribed rectangle information, and cut-out part information.

In the schematic diagram of FIG. 13, a point 200 represents the line-of-sight direction, an area 201 represents display part information, and an area 2
Reference numeral 02 represents display area circumscribing information which is a circumscribed rectangle of the area 201, and area 203 represents a cutout portion obtained by multiplying the area 202 by α times and β times in the horizontal direction and the vertical direction, respectively.
Although the video image display area is illustrated as an elliptical shape in the schematic diagram of FIG. 13, the shape of the video image display area may be another shape such as a rectangle.

On the other hand, in the block cutout unit 34 of the video image sending device 3, the cutout portion information S3 stored in the cutout portion storage unit 33 for each frame of the video image sent from the video image reading unit 31. According to the above, the video image to be displayed is cut out (step 1105). The cut-out video image is sent as a walk-through image to the video image decoding unit 13 of the walk-through image presentation device 1a in the form of streaming video (step 1106). At the same time, the cutout site information S3 is sent from the block cutout unit 34 to the display site cutout unit 17 (step 110).
7).

In the video image decoding unit 13 of the walk-through image presentation device 1a, each frame of the video image sequentially sent as a video stream from the block cutout unit 34 is decoded according to its encoding format, and the decoded video is obtained. Video display part cutout part 1
Send to 7.

In the display portion cutout unit 17, the cutout portion information S3 sent from the block cutout unit 34.
Based on the display part information S4 stored in the display part storage unit 16 and which part should be displayed from the video image sent from the video image decoding unit 13 (step 1204). The video image to be displayed is cut out (step 1205). The cut-out video image is sent to the video image display unit 14.

The video image display unit 14 displays the video image as a walkthrough image (step 120).
6).

Here, the above-mentioned display portion cutout unit 17
The contents of the process will be described in detail with reference to the flowchart of FIG.

When a video image is received from the video image decoding unit 13 (step 1401), the display part storage unit 1
The display part information S4 is acquired from 6 and the cut-out part information S3 is acquired from the block cutout part 34 (step 1402).

Next, by comparing the display part information S4 with the cut-out part information S3, the cut-out part information S
It is determined whether the area indicated by the display part information S4 is completely included in the area indicated by 3 (step 14).
03).

If it is completely included, the area indicated by the display part information S4 is cut out from the received video image (step 1404), and the cut out video image is sent to the video image display unit 14 (step 1405).

Cutout portion information S3 in step 1403.
If the area indicated by the display portion information S4 is not completely included in the area indicated by, the area indicated by the display portion information S4 is changed according to a predetermined rule described below (step 1406), and the changed display portion is displayed. The video image received is cut out based on the information.

The display part information S4 is changed in accordance with the following rules (1) and (2). (1) The shape and size of the area are not changed. (2) The position of the area is changed to a position that is within the area indicated by the cut-out portion information and that is closest to the area indicated by the display portion information.

15 to 17, the video image stored in the video image storage device 2 and the block cutout unit 34 are shown.
From the video image decoding unit 13 to the display portion cutout unit 17, and the video image display 14
The above shows the relationship with the walk-through video that the user actually observes.

The video image 211 is a scene of the video image stored in the video image storage device 2. Range 2
Reference numeral 12 indicates a range indicated by the display portion information S4 calculated by the display portion calculating unit 15, and range 213 indicates the cut-out portion information S3 calculated by the cut-out portion calculating unit 32.
Is the range indicated by (FIG. 15). Video footage 214
Is a video image cut out by the block cutout unit 34 according to the cutout portion information S3 indicating the range 213 (FIG. 16).

In the block cutout unit 34, the range 213
16 is cut out from the video image 211 shown in FIG. The video image 215 is a video image obtained by cutting out the video image 214 as a walk-through image by the display portion cutout unit 17 (FIG. 17).

In this example, since the range 212 of the surface part information S4 is completely included in the range 213 of the cut-out part information S3, the display part cut-out part 17 cuts out the video image 215 based on the range 212. . The video image 215 is presented to the user by the video image display unit 14.

Here, the clipping performed by the block clipping unit 34 when the video image stored in the video image storage device 2 is encoded in the MPEG4 format will be described.

Video images encoded in the MPEG4 format are encoded in units of blocks of 16 × 16 pixels (called macroblocks). Therefore, if the block cutout unit 34 defines that the cutout should be performed in units of macroblocks, the cutout process can be performed without decoding, cutout, and re-encoding for each frame, which reduces the amount of calculation and speeds up the process. Can be planned.

Therefore, in the case of an MPEG4 format video image, the block cutout unit 34 enumerates the macroblocks including the region indicated by the cutout portion stored in the cutout portion storage unit 33, and enumerates all the listed macros. I will cut out a block. That is, an area slightly larger than the area indicated by the cut-out site information S3 stored in the cut-out site storage unit 33 is cut out.

FIG. 18 and FIG. 19 are schematic diagrams showing cutout areas when a video image is cut out in macroblock units.

The area 221 shows the area of the video image stored in the video image storage device 2 at a certain time. Further, an area 222 shows an area of the cutout portion information S3 stored in the cutout portion storage unit 33. When the macro block of the video image and the cutout area have a relationship as shown in FIG. 18, the area of the video image actually cut out by the block cutout unit 34 is as shown in FIG.
The area 224 shown in FIG. That is, in this example, the macroblocks MB5, MB6, MB7, MB9, MB10, M including the area 222 of the cut-out part information S3 are included.
A region 224 surrounded by B11 is cut out.

It should be noted that, in the same manner as the MPEG4 format, the video image divided into macroblocks and encoded is H.264.
263, etc. Even for these video images, MP
The processing can be performed in the same manner as the EG4 video image. The video image stored in the video image storage device 2 is preferably encoded in this type of format.

Next, the processing procedure when the line-of-sight direction input to the line-of-sight direction input unit 12 is changed will be described.

In the line-of-sight direction input unit 12, when the line-of-sight direction is changed according to an instruction input from the user,
The changed gaze direction information S2 is sent to the display part calculation unit 15. The display part calculation unit 15 performs the display part information S according to the above-mentioned procedure according to the sent change information of the line-of-sight direction.
4. The display area circumscribed rectangle information S6 is calculated (step 1202 in FIG. 12) and sent to the display area storage unit 16 and the cutout area calculation unit 32 (step 120 in FIG. 12).
3).

The cut-out portion calculation unit 32 calculates the cut-out portion information S3 according to the above-described procedure in accordance with the sent display portion circumscribed rectangle information S6 (step 110 in FIG. 11).
3) Change the cut-out site. Cutout site information S
When 3 is changed, the range of the video image cut out from the video image by the block cutout unit 34 is changed.

On the other hand, as the display part information S4 stored in the display part storage section 16 is changed, the video image cut out by the display part cutout section 17 is changed. As a result, the image part that is cut out from the video image as a walk-through image is changed, so that the same change in the field of view as when changing the line-of-sight direction is obtained, and the video image corresponding to the change of the line-of-sight direction of the user is obtained. Can be changed.

21 to 23 are schematic diagrams showing how the walk-through video changes when the line-of-sight direction is changed. Here, for simplification of description, it is illustrated that the same video image is always sent from the video image reading unit 31 to the block cutout unit 34.

In FIG. 21, a video image 251 is a video image sent from the video image reading section 31. When the part indicated by the display part information S4 stored in the display part storage 16 is the part 252 and the part indicated by the cut-out part information S3 stored in the cut-out part storage 33 is the part 253, The video image finally displayed on the video image display unit 14 via the block cutout unit 34 and the display portion cutout unit 17 is shown in FIG.
The video image 254 shown in FIG.

Next, the line-of-sight direction is changed in the line-of-sight direction input unit 12, and as shown in FIG.
The portion indicated by the display portion information S4 stored in 6 is the portion 255, and the portion indicated by the cut-out portion information S3 stored in the cut-out portion storage unit 33 is the portion 256.
And it was changed. At this time, the block cutout unit 3
4, and the video image finally displayed on the video image display unit 14 via the display portion cutout unit 17 becomes a video image 257 shown in FIG.

Finally, the line-of-sight direction is changed in the line-of-sight direction input unit 12, and as shown in FIG. 21, the region indicated by the display region information S4 stored in the display region storing unit 16 is the region 258 and the cut-out region. The part indicated by the cut-out part information S3 stored in the storage unit 33 is the part 25.
Assume that the number is changed to 9. At this time, the video image finally displayed on the video image display unit 14 via the block cutout unit 34 and the display portion cutout unit 17 is the video image 260 shown in FIG.

As shown in FIGS. 21 to 24, the walk-through image presented by the walk-through image presentation device 1a changes from the video image 254 to the video image 257, and further to the video image 260, so that the user can It is possible to obtain a display as if the line of sight was moved to the right.

FIGS. 25 and 26 are schematic diagrams showing other examples of changes in the walk-through image when the line-of-sight direction is changed. Here, for simplification of description, it is illustrated that the same video image is always sent from the video image reading unit 31 to the block cutout unit 34.

In FIG. 25, a video image 251 is a video image sent from the video image reading section 31. Now, it is assumed that the part indicated by the display part information S4 stored in the display part storage 16 is the part 271, and the part indicated by the cut-out part information S3 stored in the cut-out part storage 33 is the part 272.

In this case, the part 271 indicated by the display part information S4 is the part 271 indicated by the cut-out part information S3.
Since it is not completely included in 2, the display part cutout part 1
7, the display part information S4 is changed according to the rule described above. As a result, the part indicated by the changed display part information S4 is the part 273 in FIG.

As a result, the video image finally displayed on the video image display unit 14 via the display portion cutout unit 17 becomes the video image 274 shown in FIG.

As shown in FIGS. 25 and 26, the video image sent to the walk-through image presentation device 1a is stored in the display part storage unit 16 even when the update of the cut-out part of the video image is delayed. Since the size is larger than that of the displayed portion, the line-of-sight movement can be performed without delaying the cutout by the display portion cutout unit 17.

When calculating the display region in the display region calculating unit 15, the function of changing the display region step by step is not used instead of suddenly changing the display region before the change to the display region after the change. By providing the calculation unit 15,
An embodiment is also conceivable in which the change of the walk-through image due to the change of the line of sight can be smoothly moved.

FIG. 20 is a schematic diagram for explaining the change of the display area when the display area cutout unit 17 having this function is used. In FIG. 20, it is assumed that a display part 242 is set for the video image 241 in the initial state.

At this time, when changing the display region stored in the display region storing unit 16 to the region 243 by the signal from the line-of-sight direction input unit 12, the display region calculating unit 15 changes the display region 242 to the region 244. 245, 246, 2
Change to 43 and 4 steps. That is, by the time the image of the part 243 is presented, the parts 244, 245, 246 are displayed.
As described above, images whose positions are moved little by little are sequentially presented. As a result, the walk-through image displayed by the walk-through image presentation device 1a can be smoothly scrolled. In this example, the case of changing the number of steps to four has been described, but the number of steps may be any number.

Here, when implementing the video image presentation system of the present embodiment, the walk-through image presentation device 1a and the video image transmission device 3 are connected via a network such as LAN or WAN. Examples are possible.

In the case of this embodiment, the video image transmitting device 3 and the video image storing device 2 are installed on the server machine, and the walk-through image presenting device 1a is provided on the client machine. It is possible to provide a service to the client machine of.

In this way, the walk-through image presentation device 1
When a is connected to the video image transmitting device 3 via a network, a slight time delay occurs when transmitting the display area circumscribed rectangle information S6 from the display area calculating unit 15 to the cutout area calculating unit 32. Due to this,
It is expected that the change of the part of the video image sent from the block cutout unit 34 will be delayed.

That is, although the display part information S4 stored in the display part storage section 16 is updated, the part of the video image sent from the block cutout section 34 may remain unchanged. In such a situation, there is a possibility that the display site information S4 will not be included in the cutout site information S3. In that case, the display site cutout unit 1
7 changes the display part information S4 as described with reference to FIG. 14 and cuts out.

Even when the update of the part of the video image sent in this way is delayed, in the video image presentation system according to the present embodiment, the video image sent to the walk-through image presentation device 1a (that is, Since the video image cut out by the block cutout unit 34 has a size larger than that of the display part stored in the display part storage unit 16, it is possible to realize the line-of-sight movement without delay in the cutout by the display part cutout unit 17. Is possible.

As described above, in the video image presentation system according to the present embodiment, in addition to the effects of the first embodiment, one video image transmission device 3 provides a plurality of walk-through image presentation devices 1a. , The video image can be efficiently distributed via the network. For example, by installing the video image transmission device 3 and the video image storage device 2 on the server machine and providing the walk-through image presentation device 1a on the client machine, the video image can be transmitted from one server machine to a plurality of client machines. Services can be provided efficiently.

Next, a third embodiment of the present invention will be described. FIG. 27 is a block diagram showing the configuration of the video image presentation system according to the third embodiment of the present invention.

The video image presentation system of this embodiment has the configuration of the cutout portion calculating section 32 in addition to the configuration of the video image presentation system of the second embodiment shown in FIG.
The video image sending device 3a is additionally provided with an enlargement ratio changing unit 35 that changes the enlargement ratio that is set to enlarge the display area circumscribed rectangle information S6.

The enlargement ratio changing unit 35 changes the value of the enlargement ratio for enlarging the display region circumscribed rectangle information S6 stored in the cut region calculating unit 32 in accordance with a signal from another module. When the enlargement ratio is changed, the size of the part indicated by the cut-out part information S3 calculated by the cut-out part calculator 32 is changed. Then, the size of the video image cut out by the block cutout unit 34 is changed.

That is, the enlargement ratio changing unit 35 causes the video image transmitting device 3a to change the walk-through image presenting device 1
the size of the video image sent to a, that is, the walkthrough image presentation device 1a from the video image transmission device 3a.
Since it is possible to change the amount of data sent to, it is possible to change the network bandwidth required for the video image presentation system.

As one example of the video image presentation system according to this embodiment, the enlargement ratio is changed according to the number of walk-through image presentation devices 1a connected to one server machine having the video image transmission device 3a. Examples are given below.

In this embodiment, when the number of walk-through video presenting devices 1a is large, the enlargement ratio is reduced to reduce the network band used by one session. When the number of sessions is small, the enlargement ratio is increased and a large-sized video image is transmitted to the walk-through image presentation device 1a, which can immediately respond to the gaze direction change request received by the gaze direction input unit 12. To do so.

However, in order to make the effect of the second embodiment effective even when the enlargement ratio is reduced, it is preferable to keep the cutout region as large as possible by reducing the reduction amount of the enlargement ratio as much as possible.

As described above, in the present embodiment, in addition to the effects of the first and second embodiments, by providing the enlargement ratio changing unit 35 for changing the enlargement ratio, for example, a video image transmitting device It is possible to calculate the network bandwidth available on the server machine side running 3a at any time and deliver the video image with a data communication amount adapted to the bandwidth.
The video image presentation system can be operated in various environments.

Next, a fourth embodiment of the present invention will be described. FIG. 28 is a block diagram showing the configuration of the video image presentation system according to the fourth embodiment of the present invention.

In the fourth embodiment, in addition to the configuration of the video image presentation system in the third embodiment shown in FIG. 27, the quality of the video image transmitted to the walk-through image presentation device 1b and its QoS for requesting the change of the enlargement ratio to the enlargement ratio changing unit 35 based on the processing situation and the like.
The control unit 18 is newly provided.

The following example can be given as one example of the video image presentation system according to the present embodiment.

When the QoS control unit 18 monitors the video image sent from the video image sending apparatus 3a for data delay, video frame loss, etc., and data delay, video frame loss, etc. occur at a certain frequency. To
The enlargement ratio changing unit 35 is instructed to reduce the enlargement ratio.

As a result, the size of the video image sent from the video image sending device 3a to the walkthrough image presenting device 1b becomes smaller, so that the video image is sent between the video image sending device 3a and the walkthrough image presenting device 1b. The network bandwidth required to send and receive video is reduced. That is, it becomes possible to prevent data delay and video frame loss.

Another example of the video image presentation system according to this embodiment is as follows.

The QoS control unit 18 monitors the processing status in the display portion cutout unit 17, and displays the display portion cutout unit 1.
When the processing in 7 is lower than the frame rate of the video image, the magnification rate changing unit 35 is instructed to reduce the magnification rate.

As a result, the size of the video image sent from the video image sending device 3a to the walk-through image presenting device 1b becomes smaller, so that the display portion cutout unit 1 is displayed.
The processing in 7 is reduced. That is, it is possible to prevent the frame rate of the presented video image from decreasing.

As described above, in the video image presentation system according to the present embodiment, in addition to the effects of the first to third embodiments, in the walk-through image presentation device 1b, it is caused by the change of the network condition. When detecting delay of video image from server or frame loss, so-called network QoS (Quality of Service) control is possible to control the amount of data communication of video image to be sent out in order to mitigate those phenomena. Becomes
That is, it becomes possible to provide the video image presentation system to the user with the quality according to the state of the network.

Furthermore, when the processing for the received video image exceeds the processing capability of the walk-through image presentation device 1b, the video image transmission device 3a is similarly instructed to reduce the enlargement ratio of the video image. As a result, the video image can be stably processed.

Next, a fifth embodiment of the present invention will be described. FIG. 29 is a block diagram showing the configuration of the video image presentation system according to the fifth embodiment of the present invention.

The video image presenting system of the present embodiment stores a walkthrough image presenting apparatus 1c for presenting a walkthrough image to a user and a video image continuously recording a state of moving in a street or a building. The video image storage device 2 and the video image stored in the video image storage device 2 are processed to obtain the walk-through image presentation device 1
Walk-through video with two or more n video-video sending apparatuses 3-1, 3-2, ..., 3-n that generate video video to be displayed in c and send it to the walk-through video presenting apparatus 1c. While always establishing one virtual connection with the presentation device 1c, the video images from two or more n video image transmission devices 3-1, 3-2, ..., 3-n are selectively switched. Video image switching unit 4 to be sent to the walk-through image presentation device 1c through the same connection.
, N video image transmitting devices 3-1, 3-2, ..., 3
The video image determination unit 5 is provided for instructing the video image switching unit 4 which video image to display from the video images read by -n.

The walk-through image presentation device 1c, the video image storage device 2, and the video image transmission devices 3-1, 3-2, ..., 3-n of the present embodiment are the first, second, third, and third. Any configuration of the fourth embodiment is applied.

That is, as the walk-through image presentation device 1c of the present embodiment, the walk-through image presentation devices 1, 1a, 1b of the first, second, third, and fourth embodiments can be used. Use the video image transmission devices 3 and 3a of the first, second, third, and fourth embodiments as the video image transmission devices 3-1, 3-2, ..., 3-n of the present embodiment. You can Further, the number n of the video image transmitting devices 3-1, 3-2, ..., 3-n of the present embodiment is not particularly limited, and the number can be any number of two or more.

Further, the video image transmitting devices 3-1 and 3-
, ..., 3-n and the video image storage device 2 are provided on the server machine, and the walk-through image presentation device 1c and the video image determination unit 5 are provided on the client machine. The video image switching unit 4 is located on the network independently of the server and the client machine.

Next, the operation of the video image presentation system according to this embodiment will be described with reference to the flowchart in FIG.

First, the video image deciding unit 5 decides a plurality of video image candidates to be displayed by the walk-through image presentation device 1c, and the video image names are assigned to different video image transmission devices 3-1, 3-2 ,. , 3-n (step 3001).

In the video image transmitting devices 3-1, 3-2, ..., 3-n that have received the video image name, the corresponding video image is read out from the video image storage device 2 and cut out by the block cutout unit 34. Preparations for transmission including are made (step 3002). Video image transmission devices 3-1 and 3-
The operations in 2, ..., 3-n are performed according to any of the methods described in the first, second, third, and fourth embodiments.

On the other hand, the video image determining section 5 determines the video image to be displayed first from the video image candidates and transmits it to the video image switching section 4 (step 300).
3). The video image switching unit 4 transmits the video image to the walk-through image presentation device 1c according to the instruction from the video image determining unit 5 (step 3004). The walk-through image presentation device 1c presents the transmitted video image to the user (step 300).
5).

When the video image displayed by the video image determining unit 5 is changed (step 3006), the changed video image name is transmitted to the video image switching unit 4. Upon receiving the signal from the video image determination unit 5, the video image switching unit 4 immediately switches the video image to be transmitted to the walk-through image presentation device 1c to the video image and transmits it (step 3007).

[0184] In the video image presentation system according to this embodiment, the video image after switching is on standby in the state where preparation for transmission has already been completed. Therefore, when switching the video image, the video image before and after the switching is performed. Can be sent continuously without interruption at the joint.

Further, the video image determining unit 5 determines a video image as a candidate at any time according to the situation at that time, and sends it to the video image transmitting devices 3-1, 3-2, ..., 3-n. Communicate the name and instruct to prepare the video image for transmission.
At the same time, with respect to the video image out of the reproduction candidates, the video image transmitting devices 3-1, 3-2, ..., 3-n are instructed to complete the preparation.

Next, some specific examples of this embodiment will be given. First, a first example of the present embodiment will be described.

In this embodiment, the video image determined by the video image determining unit 5 as a candidate is a video image in which the same scene is recorded at different bit rates. Then, a video image having a bit rate suitable for transmission is selected and displayed according to the network environment connecting the video image switching unit 4 and the walk-through image presentation device 1c.

By doing so, in this embodiment, when the available bandwidth of the network is wide, a detailed video image is displayed at a high bit rate, and when the available bandwidth of the network is narrow, The video image can be displayed by immediately switching to the low bit rate video image.

Next, a second example of the present embodiment will be described.

The video image determined as a candidate by the video image determining unit 5 is a video image in which different directions are recorded while moving on the same route. For example, video images recording the front, back, left, and right of the same route are determined as candidates.

Therefore, in this embodiment, the video image to be displayed is appropriately switched by the user's instruction input.
The user can move the line of sight more freely.

Next, a third example of the present embodiment will be described.

FIG. 31 is a diagram for explaining the movement path of the viewpoint in the three-dimensional space. Here, the route 511 has three routes at the point 515, that is, the routes 512, 513, and 5
It is connected to 14. These routes 511, 512, 5
The video images recording the movement of 13, 514 are referred to as video images 521, 522, 523, 524, respectively.

At this time, the video image determination unit 5 determines four videos of the video images 521, 522, 523, and 524 as candidates, and each of them is determined as the video image transmission device 3-1, 3-2, 3-3. 3-4, and the video image transmission devices 3-1, 3-2, 3-3, 3-4 read out the video images and prepare for transmission.

When actually presenting the walk-through image to the user, first, the video image determining unit 5 determines the reproduction of the video image 521 and transmits it to the video image switching unit 4, and the walk-through image presenting device 1c. At, the video image 521 is reproduced. That is, route 5
The user is presented with the appearance of walking on 11.

Next, the video image 521 is reproduced at the point 51.
A route to be passed next to the route 511 by the time of reaching 5 is selected from the routes 512, 513, and 514.

Here, it is assumed that the path 512 is selected and the video image to be reproduced next is the video image 522. The video image name determined by the video image determining unit 5 is transmitted to the video image switching unit 4. In the video image switching unit 4, the video image 52
After playing 1 to the last frame, immediately video image 522
Start playing.

As described above, in this embodiment, it is possible to present the user with a state of continuously and selectively moving on a plurality of routes.

As described above, in the video image presentation system according to the present embodiment, the video image to be displayed is appropriately switched to display at an appropriate bit rate,
It is possible to present a state in which the user can move the line-of-sight direction more freely and a state in which a plurality of routes are continuously and selectively moved.

By combining this embodiment with each of the first to fourth embodiments (that is, the walk-through image presentation device 1c and each video image transmission device 3-1,
By using those of the first to fourth embodiments as 3-2, ..., 3-n), the effects of the first to fourth embodiments can be realized together.

The video image presenting system according to each of the above-described embodiments includes the video image selecting unit 11, the line-of-sight direction input unit 12, the video image decoding unit 13, the video image display unit 14, and the display unit in the walk-through image presenting apparatus 1. Calculation unit 15, display region storage unit 16, display region cutout unit 1
7, the function of the QoS control unit 18, the video image reading unit 31, the cutout region calculation unit 32, the cutout region storage unit 33 in the video image transmission devices 3, 3a, 3-1, 3-2, ..., 3-n. Functions of the block cutout unit 34, the enlargement ratio changing unit 35, the functions of the video image switching unit 4,
The function of the video image determination unit 5 and other functions are, of course, realized by hardware, and the image processing program 90, which is a computer program having each function,
90a, 90b, 90c, 91, 91a can be realized by being loaded in the memory of the computer processing device. This video processing program 90, 90a, 90
b, 90c, 91 and 91a are stored in a recording medium such as a magnetic disk, a semiconductor memory or the like. Then, it is loaded from the recording medium to the computer processing device and controls the operation of the computer processing device to realize the above-described functions.

Although the present invention has been described with reference to the preferred embodiments and examples, the present invention is not necessarily limited to the above-described embodiments and examples, and various modifications can be made within the scope of the technical idea. It can be modified and implemented.

[0203]

As described above, according to the present invention, the following effects can be achieved.

First, playing back a video image of a walkthrough in a virtual space constructed by using three-dimensional computer graphics technology, or a video image of a walkthrough of an actual street. Thus, in the system that presents the movement in the space to the user, the user's free and smooth line-of-sight movement can be realized.

Secondly, in a system in which a plurality of video images are combined to present movement in space to the user, the video image is displayed smoothly without the user being aware of the joint between the video images. be able to. That is,
By smoothing the connection between the video images, the user's natural spatial movement can be realized.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a video image presentation system according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a network configuration to which the video image presentation system according to the first embodiment of the present invention is applied.

FIG. 3 is a flowchart illustrating an operation of the video image sending device of the video image presentation system according to the first embodiment of the present invention.

FIG. 4 is a schematic diagram showing an example of a video image according to the first embodiment of the present invention.

FIG. 5 is a schematic diagram showing a user-presented image by the video image of FIG. 2 according to the first embodiment of the present invention.

FIG. 6 is a schematic diagram showing a video image change when a user's line of sight is moved in the first embodiment of the present invention.

FIG. 7 is a schematic diagram showing a video image change when the user's line of sight is moved in the first embodiment of the present invention.

FIG. 8 is a schematic diagram showing a video image change when a user's line of sight is moved in the first embodiment of the present invention.

FIG. 9 is a schematic diagram showing changes in the cutout portion according to the first embodiment of the present invention.

FIG. 10 is a block diagram showing a configuration of a video image presentation system according to a second embodiment of the present invention.

FIG. 11 is a flowchart illustrating an operation of the video image sending device according to the second embodiment of the present invention.

FIG. 12 is a flowchart illustrating an operation of the walkthrough image presentation device according to the second embodiment of the present invention.

FIG. 13 is a schematic diagram of a relationship among a line-of-sight direction, display site information, display site circumscribed rectangle information, and cut-out site information according to the second embodiment of the present invention.

FIG. 14 is a flowchart illustrating an operation of a display portion cutout unit according to the second embodiment of the present invention.

FIG. 15 is a schematic diagram showing a video image and a user-presented image according to the second embodiment of the present invention.

FIG. 16 is a schematic diagram showing a video image and a user-presented image according to the second embodiment of the present invention.

FIG. 17 is a schematic diagram showing a video image and a user-presented image according to the second embodiment of the present invention.

FIG. 18 is a schematic diagram showing a cutout portion calculating method according to the second embodiment of the present invention.

FIG. 19 is a schematic diagram showing a cutout portion calculating method according to the second embodiment of the present invention.

FIG. 20 is a schematic diagram showing a change in cutout portion according to the second embodiment of the present invention.

FIG. 21 is a schematic diagram showing a video image change when the user's line of sight is moved in the second embodiment of the present invention.

FIG. 22 is a schematic diagram showing a change in video image when a user's line of sight is moved in the second embodiment of the present invention.

FIG. 23 is a schematic diagram showing a change in video image when the user's line of sight is moved in the second embodiment of the present invention.

FIG. 24 is a schematic diagram showing a change in video image when the user's line of sight is moved in the second embodiment of the present invention.

[Fig. 25] Fig. 25 is a schematic diagram showing another example of video image change when the user's line of sight is moved in the second embodiment of the present invention.

[Fig. 26] Fig. 26 is a schematic diagram showing another example of video image changes when the user's line of sight is moved in the second embodiment of the present invention.

FIG. 27 is a block diagram showing a configuration of a video image presentation system according to a third embodiment of the present invention.

FIG. 28 is a block diagram showing a configuration of a video image presentation system according to a fourth embodiment of the present invention.

FIG. 29 is a block diagram showing a configuration of a video image presentation system according to a fifth embodiment of the present invention.

FIG. 30 is a flowchart illustrating an operation of the video image presentation system according to the fifth embodiment of the present invention.

FIG. 31 is an explanatory diagram of a moving path of a viewpoint in a three-dimensional space according to the fifth embodiment of the present invention.

[Explanation of symbols]

1, 1a, 1b, 1c Walk-through image presentation device 11 Video image selection unit 12 Line-of-sight direction input unit 13 Video image decoding unit 14 Video image display unit 15 Display site calculation unit 16 Display site storage unit 17 Display site cutout unit 18 QoS control Part 2 Video image storage device 3, 3a, 3-1, 3-2, ..., 3-n Video image transmission device 31 Video image reading unit 32 Cutout region calculation unit 33 Cutout region storage unit 34 Block cutout unit 35 Enlargement ratio change Part 4 Video image switching unit 5 Video image deciding unit 90, 90a, 90b, 90c, 91, 91a Image processing program S1 Video image information S2 Line-of-sight input information S3 Cutout region information S4 Display region information S5 Display region information S6 Display region circumscribed rectangle information

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G09G 5/36 510 G09G 5/36 520E 520P F term (reference) 5C064 BA07 BB03 BC16 BC20 BD05 BD08 5C082 AA00 BA12 BA46 BB44 BB53 CA32 CA52 CA76 CA81 CB01 CB06 DA73 DA86 MM05 MM10 5E501 AA20 AC37 BA05 CA02 CC08 EB06 FA06 FA14

Claims (34)

[Claims] 1. A video image presentation system for presenting a captured video image, accepting designation of a line-of-sight direction of a user who views the video image, and calculating a cut-out range corresponding to the line-of-sight direction in the video image. A video image presentation system characterized by clipping the video image based on the calculated clipping range and presenting it as a new video image. 2. A video image presentation device for presenting the video image, and a video image transmission device for cutting out the video image and transmitting a new video image to the video image presentation device. The video image presentation system according to claim 1. 3. The video image transmitting device calculates the cutout range in the video image corresponding to the user's line-of-sight direction notified from the video image presentation device, and the calculated cutout of the video image. The video image presentation system according to claim 2, wherein the video image presentation system cuts out based on the range and sends out as a new video image to the video image presentation device. 4. The video image presenting device calculates the cutout range in the video image corresponding to the line-of-sight direction of the user, and based on the cutout range from the video image from the video image sending device. The video image presentation system according to claim 2, wherein a new video image is cut out and presented. 5. The video image sending device calculates a cutout range in the video image corresponding to the line-of-sight direction of the user notified from the video image presenting device, and the screen of the video image is defined. An area of each block in a range including the cutout range in the video image among the blocks divided according to size is cut out as a new video image and sent to the video image presentation device. Item 2. The video image presentation system according to Item 2. 6. The video image presenting device calculates a first cutout range in the video image corresponding to the line-of-sight direction of the user, and the video image transmitting device,
Calculating a second cutout range that is larger than the first cutout range based on the first cutout range;
Based on the second cutout range, the video image is cut out, sent as a new video image to the video image presentation device, and the video image presentation device that has received the new video image causes the new video image to be displayed. 3. The video image presentation system according to claim 2, wherein the video is presented based on the first cutout range. 7. The first cutout area is obtained by the video image presentation device receiving the second cutout range from the image sending device and comparing the first cutout range with the second cutout range. When the range is included in the second cutout range, the new video image is cut out based on the first cutout range, and the first cutout range is not included in the second cutout range. 7. The first cutting range is changed so as to be included in the second cutting range, and the new video image is cut out based on the changed first cutting range. The described video image presentation system. 8. The size of the image of the video image to be transmitted to the video image presenting device by the video image transmitting device to the video image presenting device based on the number of the video image presenting devices to which the video image is transmitted is a predetermined enlargement ratio. 8. The communication amount of the video image is suppressed below a communication band between the video image transmitting device and the video image presenting device by changing the communication amount to 1. The video image presentation system according to one. 9. The video image sending device changes the size of the image of the video image sent to the video image presenting device to a predetermined enlargement ratio based on an instruction from the video image presenting device, The video according to any one of claims 1 to 7, wherein the communication amount of the video image is suppressed below a communication band between the video image transmission device and the video image presentation device. Video presentation system. 10. The video image presenting device instructs the video image transmitting device, based on the reception state of the video image, the size of the image of the video image transmitted by the video image transmitting device. The video image presentation system according to claim 9, characterized in that 11. The video image presenting device specifies to the video image transmitting device the size of the image of the video image transmitted by the video image transmitting device based on the processing status of the received video image. The video image presentation system according to claim 9, wherein 12. The video image presenting apparatus transmits the video image to the video image presenting apparatus via a network.
The video image presentation system according to any one of 1. 13. The video image presenting apparatus compresses the video image to be sent out and sends the compressed video image, and the video image presenting apparatus decodes the compressed video image received from the video image presenting apparatus. The video image presentation system according to claim 12, characterized in that 14. A plurality of the extracted video images are simultaneously received from the video image sending device, and the video images are selectively switched by switching between the video images. The video image presentation system according to claim 12 or 13, further comprising a switching device that transmits the connection without changing the connection. 15. The video image presentation system according to claim 1, wherein the video image is a walk-through image in which a state of moving in a street or a building is recorded. . 16. The video image presentation system according to claim 15, wherein the video image is a wide-field video image including an image in a range in which the user's line of sight moves. 17. A video image transmitting device for transmitting a video image to a video image presenting device, wherein the video image in the video image corresponding to the line-of-sight direction of a user who is watching the video image notified from the video image presenting device. A video image transmitting apparatus comprising: a means for calculating a clipping range; and a clipping means for clipping the clipping range of the video image as a new video image and sending it to the video image presentation device. 18. Within each block obtained by dividing the screen of the video image into predetermined sizes, each of the ranges including the cut-out range calculated corresponding to the direction of the line of sight of the user in the video image. 18. The video image sending device according to claim 17, further comprising means for cutting out the area of the block as a new video image and sending it to the video image presenting device. 19. Means for changing the size of an image of the video image to be sent to the video image presenting device to a predetermined enlargement ratio based on the number of the video image presenting devices to which the video image is sent. 19. The video image transmitting apparatus according to claim 17, wherein the communication amount of the video image is suppressed to be equal to or lower than the communication band with the video image presenting apparatus. 20. The video image presenting device, comprising means for changing a size of an image of the video image sent to the video image presenting device to a predetermined enlargement ratio based on an instruction from the video image presenting device. The video image sending device according to claim 17 or 18, wherein the communication amount of the video image is suppressed below a communication band with the device. 21. The video image transmitting apparatus according to claim 17, further comprising means for transmitting the video image after compressing the video image. 22. The video image transmitting apparatus according to claim 17, wherein the video image is transmitted to the video image presenting apparatus via a network. 23. A video image presenting device for presenting a video image transmitted from a video image transmitting device, comprising means for instructing the video image transmitting device to specify a line-of-sight direction of a user who views the video image, A video image presentation device, characterized in that the video image transmission device receives and presents a new video image clipped from the video image in a clipping range corresponding to the line-of-sight direction. 24. Means for calculating a first cutout range in the video image corresponding to the user's line-of-sight direction, and the calculation made by the video image sending device based on the first cutout range. A cutting means for receiving a new video image cut out in a second cutting range which is larger than the first cutting range, and cutting out the new video image based on the first cutting range and presenting the new video image. 24. The video image presentation device according to claim 23. 25. The cutout means receives the first cutout range from the image sending device, and compares the first cutout range with the second cutout range to determine the first cutout range. The second
, The new video image is clipped based on the first clipping range, and when the first clipping range is not included in the second clipping range, the second video image is clipped. 25. The video image presentation according to claim 24, wherein the first cutout range is changed so as to be included in the cutout range, and the new video image is cut out based on the changed first cutout range. apparatus. 26. A means for instructing the video image sending device to change a size of an image of the video image to a predetermined enlargement ratio based on a receiving state of the video image. Claim 24 or Claim 25
The video image presentation device according to 1. 27. Means for instructing the video image sending device to change the size of the image of the video image to a predetermined enlargement ratio based on the processing status of the received video image. The video image presentation device according to claim 24 or 25. 28. The video image presenting device according to claim 23, wherein the video image is received from the video image transmitting device via a network. 29. The video image presentation according to claim 28, further comprising means for simultaneously receiving a plurality of the video images from the video image sending device and selectively switching each of the video images to present. apparatus. 30. The video image according to claim 23, wherein the video image compressed and transmitted from the video image transmission device is presented after being decoded. Presentation device. 31. By controlling a computer,
In a video image processing program for transmitting a video image from a video image transmission device to a video image presentation device, in the video image corresponding to the line-of-sight direction of a user who views the video image notified from the video image presentation device, A video image processing program having a function of calculating a cutout range and a function of cutting out the video image based on the calculated cutout range and presenting it to the video image presentation device as a new video image. 32. By controlling a computer,
A video image processing program for transmitting and presenting a video image from a video image transmitting device to a video image presenting device, having a function of instructing the video image transmitting device to specify a line-of-sight direction of a user who views the video image. A video image processing program having a function of receiving and presenting a new video image clipped from the video image in a clipping range corresponding to the line-of-sight direction in the video image sending device. 33. A function of calculating a first cut-out range in the video image corresponding to the line-of-sight direction of the user,
A new video image cut out in a second cutout range larger than the first cutout range calculated by the video image sending device is received based on the first cutout range, and the new video image is received. 33. The video image processing program according to claim 32, further comprising a function of cutting out and presenting based on the first cutout range. 34. The cutout function receives the first cutout range from the image sending device, and compares the first cutout range with the second cutout range to determine the first cutout range. When included in the second cutout range, the new video image is cut out based on the first cutout range, and when the first cutout range is not included in the second cutout range, 34. The first cutout range is changed so as to be included in the second cutout range, and the new video image is cut out based on the changed first cutout range. Video image processing program.