JP2003116126A - Signal processing method, signal processing apparatus and signal processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To simply and easily generate an attached information signal to enhance the presence corresponding to a displayed image. SOLUTION: A motion detection block 20 provides detection areas to an image of a target frame of an image signal SDC to detect a motion vector MV of the image by each area. An intermediate image information generating block 25 groups the detection areas into layers on the basis of the motion vector MV, an attached information signal generating block 50 generates a rocking signal SE depending on the motion of the image on the basis of the image signal based on the motion amount of the motion vector MV of a prescribed layer. The bocks 25, 40 generate image signals SDL, SDR independently of the target frame on the basis of the motion amount and the image signal. The image is displayed at a wide image angle on the basis of the image signals SDL, SDR and the presence can be enhanced by rocking the displayed image such as a chair with the attached information signal SE.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a signal processing method, a signal processing device, and a signal processing program. Specifically, using the input image signals of a plurality of frames, the feature amount related to the movement of the image at the predetermined position of the frame of interest is detected,
Generates an additional information signal related to the movement of the image based on the detected feature amount, and generates a peripheral image signal with a view angle that does not exist in the target frame based on the detected feature amount and the input image signals of a plurality of frames. To do.

[0002]

2. Description of the Related Art In recent years, a multi-screen display system, a curved display, a wide-angle display, a head-mounted display and the like have been put into practical use in order to display an image with a high sense of presence. Here, in order to display an image with a high sense of presence using the above-described display, for example, a three-dimensional virtual space is constructed by computer graphics. In addition, a large number of video cameras are used or a wide-angle lens is attached to the video cameras to photograph a wide space, and the photographed images are converted into a flat surface or a curved surface such as a multi-faced display or a head mounted display for display. There is.

In addition to displaying a wide range of images, it is possible to give the sensation of actually riding in a car or the like by performing an operation for enhancing the sense of presence, for example, by swinging a chair in accordance with the image. ing. In this way, as the additional information signal for performing the operation for enhancing the sense of presence, a sensor that detects an angle, an acceleration, or the like at the time of capturing an image is used, and the signal obtained by this sensor is used, or the captured image is detected. It is generated by manual observation by a person.

[0004]

When the sensor is used to generate the additional information signal, it is necessary to obtain the signal from the sensor at the time of photographing and store the signal together with the image signal. Therefore, when the display is performed based on the image signal that has already been photographed, the additional information signal cannot be generated by such a method because there is no signal from the sensor. In addition, when a person observes the photographed image and manually generates the additional information signal, it takes a long time to generate the additional information signal and the cost also increases.

Therefore, the present invention provides a signal processing apparatus and a signal processing method that can easily and easily generate an additional information signal for enhancing the sense of presence in correspondence with an image to be displayed.

[0006]

A signal processing method according to the present invention detects, by using input image signals of a plurality of frames, a feature amount related to a motion of an image at a predetermined position of a frame of interest in the input image signals, and detects the feature amount. Based on the feature amount
An additional information signal related to the movement of an image at a predetermined position is generated, and a peripheral image signal having an angle of view that does not exist in the frame of interest is generated based on the detected feature amount and the input image signals of a plurality of frames. .

Further, the signal processing device uses the input image signals of a plurality of frames to detect a feature amount relating to the movement of the image at a predetermined position of the frame of interest in the input image signals, and the feature detected by the detection means. Existing in the frame of interest on the basis of the additional information signal generating means for generating the additional information signal related to the movement of the image at the predetermined position based on the amount, and the feature amount detected by the detecting means and the input image signals of a plurality of frames. And a peripheral image signal generating means for generating a peripheral image signal having an angle of view.

Further, the signal processing program causes the computer to detect the characteristic amount relating to the movement of the image at the predetermined position of the frame of interest in the input image signal by using the input image signals of a plurality of frames, and the detected characteristic amount. Based on the procedure for generating an additional information signal related to the movement of the image at a predetermined position, and based on the detected feature amount and the input image signals of a plurality of frames, the peripheral image signal of the angle of view that does not exist in the frame of interest is detected. The procedure for generating is executed.

In the present invention, the feature amount related to the movement of the image at the predetermined position of the target frame is detected from the input image signals of the plurality of frames, and the detected feature amount is layered. A swing signal for generating swing according to the motion of the image based on the input image signal is generated based on the characteristic amount of the predetermined layer divided into layers. Also,
Based on the detected feature amount and the input image signal, a peripheral image signal having an angle of view that does not exist in the frame of interest is generated, and the signal level of the swing signal is corrected according to the generated peripheral image signal.

[0010]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described in detail below. FIG. 1 shows the overall configuration of a system using a signal processing device according to the present invention. In this system, for example, three screens are arranged on the front surface and both side surfaces of the user, and each screen 10L, 10C, 1
Images are projected from the projectors 12L, 12C, and 12R corresponding to 0R. This projector 12L, 12C, 12R
Is connected to the signal processing device 15.

The signal processing device 15 stores a captured image signal, for example, an image signal SDC of a front image captured by an in-vehicle camera or the like. The signal processing device 15 supplies the accumulated image signal SDC to the projector 12C to display a front image based on the image signal SDC on the screen 10C located in front as shown in FIG. Further, in the signal processing device 15, from the accumulated image signal SDC of the front image, an image signal indicating an image of an angle of view that is not included in the field of view of the vehicle-mounted camera, for example, left and right side images continuous with the front image ( The peripheral image signals SDL and SDR, which are image signals indicating "peripheral image"), are generated,
The peripheral image signal SDL is supplied to the projector 12L and the peripheral image signal SDR is supplied to the projector 12R.
Therefore, the left side image having continuity with the front image is displayed on the screen 10L located on the left side, and the right side image having continuity with the front image is displayed on the screen 10R located on the right side. Image presentation of corners can be performed. Further, the signal processing device 15 detects a feature amount related to the motion of the image based on the image signal SDC of the front image, and based on this feature amount, an additional information signal related to the motion of the image based on the image signal SDC, for example, a swing. Signal S
E is generated and supplied to the motion base platform 17.

The motion base platform 17 is provided with a base drive unit 170 having a plurality of pistons 171 as shown in FIG. The base 172 is supported by the piston 171 of the base drive unit 170. A chair 173 is fixed to the pedestal 172, and a person can sit on the chair 173. The pistons 171 are adapted to be able to extend and contract along their respective central axes. As the pistons 171 expand and contract, the pedestal 172 swings and the chair 173 fixed to the pedestal 172 swings. Move. The piston 171 is driven based on the swing signal SE.

FIG. 4 shows an outline of the configuration of the signal processing device 15. The image signals SDC of a plurality of frames stored in the front image signal storage area 31 of the storage unit 30 are generated by the one-frame delay processing unit 21, the motion detection processing unit 22, and the intermediate image information generation block 25 of the motion detection block 20. It is supplied to the processing unit 27.

The 1-frame delay processing section 21 delays the image signal SDC by 1 frame and supplies it to the motion detection processing section 22 as an image signal SDCa. The motion detection processing unit 22 sets a plurality of detection areas separated on the side edge side of the front image as shown by the broken line in FIG. 5, and detects the image signal SDC of the frame of interest in this detection area and the previous frame. Image signal S
Comparison with DCa is performed for each detection area, and the motion vector MV indicating the motion of the image is determined for each detection area and supplied to the layer classification unit 26 of the intermediate image information generation block 25 and the additional information signal generation block 50.

The intermediate image information generation block 25 is for generating intermediate image information by projecting an input image of a plurality of frames onto a plane by using the motion vector MV and the image signal SDC. Here, in the layer classification unit 26 of the intermediate image information generation block 25, the motion vector MV of each detection region
When the intermediate image information is generated by determining the motion pattern of the front image from, for example, a layer of a distant view image of a distant subject, a layer of a near view image of a close subject, and a distant view image and a near view image. Layers are set by discriminating how to provide a layer of the middle-view image located between them and a layer different from these layers. For example, when a front image is taken by a vehicle-mounted camera and the vehicle goes straight in one direction, the front image is an image that is sequentially zoomed in. Further, when the vehicle is moving backward, the front image is sequentially zoomed out. Furthermore, when an overtaking vehicle is photographed, the overtaking vehicle is displayed as a zoomed-out image in the zoomed-in image. Further, when a right turn or a left turn is made, for example, the upper side of the front image is horizontally moved and the lower side becomes a zoomed-in image. Therefore, it is assumed that the motion pattern of the front image is determined from the motion vector MV of each detection area, and layer setting is performed based on the determined motion pattern. For example, when it is determined that the motion pattern goes straight in one direction, layers of each image of the distant view, the middle view, and the near view are generated, and when it is determined that the motion pattern is a right turn or a left turn,
In addition to the layers for the distant view, middle view, and near view images, a layer containing images that are moved horizontally is also created. Also,
When there is a motion pattern of an overtaking vehicle, layer setting is performed so that not only the layers of the images of the distant view, the middle view, and the near view but also the layer including the zoom-out image is created.

The layer classification unit 26 classifies, based on the motion vector MV supplied from the motion detection processing unit 22, which layer each detection region set on the side edge of the front image belongs to. . In this layer classification,
Layer classification is performed using layers set according to motion patterns. For example, when a layer including an image that is moved in the horizontal direction and a layer that includes a zoomed-out image are generated, the layers are divided into three layers, that is, a distant view, a middle view, and a near view. I do. By this layer classification, layer classification information LB indicating which detection region belongs to which layer is generated, and the generated layer classification information LB is supplied to the information generation processing unit 27 and the additional information signal generation block 50. To be done.

In the information generation processing unit 27, the layer classification unit 2
Based on the layer classification information LB from 6, the detection regions set in the front image are divided into layers, and the image signals of the detection regions are used in the frame order for each layer to generate an intermediate image signal for each layer. Further, the average value of the motion amount is calculated for each layer from the motion amount of the motion vector MV of the detection area divided into layers. The intermediate image signal GYv generated by the information generation processing unit 27 and the calculated motion amount (average value) MYv for each layer are stored in the storage unit 30 as intermediate image information.
Are stored in the intermediate image information storage area 32.

When displaying an image on the screens 10C, 10R and 10L, the stored image signal SDC is read out and the image based on this image signal SDC is displayed on the screen 10C. The intermediate image information stored in the intermediate image information storage area 32 is read by the read control signal RC from the peripheral image signal generation block 40 connected to the storage unit 30, and the motion amount MYv for each layer is read.
By using the intermediate image signal GYv of each layer in accordance with the above, the images of each layer are pasted in the order of distant view, middle view, and near view. Further, when a layer that does not belong to the three layers of the distant view, the middle view, and the near view is provided, image pasting processing of these layers is also performed, and the periphery showing an image with a view angle that does not exist in the frame of interest of the image signal SDC. Image signal SD
Generates L and SDR. By supplying the peripheral image signal SDL to the projector 12L at the timing corresponding to the image signal SDC of the front image, the left side image can be displayed on the screen 10L continuously with the front image, and the peripheral image signal SDR can be displayed. By supplying the image signal SDC to the projector 12R at a timing corresponding to the image signal SDC, the right side image can be displayed on the screen 10R continuously with the front image.

The additional information signal generation block 50 generates a swing signal SE which is an additional information signal based on the motion vector MV and the layer classification information LB, and supplies it to the motion base platform 17.

Next, each part of the motion detection block 20 will be described in detail. For the sake of simplicity, only the right side surface will be described below, and the left side surface will not be described.

The motion detection block 20 determines the motion vector MV for each detection area as described above. here,
When the front image has the center of movement of the image, that is, when the front image is captured by the in-vehicle camera, the image at time T shown in FIG. 6A is displayed at time T ′ after one frame time has elapsed, for example. 6B, which is substantially the same as the zoom-in operation image centered on the position CP (hereinafter referred to as “image reference position”) as if the image appeared to come out.

Here, as shown in FIG. 6C, the enlargement ratio Z is set to reduce the detection area of the target frame to (1 / Z), and one frame is moved while moving the position of the reduced detection area. The error sum with the previous image signal is calculated. Further, the magnification rate Z is changed, and the error sum is calculated while moving the position similarly. In this way, by detecting the position where the error sum has the minimum value, the motion vector MV of each detection region can be determined for each frame of interest. Further, the enlargement ratio Z when the error sum is the minimum value is the amount of movement.

By the way, when the reduction processing is performed, some pixels have coordinate values that are not integer values in the area.
On the other hand, in the image one frame before which the reduction processing has not been performed, the coordinate values of the pixels in the region are integer values. For this reason,
Linear interpolation is performed on the reduced image to calculate the signal level at the position where the coordinate value is an integer value. For example, as shown in FIG. 7, for an image of Ka × Ka pixels (1 / Z)
Image size is Kb x Kb
When the pixel size is reached, linear interpolation is performed and Ka
From the signal of the × Ka pixel, the signal level of the image in which the number of pixels is “Kb × Kb” is calculated. By calculating the error sum of the calculated signal level and the signal level one frame before at the position corresponding to the reduced image, the motion amount can be accurately determined.

When the image reference position CP is not clear and the center of the detection area ARa is changed from the position Pa1 to the position Pa2 as shown in FIG.
When the error sum has the minimum value when the center of Rb is changed from the position Pb1 to the position Pb2, a point where the direction of the motion vector MV of the detection area ARa and the direction of the motion vector of the detection area ARb intersect is detected. As a result, the image reference position CP can be detected.

When making a right turn or a left turn, for example, the upper image of the front image moves horizontally. For this reason,
When the minimum value of the error sum is not detected even when the zoom-in operation is performed centering on the image reference position CP, the image in the detection area is moved in the horizontal direction to determine the minimum value of the error sum.
At this time, the movement amount of the detection area until the error sum reaches the minimum value can be set as the movement amount of the motion vector MV.

Next, since the image seems to be sucked into the image reference position CP when retreating,
The image at the time of retreat becomes substantially the same as the zoom-out operation image centered on the image reference position CP. Therefore, the enlargement ratio Z is set to "1" or less. That is, when moving backward, the movement of the image is opposite to that when moving forward. Therefore, a plurality of detection areas are set on the side edge side of the previous frame, and each detection area is reduced to (1 / Z). The error sum with the image of the frame of interest is calculated while moving the position of the designated area. Alternatively, each detection area of the frame of interest is reduced to (1 / Z), and the error sum with the image one frame after is calculated while moving the position of this reduced area. Further magnification Z
Is calculated, and the error sum is calculated while moving the position in the same manner. In this way, the motion vector MV at the time of backward movement can also be determined by detecting the position where the error sum is the minimum value.

When the motion vector MV is discriminated as described above, the motion amount of the motion vector MV is small because the image in the distant view has little motion, and the motion amount of the motion vector MV is small in the image in the near view. The amount will increase.

FIG. 9 shows the configuration of the motion detection processing section 22. The image signal SDC is supplied to the size conversion processing section 221, and the image signal SDCa supplied from the 1-frame delay processing section 21 is the sum of errors. It is supplied to the calculation unit 222. The size conversion processing unit 221 divides the side edge portion of the front image into a plurality of detection areas, for example, a plurality of detection areas in units of 16 × 16 pixels and sets the detection areas. Further, in the size conversion processing unit 221, the enlargement ratio Z is set from the search control unit 225 described later, and the image signal FEz obtained by multiplying the image of the detection area by (1 / Z) and the image reference position CP.
The coordinate value Pz converted by multiplying the image of the detection area by (1 / Z) is supplied to the error sum calculation unit 222. The image signal FEz is a signal when the coordinate value is converted into an integer value by interpolation processing as shown in FIG. 7, and the coordinate value Pz is the coordinate when the coordinate value is converted into an integer value by interpolation processing. Value, that is, "K" in FIG.
The pixel position is “b × Kb”.

In the error sum calculation unit 222, the signal at the position indicated by the coordinate value Pz from the size conversion processing unit 221 is selected from the image signal SDCa, and the error sum between the selected signal and the image signal FEz is calculated. It is calculated and notified to the comparison processing unit 223.

The comparison processing unit 223 compares the minimum error sum with the error sum calculated by the error sum calculation unit 222. If the error sum minimum value is not set here, the error sum calculated first is set as the error sum minimum value. When the calculated error sum is smaller than the error sum minimum value, this data error sum is set as a new error sum minimum value, and the data holding unit 2 indicates that the error sum minimum value has been updated.
Notify 24. In addition, the minimum error sum and the error sum calculation unit 2
The search control unit 225 is notified by the signal ES that the comparison with the error sum calculated in step 22 has been completed. The minimum error sum value may be set in advance to a value larger than the calculated error sum.

The data holding unit 224 includes a search control unit 22.
When the enlargement ratio Z is notified from 5 and it is notified that the minimum error sum is updated, the notified enlargement ratio Z
Memorize If the enlargement ratio has already been stored, the stored enlargement ratio is updated with the notified enlargement ratio Z. Further, when the search control unit 225 notifies the completion of the change processing of the enlargement ratio by the signal ET, the stored enlargement ratio is used as the motion amount and the motion vector MV having the image reference position direction as the vector direction is classified into layers. Part 26
Supply to.

In the search control unit 225, the lower limit value of the enlargement ratio is set to "1" and the upper limit value is also set in advance. First, the lower limit value is set as the enlargement ratio Z and the size conversion processing unit 221 and the data are set. Notify the holding unit 224. After that, the comparison processing unit 223 outputs the error sum minimum value and the error sum calculation unit 222.
Each time it is notified that the comparison with the error sum calculated in step 1 is completed, the enlargement factor Z is sequentially increased to increase the size conversion processing unit 22.
1 and the data holding unit 224. After that, the enlargement ratio Z
When reaches the upper limit, the data holding unit 224 is notified of the completion of the enlargement ratio changing process.

Further, when the minimum error sum obtained by the comparison processing unit 223 is not small, that is, when an image equal to the side edge portion of the front image cannot be detected, the search control unit 225 sets the enlargement ratio Z to "1". To the size conversion processing unit 221, and the image signal FEz in the detection area is calculated as the error sum calculation unit 2
22. Further, the control signal RP is supplied to the error sum calculation unit 222, and the signal at the position where the detection area of the image signal FEz is moved by a predetermined amount in the horizontal direction is selected from the image signal SDCa. Then, the image signal S is generated by the control signal RP.
By moving the position selected from DCa in the horizontal direction and determining the error sum minimum value, the motion vector MV of the image to be moved in the horizontal direction can also be obtained. Further, although not shown, a signal after one frame is supplied to the error sum calculation unit 222, or a detection region is set in the image one frame before and the signal of the image of the target frame is supplied to the error sum calculation unit 222. Thus, the amount of movement of the image that is sucked into the image reference position CP can also be determined.

In this way, the search direction is set to the image reference position CP.
Direction or the horizontal direction to detect the image position of the image in the detection area and the image of another frame where the error sum is the minimum. The motion vector MV can be correctly obtained even for an image of a person, a person, or an overtaking vehicle.

The layer classification unit 26 determines from the motion vector MV of each detection region what kind of motion pattern the front image is, and sets a layer based on the determined motion pattern to determine which one. The layer classification information LB indicating which layer the area belongs to is supplied to the information generation processing unit 27.

FIG. 10 shows the structure of the layer classification unit 26. Motion pattern determination unit 261 of layer classification unit 26
Then, the motion vector MV of each detection region supplied from the motion detection processing unit 22 is accumulated in frame units, and the motion pattern is determined based on the accumulated motion vector MV.

Here, it is discriminated whether or not the direction of the motion vector MV of each detection region is the radial direction from the image reference position CP, and all images are zoom-in operation directions that spring out from the image reference position CP. , All images are image reference position C
When it is the zoom-in motion direction that springs out from P, it is determined to be a straight-ahead motion. For example, as shown by the arrow in FIG. 11A, when the vector direction of the motion vector MV is the radial direction from the image reference position CP, it is determined to be a straight movement.

Next, when it is not determined that the vehicle is going straight,
It is determined whether or not the direction of the motion vector MV of each detection area is the direction opposite to the radial direction, and all the images are in the zoom-out operation direction in which the images are sucked into the image reference position CP. When it is the zoom-out operation direction that is sucked in, it is determined to be the backward operation. For example, when it is set to the direction of the image reference position CP as indicated by the arrow in FIG. 11B and it is detected that all the images are in the zoom-out operation direction in which the image is sucked into the image reference position CP, it is determined to be the backward movement.

When it is not determined that the vehicle is going straight or backward, it is determined whether or not only a part is in the zoom-out operation direction. When only part is in the zoom-out operation direction, there is an overtaking vehicle. Determine. For example, as shown in FIG. 11C, when it is detected that the movements of the detection areas at the left and right ends are in the zoom-in operation direction and only part of the movement is in the zoom-out operation direction, it is determined that there is an overtaking vehicle. Furthermore, when it is not determined that there is a straight-ahead movement, a backward movement, and an overtaking vehicle, the motion vector M as shown in FIG.
When the vector direction of V is horizontal, it is determined to be a right turn or left turn operation depending on the vector direction.
In addition, when the vector direction of the motion vector MV is the horizontal direction in a part of the detection area as shown in FIG. The motion pattern MP determined in this way is notified to the layer creation determination unit 262.

The layer creation determination unit 262 determines whether or not the determined motion pattern has continued for a predetermined number of frames or more based on the motion pattern MP determined by the motion pattern determination unit 261. Here, the layer pattern information LP corresponding to the motion pattern determined when the motion pattern continues for a predetermined number of frames or more is generated.

Here, when it is continuously determined that the motion pattern is such that the entire screen expands like a forward motion for a predetermined number of frames or more, for example, a layer for instructing the formation of a distant view, middle view, and near view layer. The pattern information LP is generated and notified to the classification processing unit 263. Also, when it is continuously determined that the upper part is a motion pattern in which the upper part moves in the horizontal direction such as a right turn or a left turn, a predetermined number of frames or more are continuously detected, the horizontal pattern is moved not only in the distant view, middle view, and near view layers. The layer pattern information LP instructing the generation of the layer including the image to be generated is generated and notified to the classification processing unit 263. Further, when it is continuously determined that the motion pattern includes an image that shrinks with time such as an overtaking vehicle or when reversing, the distant view, middle view, and near view layers are continuously identified. Not only that, the layer pattern information LP for instructing the creation of the receding layer including the image to be reduced is generated. Also, when it is continuously determined that a part of the movement pattern is a horizontal movement pattern, such as a transverse object, for a predetermined number of frames or more, an instruction to create a layer including an image that moves in the horizontal direction is issued. The layer pattern information LP is generated. In this way, when the determined motion pattern continues for a predetermined number of frames or more, the layer pattern information LP corresponding to the determined motion pattern MP is generated, so that a frame in which the motion pattern is erroneously determined occurs. However, the layer pattern information LP according to the correct motion pattern can be generated.

The threshold setting unit 264 uses the motion vector MV within a predetermined time range (for example, 30 frames before and after the frame of interest) and the motion amount of the motion vector MV whose vector direction is the radial direction from the image reference position CP. Of the average value Vavg, the maximum value Vmax, and the minimum value Vmin of
Based on the average value Vavg, the maximum value Vmax, and the minimum value Vmin, a threshold value Th for classifying into the layer indicated by the layer pattern information LP is set and supplied to the classification processing unit 263.

For example, when it is indicated by the layer pattern information LP that the layers are divided into the distant view, the middle view, and the near view,
A threshold Th1 indicating the division position between the distant view layer and the middle view layer is calculated based on the equation (1). Further, the threshold Th2 indicating the division position of the middle view layer and the near view layer is calculated based on the equation (2).

[0044]

[Equation 1]

In the setting of this threshold value, a histogram of the amount of motion may be obtained, and the minimum values of this histogram may be used to obtain the threshold values Th1 and Th2. In this way, the threshold Th is dynamically changed according to the distribution of the motion amount,
The image is not classified into only one layer, and the layers can be properly classified according to the distribution of the motion amount.

In the classification processing unit 263, the threshold setting unit 264
Based on the threshold value Th and the motion amount of the motion vector MV, each detection region of each frame has the layer pattern information LP.
The layer classification is performed by discriminating to which of the layers the creation of which is instructed. Further, the detection area that moves in the horizontal direction and the detection area in the zoom-out operation direction are assigned to the corresponding layers. When the classification processing unit 263 completes the layer classification of each detection area of the frame of interest, the layer classification information LA indicating the result of the layer classification.
Is supplied to the classification correction unit 265.

Here, the classification processing unit 263 uses the motion vector MV from the motion detection processing unit 22 to create the previous m frames for each detection region when creating three layers of the distant view, the middle view, and the near view. Minute and the amount of movement of the subsequent n frames is calculated. For example, when the motion amount of the motion vector MV changes over time as shown in FIG. 12A in the detection area provided at the right end portion of the front image (the numbers in the figure indicate the motion amount). , The amount of motion of the detection region AR1 in the Fp frame, the amount of motion of the detection region AR1 in the Fp-1 to Fp-m frames, and the amount of motion in the Fp + 1 to Fp + n frames with respect to the detection region AR1 in the Fp frame. An average value of the movement amount is calculated from the movement amount of the detection area AR1. Furthermore, the detection area AR1
By comparing the average value calculated as the amount of movement of Fp with the thresholds Th1 and Th2 set as described above, this Fp
It is determined whether the detection area AR1 in the frame belongs to the distant view, the middle view, or the near view. In this way, for the detection area AR1 of the target frame, the average value of the movement amount is calculated using the movement amounts of the previous frame and the subsequent frame, and the detection area AR1 is layered using this average value. Even if an error occurs in the movement amount of the area AR1, this detection area AR1
The layers can be classified, and even if the amount of movement of each frame varies due to the difference in the size of the subject or the difference in the distance to the subject, the influence of these variations can be prevented. Further, by performing similar processing on other areas and other frames, each detection area is classified into any of the distant view, middle view, and near view layers as shown in FIG. 12B, and the layer classification information LA To the classification correction unit 265. Figure 1
In FIG. 2B, the detection area indicated by the cross hatch is classified as the near view, the detection area indicated by the diagonal lines is classified as the middle view, and the other detection areas are classified as the distant view.

The classification corrector 265 refers to the layer classification information LA of each detection area, and corrects an area belonging to the same layer, where the number of connected areas is smaller than a predetermined number, to a layer matched with the surrounding area. For example, in FIG. 12B, in the area AR4-g of the Fp + 4 frame and the area AR6-h of the Fp-2 frame, since the areas belonging to the same layer are not connected, these areas are surrounded as shown in FIG. 12C. The layer classification information LB that corrects the combined area and indicates which area is set as which layer is supplied to the information generation processing unit 27.

When there is an overtaking vehicle, not only the above-mentioned distant view, middle view, and near view, but also a retreat layer including an image of the overtaking vehicle is generated as shown in FIG. 12D.
Here, since the passing vehicle has the direction of the motion vector MV that is opposite to that of the image of the distant view, the middle view, and the close view, the backward layer is generated with the direction of the time axis being opposite to that of the other layers.

FIG. 13 shows the configuration of the information generation processing unit 27. The layer classification information LB supplied from the layer classification unit 26 is supplied to the motion amount average value calculation unit 271 and the image reading unit 272.

The motion amount average value calculation unit 271 calculates the average value of the motion amount for each layer for each frame using the motion amount of each detection area. For example, in the case of the Fe frame, when there are ny detection areas as distant view layers, an average value is calculated using the movement amounts of the ny detection areas and the average value is supplied to the image reading unit 272, and this calculation is performed. The calculated motion amount MYv is stored in the intermediate image information storage area 32 of the storage unit 30 as intermediate image information.

In the image reading unit 272, based on the layer classification information LB supplied from the layer classification unit 26, the motion amount M calculated by the motion amount average value calculation unit 271 from the side end portion.
The image signal SDC is extracted for each layer according to Yv to generate an intermediate image signal GFv for each layer, and the intermediate image signal GFv is supplied to the intermediate image signal interpolation unit 273.

Here, when the image signal SDC is read from the side end portion by the signal amount corresponding to the movement amount MYv of each layer to generate an intermediate image signal, for example, an image based on the intermediate image signal is as shown in FIG. . 14A shows an image based on the intermediate image signal of the distant view layer, FIG. 14B shows an image based on the intermediate image signal of the middle view layer, and FIG. 14C shows an image based on the intermediate image signal of the near view layer.

In the intermediate image signal interpolating section 273, the motion amount M
Even if the images of the layers with different Yv are pasted in the order of the distant view, the middle view, and the near view, the intermediate image signal GFv is corrected so that an uncovered area without an image does not occur. Is stored in the intermediate image information storage area 32 as the intermediate image signal GYv. In this correction of the intermediate image signal, if all of the layers in the front are interpolated, the layers in the back will be hidden. Therefore, for the innermost layer, the entire gap is interpolated, and for the layer located in the middle, only signals that are horizontally adjacent to the gap created by the region belonging to the layer before this layer. Interpolation processing is performed to generate an image signal of a gap area where there is no image and paste it to the intermediate image signal. By performing the interpolation process in this manner, the image based on the intermediate image signal GFv shown in FIG. 14A is interpolated using signals in which the image-less portion (the shaded area) is horizontally adjacent, and the image shown in FIG. The image is as shown in. Similarly, the image based on the intermediate image signal GFv shown in FIG. 14B becomes an image corrected as shown in FIG. 15B. When the layer located closest to the foreground is, for example, the near view layer shown in FIG. 14C,
Since the uncovered area does not occur in the layer located closest to the foreground, interpolation processing is unnecessary, and the near view layer shown in FIG. 15C is the same as that in FIG. 14C.

In the intermediate image information storage area 32 of the storage unit 30, the motion amount MYv for each layer generated in the intermediate image information generation block 25 and the intermediate image signal GYv for each layer are associated with each other for each frame. It is stored as information.
Here, since the amount of motion of the intermediate image signal of the distant view layer stored in the intermediate image information storage area 32 is small, FIG.
As shown in A, the signal amount of the image signal decreases. In addition, the intermediate image signal of the near view layer has a large amount of motion and thus has a large amount of signal as shown in FIG. 15C. In the middle view layer, the signal amount is between the distant view and the middle view, as shown in FIG. 15B. It should be noted that, in the backward layer, the movement directions are opposite, so that in the intermediate image signal of the backward layer, FIG.
The frame direction is opposite to that in FIG. 15C.

When it is instructed to create a horizontal movement layer based on the layer pattern information LP, the image signal of the detection area in which the direction of the motion vector is the horizontal direction is used as the intermediate image signal of this horizontal movement layer. Set.
Here, in an image that moves in the horizontal direction, an image that moves outward from the front image and an image that enters the front image occur. For example,
When making a right turn, the image enters from the right end of the front image and the image exits from the left end. Therefore, the horizontal movement layer of the image moving outward has the same time axis direction as that of the distant view layer, and the horizontal movement layer of the image moving inward has the same time axis direction as that of the backward view layer. Direction.

In this way, the intermediate image information storage area 3
In 2, the intermediate image signals of the distant view, middle view, and near view layers and the intermediate image signals of the backward layer and the horizontal movement layer are generated and stored according to the amount of motion. Further, the amount of movement of each layer is also stored together as described above.

Next, the case where the peripheral image signal SDR of the image on the right side is generated using the intermediate image information stored in the intermediate image information storage area 32 of the storage section 30 will be described.
As described above, when the peripheral image signal SDR is generated, the motion amount MYv stored as the intermediate image information is read from the intermediate image information storage area 32, and the signal read amount of the intermediate image signal GYv for each layer is set as the motion amount MYv. Based on. Further, an intermediate image signal having a signal amount corresponding to the motion amount MYv is used for each layer, and the distant view layer, the middle view layer, and the near view layer are pasted in this order. Further, when a layer different from the distant view, middle view, and near view layers is provided, the image pasting process of this layer can also be performed to generate the peripheral image signal SDR. By supplying the generated peripheral image signal SDR to the projector 12R in synchronization with the output timing of the image signal SDC of the front image, not only the front image but also the right side image can be continuously displayed.

FIG. 16 shows the configuration of a peripheral image signal generation block 40 which reads out the intermediate image information stored in the intermediate image information storage area 32 and generates a peripheral image signal. In the image generation control unit 41 of the peripheral image signal generation block 40, the motion amount MYv for each layer stored as the intermediate image information in the intermediate image information storage area 32 is read in the frame order, and the layer is read based on the read motion amount MYv. The read amount of the intermediate image signal for each is determined. Based on the read amount determined by the image generation control unit 41, the intermediate image information GYv is read from the intermediate image information storage area 32.
Is read for each layer, and the pasting units 421-1 to 421 for each layer provided in the layer image generation processing unit 42 are read.
-Supply to 5.

Here, the intermediate image signal of the horizontal moving layer which moves in the horizontal direction when turning right or left is supplied to the pasting unit 421-1. Further, the intermediate image signal of the distant view layer is supplied to the attaching unit 421-2, and the intermediate image signal of the middle view layer and the intermediate image signal of the near view layer are attached to the attaching unit 421-3,
Supply to 421-4. Further, the intermediate image signal of the backward layer including the image of the overtaking vehicle is supplied to the pasting unit 421-5.

An image shift unit 423-1 described later is connected to the pasting unit 421-1, and the image shift unit 423 is connected.
-1 to the image signal supplied from the intermediate image information storage area 3
The image signal of the corresponding layer read from 2 is pasted. The image signal of the horizontal movement layer obtained by pasting the image by the pasting unit 421-1 is supplied to the one-frame delay unit 422-1 and the distant view pasting unit 441 of the image signal generation unit 44. .

The one-frame delay unit 422-1 delays the image signal supplied from the pasting unit 421-1 by one frame and supplies the delayed image signal to the image shift unit 423-1. The image shift unit 423-1 moves the image based on the image signal supplied from the 1-frame delay unit 422-1 in the horizontal direction based on the movement amount MYv of the horizontal movement layer supplied from the image generation control unit 41. . Further, the image signal of the image moved in the horizontal direction is supplied to the pasting unit 421-1.

The image based on the image signal delayed by one frame as described above is moved in the horizontal direction based on the movement amount MYv supplied from the image generation control unit 41, and
The image signal of the moving image can be generated by pasting the intermediate image signal GYv of the horizontal moving layer read by the amount of movement MYv from the intermediate image information storage area 32 to the image moved in the horizontal direction.

Similarly, 1-frame delay units 422-2 to 42
The image based on the image signal delayed by one frame at 2-5 is moved in the horizontal direction based on the motion amount MYv of each layer supplied from the image generation control unit 41 at the image shift units 423-2 to 423-5. At the same time, the intermediate image signal GYv read from the intermediate image information storage area 32 by the amount of movement of each layer is supplied to the pasting units 421-2 to 421-5 with respect to the horizontally moved image. By pasting the image based on the read intermediate image signal GYv,
An image signal in which images sequentially move can be generated for each layer.

Since the intermediate image signal is a signal read from the front image signal, the image based on this intermediate image signal is an image on the same surface as the screen 10C. However, the screen 10R for displaying the peripheral image is provided so as to be inclined with respect to the screen 10C on the front side. Therefore, the image of each layer is pasted using the image signal of the layer in which the direction of the motion vector is the direction of the image reference position CP or the opposite direction, such as the distant view, middle view, near view layer, and backward layer. If the peripheral image signal SDR is generated and an image is displayed, the screen 10R is displayed.
The image displayed at is not in the correct shape due to the inclination of the screen 10R with respect to the screen 10C. Therefore, the image signal of the layer in which the direction of the motion vector is the direction of the image reference position CP or the opposite direction is the projective transformation unit 4
3 is supplied to the respective conversion processing units 431 to 434, and when the image of each layer is displayed on the screens 10L and 10R, the projective conversion is performed so that the image correctly moves in the vector direction of the motion vector MV. Further, since the direction of the motion vector MV is the horizontal direction, the image of the horizontal movement layer is not subjected to projective transformation like the distant view, middle view, near view layer, and backward layer.

Conversion processing units 431 to 43 of the projective conversion unit 43
In 4, the projective transformation is performed on the image signals supplied from the pasting units 421-2 to 421-5 so that the image has a shape corresponding to the orientation of the screen 10R. Here, when the image signals are sequentially read from the front image, the display surface 10CR and the screen 10R of the read image are shown in FIG. 17A and FIG.
Since the images are not on the same plane as shown in 7B, when the image read from the front image is displayed on the screen 10R, the image on the display surface 10CR is projected as it is on the surface of the screen 10R, and the shape is not correct.

Therefore, the projective transformation is performed so that the vertical direction is expanded from the center Czm of zoom-in or zoom-out of the image at an expansion rate proportional to the distance, and the speed is horizontally changed from the center Czm of zoom-in or zoom-out to the center. The magnification is such that it is proportional to the distance.

Here, as shown in FIG. 17C, the length from the front edge of the screen of the right side image to the center Czm is L, the length in the peripheral direction of the right side image is A, and the front side image and the right side image are formed. Assuming that the proportional constant based on the angle “θs” is α, the position (xa, ya) of the image directly read from the front screen edge and the position (Xa, after conversion processing so that the image can be correctly displayed on the right side image. Ya) has the relationship of Expression (3) in the vertical direction and the relationship of Expression (4) in the horizontal direction. Therefore, as shown in FIG. 17D, as the image signal of (Xa, Ya) on the screen 10R, by using the signal of the position (xa, ya) that satisfies the equations (3) and (4), the image signal of FIG. The right side image can be correctly displayed on the screen 10R as shown in FIG.

[0069]

[Equation 2]

The image signal of the distant view layer after the projective transformation obtained by the transform processing unit 431 is supplied to the distant view pasting unit 441 of the image signal generating unit 44. Also, the conversion processing unit 4
The image signal of the middle-view layer after projective conversion obtained in 32 is supplied to the middle-view pasting unit 442 and the conversion processing unit 4
The image signal of the near view layer after the projective transformation obtained in 33 is supplied to the near view pasting unit 443. Furthermore, the image signal of the receding layer obtained by the conversion processing unit 434 is supplied to the reduction and pasting unit 444.

In the distant view pasting unit 441, the pasting unit 4
21 and the image signal supplied from the conversion processing unit 431, an image signal in which the image of the distant view layer is pasted on the image of the horizontal movement layer is generated and supplied to the middle-view pasting unit 442.

In the middle ground pasting unit 442, the conversion processing unit 4
32 and the image signal supplied from the distant view pasting unit 441, an image signal in which the image in the middle view layer is pasted to the image in which the image in the distant view layer is pasted is generated and supplied to the near view pasting unit 443. .

In the close view pasting unit 443, the conversion processing unit 4
33 and the image signal supplied from the middle ground pasting unit 442, an image signal in which the image in the middle ground layer is pasted to the image in which the image in the middle ground layer is pasted is generated and supplied to the reduction pasting unit 444. To do.

In the reduction pasting unit 444, the conversion processing unit 4
34 and the image signal supplied from the near-field pasting unit 443, an image signal in which the image in the near-field layer is pasted with the image in the receding layer whose image shrinks with time is generated. The image signal generated by the reduction and pasting unit 444 becomes the image signal of the side image to which each layer from the horizontal movement layer to the reduction layer is pasted.

Therefore, by supplying the image signal generated by the reduction and pasting unit 444 to the projector 12R as the peripheral image signal SDR, it is possible to display the image on the right side, which is continuous with the front image, on the screen 10R.

When the intermediate image signal stored in the intermediate image information accumulating area 32 is not subjected to the process of interpolating a portion without an image, or the reduction paste section 4
When an image-free portion is generated in the image signal generated at 44, a pre-interpolation processing unit is provided to determine what layer image is combined in an area adjacent to the image-free portion, and Interpolation processing is performed using an image of a deep layer. For example, when there is no image in a part that is horizontally adjacent to the region where the images of the middle-view layer and the foreground layer are pasted together, interpolation processing can be performed using the image of the deep-view middle-layer. A good peripheral image signal SDR without image loss can be generated.

Further, by generating the peripheral image signal SDL in the same manner as the peripheral image signal SDR and supplying it to the projector 12L, it is possible to display an image on the left side surface continuous with the front image on the screen 10L.

Next, in the additional information signal generation block 50, an additional information signal is generated based on the motion vector MV of each detection region supplied from the motion detection block 20 and the layer classification information LB supplied from the intermediate image information generation block 25. A certain swing signal SE is generated.

In the generation of the rocking signal SE, in the detection area provided on the left side of the front image, the motion amount MSL is generated from the motion vector MV of the detection area indicated by the layer classification information LB to be the middle ground layer. . For example, an average value or a mode value is calculated from the motion amount of each motion vector MV of the selected middle-view layer to obtain the motion amount MSL. Similarly, for the right side of the front image, the movement amount MSR
To generate.

The movement amounts MSL and MSR are defined as the movement amount MSL and the movement amount MSR, where the direction going out from the front image is the positive (+) direction and the direction going inward from the front image is the negative (-) direction. The added value MX indicates information about the speed in the front-rear direction when the front image is captured. Further, the difference value MY between the motion amount MSL and the motion amount MSR indicates information regarding turning when the front image is captured. Note that FIG. 18 shows an example of the motion pattern, the added value MX, and the difference value MY, and FIG.
B is the case of a left turn operation. MSL for straight movement
And the movement amount MSR is “+10”, the added value M
X is “20” and the difference value MY is “0”. Also, the amount of movement MSL in the left turn motion is “−8”, and the amount of movement MSR is “+”.
When the value is 10 ", the added value MX is" 2 "and the difference value MY is" 18 ".

Here, when the chair is swung corresponding to the display image in order to enhance the sense of presence, the differential of the added value MX indicating the information on the speed in the front-rear direction indicates the acceleration. Acceleration information MX by calculating the differential value
d is generated, and the swing signal SEp for pitch angle control is generated based on this acceleration information MXd. The swing signal SEp for controlling the pitch angle is used as the motion base platform 1
Supply to 7. Further, since the difference value MY is information regarding turning when the front image is captured, the difference value M
Y is multiplied by the addition value MX to generate centrifugal force information MXy, and the roll angle control swing signal SEr is generated based on the centrifugal force information MXy. This roll angle control swing signal S
The Er is supplied to the motion base platform 17.

In the motion base platform 17, the chair 173 is based on the swing signal SEp for pitch angle control.
By performing an operation (pitching operation) of tilting the vehicle in the front-back direction, a load corresponding to the acceleration information MXd is given to the person sitting on the chair 173. Further, by performing an operation (rolling operation) of tilting the chair 173 in the left-right direction based on the roll angle control swing signal SEr, a load sitting on the chair 173 is applied according to the centrifugal force information MXy.

As described above, the swing signal SEp for pitch angle control and the swing signal SEr for roll angle control are added information signals.
By generating the pitch angle control swing signal SEp and the roll angle control swing signal SEr to the motion base platform 17 to generate the acceleration information MXd.
By performing the pitching operation of the chair 173 in accordance with the movement and the rolling operation of the chair 173 in accordance with the centrifugal force information MXy, it is possible to give the person sitting on the chair 173 an acceleration feeling and a centrifugal force according to the display image. .

By the way, when the peripheral image is presented and rocked at the same time as described above, the person sitting on the chair 173 feels acceleration and the like from both the peripheral image and rocking. Here, assuming that the surrounding image has many near views and the surrounding image has many distant views, the movement in the surrounding image becomes very large when the surrounding image has many nearby views. The acceleration feeling and the like obtained from is increased. In addition, when there are many distant views in the peripheral image, the motion in the peripheral image becomes small, so that the sense of acceleration and the like obtained from the peripheral image becomes small. For this reason, when performing the swing based on the image of the middle view layer, if the surrounding image includes many near views, a sufficient sense of acceleration or the like can be obtained from the surrounding image, so the swing is reduced. Further, when the peripheral image includes many distant views, a sufficient sense of acceleration or the like cannot be obtained from the peripheral image, so that the swing is increased.

The area of each of the distant view, middle view, and near view regions in the peripheral image is determined by the signals of the distant view layer, the middle view layer, and the near view layer which are pasted by the image signal generating unit 44 of the peripheral image signal generating block 40. It can be calculated using the signal. Here, the image signal of the distant view layer supplied from the conversion processing unit 431 to the distant view pasting unit 441, the image signal of the middle view layer supplied from the conversion processing unit 432 to the middle view pasting unit 442, and the conversion processing unit The image signal of the foreground layer supplied from 433 to the foreground attachment unit 443 is supplied to the additional information signal generation block 50, and, for example, the image signal of the distant layer is pasted on the image of the distant layer. And the number of pixels of the image of the distant view layer excluding the image of the near view layer is calculated as the area of the distant view region. In addition, the number of pixels of the image of the middle view layer, which is obtained by removing the image of the near view layer pasted on the image of the middle view layer, from the image signal of the middle view layer is set as the area of the middle view region, and Based on this, the number of pixels of the image of the near view layer is set as the area of the near view region.

By comparing the calculated numbers of pixels in this way, the area ratio of each of the distant view area, the middle view area, and the near view area can be determined, and the magnitude of the swing can be determined based on this area ratio. As shown in FIG. 19A, when the area of the distant view region is large, the signal levels of the pitch angle control rocking signal SEp and the roll angle control rocking signal SEr are increased to increase the rocking. . Further, when the area of the near view area is large as shown in FIG. 19B, the swing is reduced by setting the signal levels of the pitch angle control swing signal SEp and the roll angle control swing signal SEr to be small. in this way,
By correcting the shake depending on whether the peripheral image includes many distant images or many near-view images, it is possible to perform the shake so as to supplement the sense of acceleration obtained from the peripheral images. Therefore, a better sense of presence can be obtained.

Further, in the above-described embodiment, the intermediate image for each layer is generated using the accumulated image signal of the front image, and the image signal of the peripheral image is generated using the signal of the intermediate image. Although the swing signal is generated based on the motion vector when the intermediate image is generated, the image signal of the front image is not limited to the accumulated signal. For example, as in an image taken from a vehicle traveling straight ahead, a peripheral image is obtained by using a real-time image signal of the front image that satisfies the constraint condition that the movement of the image is the radial direction centered on the image reference position CP of the front image. At the same time, a swing signal according to the peripheral image can be generated.

In this case, the detection area is set as shown in FIG. 5, and the zoom-out operation centering on the image reference position CP, that is, the enlargement ratio Z is set to set the detection area to (1 / Z). By reducing, the sum of the errors between the reduced area and the image signal at the corresponding position in the image one frame before is calculated, and the enlargement ratio Z that minimizes the sum of the errors is detected as shown in FIG. Thus, the amount of movement for each detection area is calculated.

Next, each detection area is divided into, for example, a distant view, a middle view, and a near view based on the amount of movement, and the amount of movement of the image is set from the amount of movement for each layer.
The image of the detection area divided into layers is extracted according to the movement amount of the corresponding layer, and the image signals of the extracted images are sequentially attached from the distant view layer with a small movement amount to the near view layer with a large movement amount. By attaching it, a peripheral image signal that does not exist in the angle of view of the frame of interest can be generated. In addition, since the motion amount is obtained for each detection area as in the case described above, the swing signal SE is generated based on this motion amount, so that a wide-angle image presentation is performed based on the real-time image signal SDC. In addition to being able to perform, it is possible to increase the sense of presence by swinging in accordance with the movement of the image.

Further, the method of enhancing the feeling of presence is not limited to the method of giving a feeling of acceleration or the like by tilting the chair forward, backward, leftward or rightward based on the swing signal SE. For example, a blower control signal is generated as an additional information signal and supplied to the blower,
While applying a breeze to the person sitting on the chair 173,
It is possible to further enhance the sense of presence by controlling the air volume according to the obtained information regarding the forward speed and varying the wind pressure, or by controlling the wind direction according to the information regarding the turning and varying the wind direction. In addition, a voice control signal is generated as an additional information signal and supplied to a speaker, and a running sound, wind noise, engine sound, etc. are output from the speaker, and the volume and timbre are varied according to the information on the forward speed. But it can enhance the sense of presence. In this way, by detecting the feature amount related to the motion of the image from the image signal of the front image and generating the additional information signal related to the motion of the image based on the feature amount,
You can increase the sense of presence.

Further, the processing performed in each block described above may be realized not only by hardware but also by software. The configuration in this case is shown in FIG.

The computer 90 has a built-in CPU (Central Processing Unit) 901 as shown in FIG. 20, and the CPU 901 is provided with a ROM 9 via a bus 920.
02, RAM 903, hard disk drive 90
4, the input / output interface 905 is connected. Furthermore, an input unit 911, a recording medium drive 912, a communication unit 913, and an image input / output unit 914 are connected to the input / output interface 905.

When a command is input from the input unit 911 that is configured by using an operation input unit such as a keyboard or a mouse or a voice input unit such as a microphone, the command is supplied to the CPU 901 through the input / output interface 905. To be done.

The CPU 901 has a ROM 902 and a RAM.
The program stored in the hard disk drive 903 or the hard disk drive 904 is executed to perform the processing according to the supplied instruction. Furthermore, ROM902 and RAM90
3 or the hard disk drive 904 also stores a program for executing the processing in each block described above, and also generates a side image.

Here, the signal processing program for generating the intermediate image information and for generating and outputting the peripheral image signal from the intermediate image information and the additional information signal is the ROM 902 which is an information recording medium or the hard disk drive 904.
Can be stored in advance.

Alternatively, a removable information recording / transmission medium composed of a removable recording medium using magnetism or light or a semiconductor element, for example, an optical disk such as a floppy (R) disk or a CD-ROM, or an optical disk such as an MO disk. For recording a signal processing program on a magnetic disk, tape cartridge, semiconductor memory, or the like,
The removable information recording / transmission medium is mounted on the recording medium drive 912 to read the recorded signal processing program, and the read program is read via the input / output interface 905 and the bus 920 to the hard disk drive.
It may be installed by storing it in the drive 904 or the like.

Further, a communication unit 913 is provided to receive a signal processing program supplied using a wired or wireless transmission path, for example, a network such as LAN or the Internet, or satellite broadcast wave or terrestrial broadcast wave. , The received signal processing program, the input / output interface 905
And hard disk drive 9 via bus 920
It can also be installed on 04 etc.

Here, when the signal processing program is executed and the intermediate image information is generated, the hard disk drive 904 or the recording medium drive 912.
The image signal SDC of the front image recorded on the recording medium mounted on the recording medium is read out, the above-described processing is performed, and the intermediate image information and the additional information signal are, for example, the image signal S.
It is recorded on a recording medium mounted on the hard disk drive 904 or recording medium drive 912 together with DC.
When the output of the image signal of the peripheral image is requested, the intermediate image information is read out to generate the peripheral image signals SDR and SDL indicating the left and right side images, and the peripheral image signals SDR and SDL and the additional information. The signal SE is output at the same timing as the image signal SDC of the front image. Therefore, continuous images with a wide angle of view can be displayed using the front and left and right screens, and rocking or the like can be performed corresponding to the images to enhance the sense of presence.

When a real-time image signal is supplied, the image signal extracted from the image signal SDC is used to generate a peripheral image signal SD having an angle of view that does not exist in the frame of interest.
R and SDL are generated and output, and an additional information signal is generated and output from the amount of movement of the front image. Therefore, even based on the image signal of the front image in real time,
Along with being able to display continuous images with a wide angle of view,
It is possible to enhance the sense of reality by performing rocking or the like in accordance with the image.

As described above, according to the above-described embodiment,
Not only can the movement of the front image be discriminated, but a peripheral image having a field of view different from that of the front image can be generated, and since an additional information signal related to the movement of the image is generated, not only can the image be displayed at a wide angle of view, It is possible to enhance the sense of presence by performing rocking or the like corresponding to the image. Further, since the additional information signal can be generated from the image signal, it is possible to enhance the sense of presence without using a sensor or the like. Furthermore, even if an image signal already captured is used, it is possible to display an image with a wide angle of view and enhance the sense of presence.

[0101]

According to the present invention, the feature amount relating to the movement of the image at the predetermined position of the target frame in the input image signal is detected by using the input image signals of a plurality of frames, and based on the feature amount, An additional information signal related to the movement of the image at a predetermined position is generated, and a peripheral image signal having an angle of view that does not exist in the frame of interest is generated based on the feature amount and the input image signal. Therefore, based on the input image signal, an image display having a wider angle of view than the image based on the input image signal and, for example, rocking or the like according to the displayed image are performed, and an image display having a high sense of presence is performed. be able to.

Further, since the detected characteristic amount is divided into layers and the additional information signal is generated based on the characteristic amount of a predetermined layer, it is possible to perform appropriate swing or the like according to the displayed image. Furthermore, since the additional information signal is corrected according to the image signal of the generated peripheral image, the sense of presence can be further enhanced.

[Brief description of drawings]

FIG. 1 is a diagram showing a display system.

FIG. 2 is a diagram showing an image projection state.

FIG. 3 is a diagram showing a motion-based platform.

FIG. 4 is a diagram showing a configuration of a signal processing device.

FIG. 5 is a diagram showing a motion vector of each detection area.

FIG. 6 is a diagram showing a method of discriminating a motion vector.

FIG. 7 is a diagram showing a change in area size during zooming.

FIG. 8 is a diagram showing a method of determining an image reference position.

FIG. 9 is a diagram illustrating a configuration of a motion detection processing unit.

FIG. 10 is a diagram showing a configuration of a layer classification unit.

FIG. 11 is a diagram showing a relationship between a vector direction and a motion pattern.

FIG. 12 is a diagram showing an operation of generating layer classification information.

FIG. 13 is a diagram illustrating a configuration of an information generation processing unit.

FIG. 14 is a diagram showing an intermediate image for each layer.

FIG. 15 is a diagram showing an intermediate image for each layer after interpolation processing.

FIG. 16 is a diagram showing a configuration of a peripheral image signal generation block.

FIG. 17 is a diagram showing an image conversion process.

FIG. 18 is a diagram showing an operation of generating an additional information signal.

FIG. 19 is a diagram for explaining correction processing of a swing signal.

FIG. 20 is a diagram showing a configuration using a computer.

[Explanation of symbols]

10L, 10C, 10R ... Screen, 12L, 12
C, 12R ... Projector, 15 ... Image processing device, 17 ... Motion base platform, 2
0 ... Motion detection block, 21 ... Frame delay processing unit, 22 ... Motion detection processing unit, 25 ... Intermediate image information generation block, 26 ... Layer classification unit, 27 ...
Information generation processing unit, 30 ... Storage unit, 31 ... Front image signal storage region, 32 ... Intermediate image information storage region,
40 ... Peripheral image signal generation block, 41 ... Image generation control unit, 42 ... Layer image generation processing unit, 43 ...
..Projection conversion unit, 44 ... Image signal generation unit, 50 ...
-Additional information signal generation block, 90 ... Computer, 221 ... Size conversion processing unit, 222 ... Error sum calculation unit, 223 ... Comparison processing unit, 224 ... Data holding unit, 225 ... Search control unit 261, pattern determination unit 262 layer creation determination unit 263
..Classification processing unit, 264 ... Threshold setting unit, 265 ...
-Classification correction unit, 271 ... Motion amount average value calculation unit, 27
2 ... Image reading unit, 273 ... Intermediate image signal interpolation unit, 421-1 to 421-5 ... Pasting unit, 422-1
~ 422-5 ... 1 frame delay unit, 423-1 to 423
-5 ... image shift unit, 431-434 ... conversion processing unit, 441 ... distant view pasting unit, 442 ... middle view pasting unit, 443 ... near view pasting unit, 444 ...・
Reduced pasting unit, 901 ... CPU, 902 ... R
OM, 903 ... RAM, 904 ... Hard disk drive, 905 ... Input / output interface,
911 ... Input unit, 912 ... Recording medium drive,
913 ... Communication unit, 914 ... Image input / output unit, 92
0 ... bus

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Satoshi Kokubo             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation (72) Inventor Daisuke Kikuchi             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation (72) Inventor Hideo Kasama             6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni             -Inside the corporation F-term (reference) 5B050 AA10 BA06 CA06 CA07 DA01                       EA04 EA07 EA12 EA13 EA19                       EA27 FA02                 5B057 CD02 CD05 CD11 CE08 DA16                       DC32                 5C054 EA05 FC13 FD02 FE11 HA15                 5L096 BA08 CA04 DA04 FA66 FA67                       HA04 LA17

Claims (11)

[Claims] 1. Using input image signals of a plurality of frames,
Detecting a feature amount relating to the movement of the image at a predetermined position of the frame of interest in the input image signal, and based on the detected feature amount, generating an additional information signal related to the movement of the image at the predetermined position, A signal processing method, wherein a peripheral image signal having an angle of view that does not exist in the frame of interest is generated based on the detected feature amount and the input image signals of the plurality of frames. 2. The signal processing method according to claim 1, wherein the detected characteristic amount is divided into layers, and the additional information signal is generated based on the characteristic amount of a predetermined layer. 3. The signal processing method according to claim 1, wherein the additional information signal is corrected according to the generated peripheral image signal. 4. The signal processing method according to claim 1, wherein the additional information signal is a rocking signal for generating rocking in accordance with a motion of an image based on the input image signal. 5. Intermediate image information is generated by projecting the input images of the plurality of frames onto a plane based on the detected feature amount and the input image signals of the plurality of frames,
The signal processing method according to claim 1, wherein a peripheral image signal having an angle of view that does not exist in the frame of interest is generated based on the intermediate image information. 6. Using multiple frames of input image signals,
A detection unit that detects a feature amount related to the movement of the image at a predetermined position of the frame of interest in the input image signal, and an addition related to the movement of the image at the predetermined position based on the feature amount detected by the detection unit. An additional information signal generation unit that generates an information signal, and a peripheral image that generates a peripheral image signal of an angle of view that does not exist in the frame of interest based on the feature amount detected by the detection unit and the input image signals of the plurality of frames. A signal processing device comprising: a signal generating unit. 7. A layer classification means for classifying the feature quantity detected by the detection means into layers, wherein the additional information signal generation means is based on the feature quantity of a predetermined layer classified by the layer classification means. 7. The signal processing device according to claim 6, wherein the additional information signal is generated. 8. The signal processing according to claim 6, wherein the additional information signal generating means corrects the generated additional information signal according to the peripheral image signal generated by the peripheral image signal generating means. apparatus. 9. The additional information signal generating means generates, as the additional information signal, a swing signal for controlling swing according to the movement of an image based on the input image signal. The signal processing device described. 10. An intermediate image information generating means for generating intermediate image information by projecting the input images of the plurality of frames onto a plane based on the detected feature amount and the input image signals of the plurality of frames, 7. The signal processing device according to claim 6, wherein the peripheral image signal generation means generates a peripheral image signal having an angle of view that does not exist in the frame of interest based on the intermediate image information. 11. A procedure for detecting a feature amount relating to a motion of an image at a predetermined position of a frame of interest in the input image signal by using a plurality of input image signals in a computer, and based on the detected feature amount. A procedure for generating an additional information signal related to the movement of an image at the predetermined position, and a peripheral image signal having an angle of view that does not exist in the frame of interest based on the detected feature amount and the input image signals of the plurality of frames. And a signal processing program for executing the procedure for generating.