JP2003052055A - Picture processor, picture signal generating method, picture processing program and information recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily display a wide-angle picture with a high feeling of being at an actual spot. SOLUTION: A motion detection block 20 detects the motion in an interested frame at a specified position, using a plurality of frames of information picture signals SDC. An intermediate picture information generating block 25 generates intermediate picture information, based on the motion detected by the detecting block 20 and the plurality of frames of input picture signals. A surrounding picture signal generating block 40 generates surrounding picture signals SDL, SDR e.g. at the left and right sides not existing in the interested frame, based on the intermediate picture information, and outputs them in the same timing as that of the input picture signals SDC. Thus, the left and right side pictures continuing the picture based on the input picture signals SDC can be displayed with the surrounding picture signals SDL, SDR to display a wide-angle picture with a high feeling of being at an actual spot.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing device, an image signal generating method, an image processing program and an information recording medium. More specifically, it is assumed that the motion of the target frame at a predetermined position is detected using the input image signals of a plurality of frames, the intermediate image information is generated based on the detected motion and the input image signals of the plurality of frames, and the generated intermediate image information is generated. Based on the intermediate image information, a peripheral image signal having an angle of view that does not exist in the frame of interest is generated.

[0002]

2. Description of the Related Art In recent years, a multi-screen display system, a curved display, a wide-angle display, a head-mounted display and the like have been put into practical use in order to display an image with a high sense of presence.

Here, in order to display an image with a high sense of presence using the above-mentioned display, for example, a three-dimensional virtual space is constructed by computer graphics. In addition, a large number of video cameras are used or a wide-angle lens is attached to the video cameras to photograph a wide space, and the photographed images are converted into a flat surface or a curved surface such as a multi-faced display or a head mounted display for display. There is.

[0004]

By the way, in order to construct a three-dimensional virtual space by computer graphics when displaying an image with a high sense of realism using such a multi-faced display or a wide-angle display, etc. Requires processing at high speed using a high-speed computer device, which requires a large amount of cost and time, and results in an image with less realism and reality than a real image.

Further, in the case of using a real image, in order to present a wide space, it is necessary to photograph the entire presented range without any gaps, such as a large number of video cameras and video cameras equipped with special lenses. A large-scale image capturing device is required, and a large amount of cost is required.

Furthermore, when an image source shot by a general video camera already exists, a wide-angle image cannot be presented unless the same scene is shot again by a plurality of video cameras.

Therefore, according to the present invention, an image processing apparatus, an image signal generating method, an image processing program, and information which can easily present an image with a wide angle of view without using a plurality of video cameras or a video camera using a special lens are provided. A recording medium is provided.

[0008]

SUMMARY OF THE INVENTION An image processing apparatus according to the present invention uses a plurality of frames of input image signals to detect a motion at a predetermined position of a frame of interest in the input image signals, and the motion detecting unit. The intermediate image information generating means for generating intermediate image information based on the motion detected by the detecting means and the input image signals of the plurality of frames, and the attention based on the intermediate image information generated by the intermediate image information generating means. And a peripheral image signal generating means for generating a peripheral image signal having an angle of view that does not exist in the frame.

Further, the image processing method uses the input image signals of a plurality of frames to detect the movement of the target frame at a predetermined position in the input image signal, and detects the detected movement and the input image signals of the plurality of frames. Intermediate image information is generated based on the generated intermediate image information, and a peripheral image signal having an angle of view that does not exist in the frame of interest is generated based on the generated intermediate image information.

Further, the image processing program causes a computer for processing the image signal to use a plurality of frames of the input image signal to detect a motion at a predetermined position of a frame of interest in the input image signal, and the motion detecting procedure. The intermediate image information generation procedure for generating intermediate image information based on the motion detected in the detection procedure and the input image signals of the plurality of frames, and the intermediate image information generated in the intermediate image information generation procedure based on the attention A peripheral image signal generation procedure for generating a peripheral image signal having an angle of view that does not exist in the frame is executed.

Further, the information recording medium is detected by the motion detection procedure, in which the computer uses the input image signals of a plurality of frames to detect the movement of the target frame in the input image signal at a predetermined position, and the motion detection procedure. Based on the intermediate motion information and the intermediate image information generated by the intermediate image information generation procedure and the intermediate image information generation procedure for generating the intermediate image information based on the motion and the input image signals of the plurality of frames. A program for executing a peripheral image signal generation procedure for generating a peripheral image signal of a corner is recorded.

In the present invention, an image at a predetermined position of the frame of interest is used by using the accumulated input image signal,
By detecting the correlation between the frame of interest and the image of the previous frame or the subsequent frame, the moving direction and the amount of motion of the image at the predetermined position are detected from the position of the image having the highest correlation with the predetermined position of the frame of interest. A motion pattern is detected based on the detected motion direction, and a plurality of layers indicating the depth are set according to the detection result. In addition, a threshold used when classifying a predetermined position into layers is set based on the detected motion amount. Layer classification is performed using this threshold value, and a motion amount for each layer and an image signal for each layer are generated as intermediate image information based on the layer classification result at a predetermined position, the input image signal, and the motion at the predetermined position.

Further, an image signal of a projected image is generated by projecting an image based on the input image signal onto a predetermined surface, and the image signal of the predetermined region of the projected image is extracted for each frame using the image signal of the projected image. An image signal of the motion determination image is generated. Further, an average value image signal for a predetermined frame is sequentially generated by using an image signal in which the position of the projected image is shifted for each frame, and the image signal is extracted for each frame from the average value image signal to obtain an image of the integrated image. A signal is generated. A plurality of image signals of this integral image are generated by changing the shift amount of the projected image. Further, the correlation between the image signal of the motion determination image and the image signals of the plurality of integrated images is detected, and the predetermined amount of the motion determination image is determined based on the shift amount when the image signal of the integrated image having the highest correlation is generated. The amount of position movement is determined. Layer classification at a predetermined position is performed based on the motion amount, and the motion amount and the image signal for each layer are used as intermediate image information.

When generating an image signal for each layer, when an image disappears region when the images based on the image signal for each layer are combined in a predetermined order, an image of a peripheral region of the image disappearing region is generated. Interpolation processing is performed using the signal.

When outputting the image signal of the peripheral image,
The intermediate image information is read out, the image signal is read out for each layer based on the amount of movement of each layer, and the image signals are pasted in a predetermined layer order. The image signal used for pasting is moved for each layer according to the amount of movement, and a new image signal is pasted for each layer to the moved signal.
The image signal pasted in the predetermined layer order is output as an image signal indicating a peripheral image having an angle of view that does not exist in the frame of interest in association with the input image signal.

If the image signal obtained by pasting the image signals for each layer in a predetermined layer order has an image-free area, interpolation processing is performed using the image signals in the peripheral area of the image-free area. , An image signal of a region without an image signal is created. Further, the projective conversion is performed on the image signal pasted for each layer according to the positional relationship between the image display surface of the image based on the input image signal and the image display surface of the peripheral image, or the pasting is performed for each layer. Projection conversion is performed on the image signal obtained by further attaching the attached image signal in a predetermined layer order.

[0017]

DETAILED DESCRIPTION OF THE INVENTION An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a display system using an image processing device according to the present invention. In this display system, for example, three screens are arranged on the front surface and both side surfaces of the user, and the projector 1 corresponding to each screen 10L, 10C, 10R is provided.
Images are projected from 2L, 12C, and 12R. The projectors 12L, 12C and 12R are connected to the image processing device 15.

The image processing device 15 stores a captured image signal, for example, an image signal SDC of a front image captured by an in-vehicle camera or the like. The image processing device 15 supplies the accumulated image signal SDC to the projector 12C to display a front image based on the image signal SDC on the screen 10C located in the front as shown in FIG. Further, in the image processing device 15, from the accumulated image signal SDC of the front image, an image signal indicating a peripheral image having an angle of view not included in the field of view of the vehicle-mounted camera, for example, left and right side images continuous with the front image. The intermediate image information for generating the peripheral image signals SDL and SDR, which are image signals indicating, are generated and accumulated. When displaying the front image on the screen 10C, the peripheral image signals SDL and SDR of the peripheral image continuous with the front image are generated by using the accumulated intermediate image information, and the peripheral image signal SDL is supplied to the projector 12L. At the same time, the peripheral image signal SDR is supplied to the projector 12R. Therefore, the left side image having continuity with the front image is displayed on the screen 10L located on the left side, and
A right side image having continuity with the front image is displayed on the screen 10R located on the right side, and an image with a wide angle of view can be presented.

FIG. 3 schematically shows the structure of the image processing apparatus 15. The image signal SDC accumulated in the front image signal accumulating area 31 of the accumulating section 30 is the motion detection block 2
1-frame delay processing unit 21 of 0, motion detection processing unit 22, and information generation processing unit 2 of intermediate image information generation block 25
7 is supplied.

The 1-frame delay processing section 21 delays the image signal SDC by 1 frame and supplies it to the motion detection processing section 22 as an image signal SDCa. The motion detection processing unit 22 sets a plurality of detection areas on the side edge side of the front image, which are divided as shown by the broken lines in FIG. 4, and the image signal SDC of the frame of interest in this detection area and one frame before are detected. Image signal S
Comparison with DCa is performed for each detection area, and the motion vector MV indicating the movement of the image is determined for each detection area and supplied to the layer classification unit 26 of the intermediate image information generation block 25.

The layer classification unit 26 determines the motion pattern of the front image from the motion vector MV of each detection area,
When generating the intermediate image information, for example, a distant view image layer of a distant subject, a near view image layer of a close subject, a middle view image layer located between the distant view image and the near view image, and The layer setting is performed by discriminating how to provide a layer different from these layers. For example, when a front image is taken by a vehicle-mounted camera and the vehicle goes straight in one direction, the front image is an image that is sequentially zoomed in. Further, when the vehicle is moving backward, the front image is sequentially zoomed out. Furthermore, when an overtaking vehicle is photographed, the overtaking vehicle is displayed as a zoomed-out image in the zoomed-in image. Further, when a right turn or a left turn is made, for example, the upper side of the front image is horizontally moved and the lower side becomes a zoomed-in image. Therefore, it is assumed that the motion pattern of the front image is determined from the motion vector MV of each detection area, and layer setting is performed based on the determined motion pattern. For example, when it is determined that the motion pattern goes straight in one direction, layers of each image of the distant view, the middle view, and the near view are generated, and when it is determined that the motion pattern is a right turn or a left turn, the distant view, the middle view, and the Not only the layer of each image in the foreground but also the layer containing the image to be moved in the horizontal direction is created. Also, when there is a motion pattern of an overtaking vehicle,
Layer settings are made so that not only the distant view, middle view, and near view images, but also the layers that include zoomed-out images are created.

Further, the layer classification unit 26 classifies, based on the motion vector MV supplied from the motion detection processing unit 22, to which layer each detection region set on the side edge side of the front image belongs. . In this layer classification,
Layer classification is performed using layers set according to motion patterns. For example, when a layer including an image that is moved in the horizontal direction and a layer that includes a zoomed-out image are generated, the layers are divided into three layers, that is, a distant view, a middle view, and a near view. I do. By this layer classification, layer classification information LB indicating which detection region belongs to which layer is generated, and the generated layer classification information LB is supplied to the information generation processing unit 27.

In the information generation processing unit 27, the layer classification unit 2
Based on the layer classification information LB from 6, the detection regions set in the front image are divided into layers, and the image signals of the detection regions are used in the frame order for each layer to generate an intermediate image signal for each layer. Further, the average value of the motion amount is calculated for each layer from the motion amount of the motion vector MV of the detection area divided into layers. The intermediate image signal GYv generated by the information generation processing unit 27 and the calculated motion amount (average value) MYv for each layer are stored in the storage unit 3 as intermediate image information.
It is stored in the intermediate image information storage area 32 of 0.

When displaying an image on the screens 10C, 10R and 10L, the stored image signal SDC is read out and the image based on the image signal SDC is displayed on the screen 10C. The intermediate image information stored in the intermediate image information storage area 32 is read by the read control signal RC from the peripheral image signal generation block 40 connected to the storage unit 30, and the motion amount MYv for each layer is read.
By using the intermediate image signal GYv of each layer in accordance with the above, the images of each layer are pasted in the order of distant view, middle view, and near view. Further, when a layer that does not belong to the three layers of the distant view, the middle view, and the near view is provided, image pasting processing of these layers is also performed, and the peripheral image signals SDL, S
Generate DR. At this timing, the peripheral image signal SDL corresponds to the image signal SDC of the front image.
By supplying 2 L to the front side image, the left side image can be continuously displayed on the screen 10 L, and by supplying the peripheral image signal SDR to the projector 12R at a timing corresponding to the image signal SDC of the front image, The right side image can be displayed on the screen 10R continuously with the image.

Next, each part of the motion detection block 20 will be described in detail. For the sake of simplicity, only the right side surface will be described below, and the left side surface will not be described.

The motion detection block 20 determines the motion vector MV for each detection area as described above. here,
When the front image has a center of movement of the image, that is, when the front image is captured by a vehicle-mounted camera, the image at time T shown in FIG. 5A is displayed at time T ′ after one frame time has elapsed, for example. 5B, which is substantially the same as the zoom-in operation image centered on the position CP (hereinafter referred to as “image reference position”) as if the image appeared to come out.

Here, as shown in FIG. 5C, the enlargement ratio Z is set to reduce the detection area of the target frame to (1 / Z), and one frame is moved while moving the position of the reduced detection area. The error sum with the previous image signal is calculated. Further, the magnification rate Z is changed, and the error sum is calculated while moving the position similarly. In this way, by detecting the position where the error sum has the minimum value, the motion vector MV of each detection region can be determined for each frame of interest. Further, the enlargement ratio Z when the error sum is the minimum value is the amount of movement.

By the way, when the reduction processing is performed, some pixels do not have an integer coordinate value of the pixels in the area.
On the other hand, in the image one frame before which the reduction processing has not been performed, the coordinate values of the pixels in the region are integer values. For this reason,
Linear interpolation is performed on the reduced image to calculate the signal level at the position where the coordinate value is an integer value. For example, as shown in FIG. 6, for an image of Ka × Ka pixels (1 / Z)
Image size is Kb x Kb
When the pixel size is reached, linear interpolation is performed and Ka
From the signal of the × Ka pixel, the signal level of the image in which the number of pixels is “Kb × Kb” is calculated. By calculating the error sum of the calculated signal level and the signal level one frame before at the position corresponding to the reduced image, the motion amount can be accurately determined.

When the image reference position CP is not clear, the error sum becomes the minimum value when the center of the detection area ARa is changed from the position Pa1 to the position Pa2 as shown in FIG.
When the error sum has the minimum value when the center of Rb is changed from the position Pb1 to the position Pb2, a point where the direction of the motion vector MV of the detection area ARa and the direction of the motion vector of the detection area ARb intersect is detected. As a result, the image reference position CP can be detected.

When making a right turn or a left turn, for example, the upper image of the front image moves horizontally. For this reason,
When the minimum value of the error sum is not detected even when the zoom-in operation is performed centering on the image reference position CP, the image in the detection area is moved in the horizontal direction to determine the minimum value of the error sum.
At this time, the movement amount of the detection area until the error sum reaches the minimum value can be set as the movement amount of the motion vector MV.

Next, since the image seems to be sucked into the image reference position CP when retreating,
The image at the time of retreat becomes substantially the same as the zoom-out operation image centered on the image reference position CP. Therefore, the enlargement ratio Z is set to "1" or less. That is, when moving backward, the movement of the image is opposite to that when moving forward. Therefore, a plurality of detection areas are set on the side edge side of the previous frame, and each detection area is reduced to (1 / Z). The error sum with the image of the frame of interest is calculated while moving the position of the designated area. Alternatively, each detection area of the frame of interest is reduced to (1 / Z), and the error sum with the image one frame after is calculated while moving the position of this reduced area. Further magnification Z
Is calculated, and the error sum is calculated while moving the position in the same manner. In this way, the motion vector MV at the time of backward movement can also be determined by detecting the position where the error sum is the minimum value.

When the motion vector MV is discriminated as described above, the amount of motion of the motion vector MV is small because the image in the distant view has little motion, and the motion amount of the motion vector MV is small in the image in the near view. The amount will increase.

FIG. 8 shows the configuration of the motion detection processing section 22. The image signal SDC is supplied to the size conversion processing section 221, and the image signal SDCa supplied from the 1-frame delay processing section 21 is the sum of errors. It is supplied to the calculation unit 222. The size conversion processing unit 221 divides the side edge portion of the front image into a plurality of detection areas, for example, a plurality of detection areas in units of 16 × 16 pixels and sets the detection areas. Further, in the size conversion processing unit 221, the enlargement ratio Z is set from the search control unit 225 described later, and the image signal FEz obtained by multiplying the image of the detection area by (1 / Z) and the image reference position CP.
The coordinate value Pz converted by multiplying the image of the detection area by (1 / Z) is supplied to the error sum calculation unit 222. The image signal FEz is a signal when the coordinate value is converted into an integer value by the interpolation process as shown in FIG. 6, and the coordinate value Pz is the coordinate when the coordinate value is converted into the integer value by the interpolation process. Value, that is, "K" in FIG.
The pixel position is “b × Kb”.

The error sum calculation unit 222 selects the signal at the position indicated by the coordinate value Pz from the size conversion processing unit 221 from the image signal SDCa, and calculates the error sum between the selected signal and the image signal FEz. It is calculated and notified to the comparison processing unit 223.

The comparison processing unit 223 compares the minimum error sum value with the error sum calculated by the error sum calculation unit 222. If the error sum minimum value is not set here, the error sum calculated first is set as the error sum minimum value. When the calculated error sum is smaller than the error sum minimum value, this data error sum is set as a new error sum minimum value, and the data holding unit 2 indicates that the error sum minimum value has been updated.
Notify 24. In addition, the minimum error sum and the error sum calculation unit 2
The search control unit 225 is notified by the signal ES that the comparison with the error sum calculated in step 22 has been completed. The minimum error sum value may be set in advance to a value larger than the calculated error sum.

The data holding unit 224 includes a search control unit 22.
When the enlargement ratio Z is notified from 5 and it is notified that the minimum error sum is updated, the notified enlargement ratio Z
Memorize If the enlargement ratio has already been stored, the stored enlargement ratio is updated with the notified enlargement ratio Z. Further, when the search control unit 225 notifies the completion of the change processing of the enlargement ratio by the signal ET, the stored enlargement ratio is used as the motion amount and the motion vector MV having the image reference position direction as the vector direction is classified into layers. Part 26
Supply to.

In the search control unit 225, the lower limit value of the enlargement ratio is set to "1" and the upper limit value is also preset. First, the lower limit value is set as the enlargement ratio Z and the size conversion processing unit 221 and the data are set. Notify the holding unit 224. After that, the comparison processing unit 223 outputs the error sum minimum value and the error sum calculation unit 222.
Each time it is notified that the comparison with the error sum calculated in step 1 is completed, the enlargement factor Z is sequentially increased to increase the size conversion processing unit 22.
1 and the data holding unit 224. After that, the enlargement ratio Z
When reaches the upper limit, the data holding unit 224 is notified of the completion of the enlargement ratio changing process.

When the minimum error sum obtained by the comparison processing unit 223 is not small, that is, when an image equal to the side edge portion of the front image cannot be detected, the search control unit 225 sets the enlargement ratio Z to "1". To the size conversion processing unit 221, and the image signal FEz in the detection area is calculated as the error sum calculation unit 2
22. Further, the control signal RP is supplied to the error sum calculation unit 222, and the signal at the position where the detection area of the image signal FEz is moved by a predetermined amount in the horizontal direction is selected from the image signal SDCa. Then, the image signal S is generated by the control signal RP.
By moving the position selected from DCa in the horizontal direction and determining the error sum minimum value, the motion vector MV of the image to be moved in the horizontal direction can also be obtained. Further, although not shown, a signal after one frame is supplied to the error sum calculation unit 222, or a detection region is set in the image one frame before and the signal of the image of the target frame is supplied to the error sum calculation unit 222. Thus, the amount of movement of the image that is sucked into the image reference position CP can also be determined.

In this way, the search direction is set to the image reference position CP.
Direction or the horizontal direction to detect the image position of the image in the detection area and the image of another frame where the error sum is the minimum. The motion vector MV can be correctly obtained even for an image of a person, a person, or an overtaking vehicle.

The layer classification unit 26 determines from the motion vector MV of each detection region what kind of motion pattern the front image is, and sets a layer based on the determined motion pattern to determine which one. The layer classification information LB indicating which layer the area belongs to is supplied to the information generation processing unit 27.

FIG. 9 shows the structure of the layer classification unit 26. The motion pattern determination unit 261 of the layer classification unit 26 accumulates the motion vector MV of each detection region supplied from the motion detection processing unit 22 on a frame-by-frame basis, and determines the motion pattern based on the accumulated motion vector MV.

Here, it is discriminated whether or not the direction of the motion vector MV of each detection area is the radial direction from the image reference position CP and all images are zoom-in operation directions that boil out from the image reference position CP. , All images are image reference position C
When it is the zoom-in operation direction that comes out of P, it is determined to be a straight-ahead operation. For example, when the vector direction of the motion vector MV is the radial direction from the image reference position CP as shown by the arrow in FIG.

Next, when it is not determined that the vehicle is going straight,
It is determined whether or not the direction of the motion vector MV of each detection area is the direction opposite to the radial direction, and all the images are in the zoom-out operation direction in which the images are sucked into the image reference position CP. When it is the zoom-out operation direction that is sucked in, it is determined to be the backward operation. For example, when it is set to the direction of the image reference position CP as indicated by the arrow in FIG. 10B and it is detected that all the images are in the zoom-out operation direction in which the image is sucked into the image reference position CP, it is determined to be the backward movement.

When it is not determined that the vehicle is going straight or backward, it is determined whether only a part is in the zoom-out operation direction. When only part is in the zoom-out operation direction, there is an overtaking vehicle. Determine. For example,
As shown in FIG. 10C, it is determined that there is an overtaking vehicle when it is detected that the movements of the detection regions at the left and right ends are in the zoom-in operation direction and only part of the movement is in the zoom-out operation direction. Further, when it is not determined that there is a straight ahead motion, a backward motion, and an overtaking vehicle, in the detection area on the upper side of the front image, when the vector direction of the motion vector MV is horizontal as shown in FIG. A right turn or a left turn operation is determined depending on the direction. Further, in a part of the detection area, when the motion vector MV has a horizontal vector direction as shown in FIG. The motion pattern MP determined in this way is notified to the layer creation determination unit 262.

The layer creation determination unit 262 determines whether or not the determined motion pattern has continued for a predetermined number of frames or more based on the motion pattern MP determined by the motion pattern determination unit 261. Here, the layer pattern information LP corresponding to the motion pattern determined when the motion pattern continues for a predetermined number of frames or more is generated.

Here, when it is continuously determined that the motion pattern is such that the entire screen is enlarged like a forward motion for a predetermined number of frames or more, for example, a layer for instructing the formation of a distant view, middle view, and near view layer. The pattern information LP is generated and notified to the classification processing unit 263. Also, when it is continuously determined that the upper part is a motion pattern in which the upper part moves in the horizontal direction such as a right turn or a left turn, a predetermined number of frames or more are continuously detected, the horizontal pattern is moved not only in the distant view, middle view, and near view layers. The layer pattern information LP instructing the generation of the layer including the image to be generated is generated and notified to the classification processing unit 263. Further, when it is continuously determined that the motion pattern includes an image that shrinks with time such as an overtaking vehicle or when reversing, the distant view, middle view, and near view layers are continuously identified. Not only that, the layer pattern information LP for instructing the creation of the receding layer including the image to be reduced is generated. Also, when it is continuously determined that a part of the movement pattern is a horizontal movement pattern, such as a transverse object, for a predetermined number of frames or more, an instruction to create a layer including an image that moves in the horizontal direction is issued. The layer pattern information LP is generated. In this way, when the determined motion pattern continues for a predetermined number of frames or more, the layer pattern information LP corresponding to the determined motion pattern MP is generated, so that a frame in which the motion pattern is erroneously determined occurs. However, the layer pattern information LP according to the correct motion pattern can be generated.

The threshold setting unit 264 uses the motion vector MV in a predetermined time range (for example, 30 frames before and after the frame of interest) to calculate the motion amount of the motion vector MV whose vector direction is the radial direction from the image reference position CP. Of the average value Vavg, the maximum value Vmax, and the minimum value Vmin of
Based on the average value Vavg, the maximum value Vmax, and the minimum value Vmin, a threshold value Th for classifying into the layer indicated by the layer pattern information LP is set and supplied to the classification processing unit 263.

For example, when it is indicated by the layer pattern information LP that the layers are divided into the distant view, the middle view, and the near view,
A threshold Th1 indicating the division position between the distant view layer and the middle view layer is calculated based on the equation (1). Further, the threshold Th2 indicating the division position of the middle view layer and the near view layer is calculated based on the equation (2).

[0049]

[Equation 1]

In setting the threshold value, a histogram of the amount of movement may be obtained as shown in FIG. 11, and the minimum value of the histogram may be used to obtain the threshold values Th1 and Th2. In this way, the threshold value Th is dynamically changed according to the distribution of the motion amount, so that the image is not classified into only one layer, and the layer is favorably classified according to the distribution of the motion amount. be able to.

In the classification processing unit 263, the threshold setting unit 264
Based on the threshold value Th and the motion amount of the motion vector MV, each detection region of each frame has the layer pattern information LP.
The layer classification is performed by discriminating to which of the layers the creation of which is instructed. Further, the detection area that moves in the horizontal direction and the detection area in the zoom-out operation direction are assigned to the corresponding layers. When the classification processing unit 263 completes the layer classification of each detection area of the frame of interest, the layer classification information LA indicating the result of the layer classification.
Is supplied to the classification correction unit 265.

Here, when the classification processing unit 263 creates three layers of the distant view, the middle view, and the near view, the motion vector MV from the motion detection processing unit 22 is used to detect the previous m frames for each detection region. Minute and the amount of movement of the subsequent n frames is calculated. For example, when the motion amount of the motion vector MV changes over time as shown in FIG. 12A in the detection area provided at the right end portion of the front image (the numbers in the figure indicate the motion amount). , The amount of motion of the detection region AR1 in the Fp frame, the amount of motion of the detection region AR1 in the Fp-1 to Fp-m frames, and the amount of motion in the Fp + 1 to Fp + n frames with respect to the detection region AR1 in the Fp frame. An average value of the movement amount is calculated from the movement amount of the detection area AR1. Furthermore, the detection area AR1
By comparing the average value calculated as the amount of movement of Fp with the thresholds Th1 and Th2 set as described above, this Fp
It is determined whether the detection area AR1 in the frame belongs to the distant view, the middle view, or the near view. In this way, for the detection area AR1 of the target frame, the average value of the movement amount is calculated using the movement amounts of the previous frame and the subsequent frame, and the detection area AR1 is layered using this average value. Even if an error occurs in the movement amount of the area AR1, this detection area AR1
The layers can be classified, and even if the amount of movement of each frame varies due to the difference in the size of the subject or the difference in the distance to the subject, the influence of these variations can be prevented. Further, by performing similar processing on other areas and other frames, each detection area is classified into any of the distant view, middle view, and near view layers as shown in FIG. 12B, and the layer classification information LA To the classification correction unit 265. Figure 1
In FIG. 2B, the detection area indicated by the cross hatch is classified as the near view, the detection area indicated by the diagonal lines is classified as the middle view, and the other detection areas are classified as the distant view.

The classification correction unit 265 refers to the layer classification information LA of each detection area, and corrects an area belonging to the same layer where the number of connected areas is smaller than a predetermined number to a layer matched with the surrounding area. For example, in FIG. 12B, in the area AR4-g of the Fp + 4 frame and the area AR6-h of the Fp-2 frame, since the areas belonging to the same layer are not connected, these areas are surrounded as shown in FIG. 12C. The layer classification information LB that corrects the combined area and indicates which area is set as which layer is supplied to the information generation processing unit 27.

When there is an overtaking vehicle, not only the above-mentioned distant view, middle view, and near view, but also a retreat layer including an image of the overtaking vehicle is generated as shown in FIG. 12D.
Here, since the passing vehicle has the direction of the motion vector MV that is opposite to that of the images of the distant view, the middle view, and the near view, the time axis direction of the backward layer is generated with the direction opposite to that of the other layers.

FIG. 13 shows the configuration of the information generation processing unit 27. The layer classification information LB supplied from the layer classification unit 26 is supplied to the motion amount average value calculation unit 271 and the image reading unit 272.

The motion amount average value calculation unit 271 calculates the average value of the motion amount for each layer for each frame using the motion amount of each detection area. For example, in the case of the Fe frame, when there are ny detection areas as distant view layers, an average value is calculated using the movement amounts of the ny detection areas and the average value is supplied to the image reading unit 272, and this calculation is performed. The calculated motion amount MYv is stored in the intermediate image information storage area 32 of the storage unit 30 as intermediate image information.

In the image reading unit 272, based on the layer classification information LB supplied from the layer classification unit 26, the motion amount M calculated by the motion amount average value calculation unit 271 from the side end portion.
The image signal SDC is extracted for each layer according to Yv to generate an intermediate image signal GFv for each layer, and the intermediate image signal GFv is supplied to the intermediate image signal interpolation unit 273.

Here, when the image signal SDC is read from the side edge portion by the signal amount corresponding to the movement amount MYv of each layer to generate an intermediate image signal, for example, an image based on the intermediate image signal is as shown in FIG. . 14A shows an image based on the intermediate image signal of the distant view layer, FIG. 14B shows an image based on the intermediate image signal of the middle view layer, and FIG. 14C shows an image based on the intermediate image signal of the near view layer.

In the intermediate image signal interpolating unit 273, the motion amount M
Even if the images of the layers with different Yv are pasted in the order of the distant view, the middle view, and the near view, the intermediate image signal GFv is corrected so that an uncovered area without an image does not occur. Is stored in the intermediate image information storage area 32 as the intermediate image signal GYv. In this correction of the intermediate image signal, if all of the layers in the front are interpolated, the layers in the back will be hidden. Therefore, for the innermost layer, the entire gap is interpolated, and for the layer located in the middle, only signals that are horizontally adjacent to the gap created by the region belonging to the layer before this layer. Interpolation processing is performed to generate an image signal of a gap area where there is no image and paste it to the intermediate image signal. By performing the interpolation process in this manner, the image based on the intermediate image signal GFv shown in FIG. 14A is interpolated using signals in which the image-less portion (the shaded area) is horizontally adjacent, and the image shown in FIG. The image is as shown in. Similarly, the image based on the intermediate image signal GFv shown in FIG. 14B becomes an image corrected as shown in FIG. 15B. When the layer located closest to the foreground is, for example, the near view layer shown in FIG. 14C,
Since the uncovered area does not occur in the layer located closest to the foreground, interpolation processing is unnecessary, and the near view layer shown in FIG. 15C is the same as that in FIG. 14C.

In the intermediate image information storage area 32 of the storage section 30, the motion amount MYv for each layer generated in the intermediate image information generation block 25 and the intermediate image signal GYv for each layer are associated with each other for each frame, It is stored as information.
Here, since the amount of motion of the intermediate image signal of the distant view layer stored in the intermediate image information storage area 32 is small, FIG.
As shown in A, the signal amount of the image signal decreases. In addition, the intermediate image signal of the near view layer has a large amount of motion and thus has a large amount of signal as shown in FIG. 15C. In the middle view layer, the signal amount is between the distant view and the middle view, as shown in FIG. 15B. It should be noted that, in the backward layer, the movement directions are opposite, so that in the intermediate image signal of the backward layer, FIG.
The frame direction is opposite to that in FIG. 15C.

When it is instructed to create a horizontal movement layer based on the layer pattern information LP, the image signal of the detection area in which the direction of the motion vector is the horizontal direction is used as the intermediate image signal of this horizontal movement layer. Set.
Here, in an image that moves in the horizontal direction, an image that moves outward from the front image and an image that enters the front image occur. For example,
When making a right turn, the image enters from the right end of the front image and the image exits from the left end. Therefore, the horizontal movement layer of the image moving outward has the same time axis direction as that of the distant view layer, and the horizontal movement layer of the image moving inward has the same time axis direction as that of the backward view layer. Direction.

In this way, the intermediate image information storage area 3
In 2, the intermediate image signals of the distant view, middle view, and near view layers and the intermediate image signals of the backward layer and the horizontal movement layer are generated and stored according to the amount of motion. Further, the amount of movement of each layer is also stored together as described above.

Next, a case will be described in which the peripheral image signal SDR of the image on the right side surface is generated using the intermediate image information stored in the intermediate image information storage area 32 of the storage section 30.
As described above, when the peripheral image signal SDR is generated, the motion amount MYv stored as the intermediate image information is read from the intermediate image information storage area 32, and the signal read amount of the intermediate image signal GYv for each layer is set as the motion amount MYv. Based on. Further, an intermediate image signal having a signal amount corresponding to the motion amount MYv is used for each layer, and the distant view layer, the middle view layer, and the near view layer are pasted in this order. Further, when a layer different from the distant view, middle view, and near view layers is provided, the image pasting process of this layer can also be performed to generate the peripheral image signal SDR. By supplying the generated peripheral image signal SDR to the projector 12R in synchronization with the output timing of the image signal SDC of the front image, not only the front image but also the right side image can be continuously displayed.

FIG. 16 shows the configuration of a peripheral image signal generation block 40 which reads the intermediate image information stored in the intermediate image information storage area 32 and generates a peripheral image signal. In the image generation control unit 41 of the peripheral image signal generation block 40, the motion amount MYv for each layer stored as the intermediate image information in the intermediate image information storage area 32 is read in the frame order, and the layer is read based on the read motion amount MYv. The read amount of the intermediate image signal for each is determined. Based on the read amount determined by the image generation control unit 41, the intermediate image information GYv is read from the intermediate image information storage area 32.
Is read for each layer, and the pasting units 421-1 to 421 for each layer provided in the layer image generation processing unit 42 are read.
-Supply to 5.

Here, the intermediate image signal of the horizontal moving layer which moves in the horizontal direction when turning right or left is supplied to the pasting unit 421-1. The intermediate image signal of the distant view layer is supplied to the pasting unit 421-2, and the intermediate image signal of the middle view layer and the intermediate image signal of the near view layer are pasted to the pasting unit 421-.
Supply to 3,421-4. Further, the intermediate image signal of the backward layer including the image of the overtaking vehicle is attached to the pasting unit 421-5.
Supply to.

An image shift unit 423-1 to be described later is connected to the pasting unit 421-1, and the image shift unit 423 is connected.
-1 to the image signal supplied from the intermediate image information storage area 3
The image signal of the corresponding layer read from 2 is pasted. The image signal of the horizontal movement layer obtained by pasting the image by the pasting unit 421-1 is supplied to the one-frame delay unit 422-1 and the distant view pasting unit 441 of the image signal generation unit 44. .

The one-frame delay section 422-1 delays the image signal supplied from the pasting section 421-1 by one frame and supplies the delayed image signal to the image shift section 423-1. The image shift unit 423-1 moves the image based on the image signal supplied from the 1-frame delay unit 422-1 in the horizontal direction based on the movement amount MYv of the horizontal movement layer supplied from the image generation control unit 41. . Further, the image signal of the image moved in the horizontal direction is supplied to the pasting unit 421-1.

The image based on the image signal delayed by one frame as described above is moved in the horizontal direction based on the movement amount MYv supplied from the image generation control unit 41, and
The image signal of the moving image can be generated by pasting the intermediate image signal GYv of the horizontal moving layer read by the amount of movement MYv from the intermediate image information storage area 32 to the image moved in the horizontal direction.

Similarly, 1-frame delay units 422-2 to 42
The image based on the image signal delayed by one frame at 2-5 is moved in the horizontal direction based on the motion amount MYv of each layer supplied from the image generation control unit 41 at the image shift units 423-2 to 423-5. At the same time, the intermediate image signal GYv read from the intermediate image information storage area 32 by the amount of movement of each layer is supplied to the pasting units 421-2 to 421-5 with respect to the horizontally moved image. By pasting the image based on the read intermediate image signal GYv,
An image signal in which images sequentially move can be generated for each layer.

Since the intermediate image signal is a signal read from the front image signal, the image based on this intermediate image signal is an image on the same surface as the screen 10C. However, the screen 10R for displaying the peripheral image is provided so as to be inclined with respect to the screen 10C on the front side. Therefore, the image of each layer is pasted using the image signal of the layer in which the direction of the motion vector is the direction of the image reference position CP or the opposite direction, such as the distant view, middle view, near view layer, and backward layer. If the peripheral image signal SDR is generated and an image is displayed, the screen 10R is displayed.
The image displayed at is not in the correct shape due to the inclination of the screen 10R with respect to the screen 10C. Therefore, the image signal of the layer in which the direction of the motion vector is the direction of the image reference position CP or the opposite direction is the projective transformation unit 4
3 is supplied to the respective conversion processing units 431 to 434, and when the image of each layer is displayed on the screens 10L and 10R, the projective conversion is performed so that the image correctly moves in the vector direction of the motion vector MV. Further, since the direction of the motion vector MV is the horizontal direction, the image of the horizontal movement layer is not subjected to projective transformation like the distant view, middle view, near view layer, and backward layer.

Conversion processing units 431 to 43 of the projective conversion unit 43
In 4, the projective transformation is performed on the image signals supplied from the pasting units 421-2 to 421-5 so that the image has a shape corresponding to the orientation of the screen 10R. Here, when the image signals are sequentially read from the front image, the display surface 10CR and the screen 10R of the read image are not on the same surface as shown in FIGS. 17A and 17B, so the image read from the front image is displayed on the screen 10R. When displayed, the image on the display surface 10CR is projected as it is on the surface of the screen 10R, and the shape is not correct.

Therefore, the projective transformation is performed so that the vertical direction is expanded from the center Czm of zoom-in or zoom-out of the image at an expansion ratio proportional to the distance, and the speed is horizontally changed from the center Czm of zoom-in or zoom-out in the horizontal direction. The magnification is such that it is proportional to the distance.

Here, as shown in FIG. 17C, the length from the screen front edge of the right side image to the center Czm is L, the length in the peripheral direction of the right side image is A, and the front side image and the right side image are formed. Assuming that the proportional constant based on the angle “θs” is α, the position (xa, ya) of the image directly read from the front screen edge and the position (Xa, after conversion processing so that the image can be correctly displayed on the right side image. Ya) has the relationship of Expression (3) in the vertical direction and the relationship of Expression (4) in the horizontal direction. Therefore, as shown in FIG. 17D, as the image signal of (Xa, Ya) on the screen 10R, by using the signal of the position (xa, ya) that satisfies the equations (3) and (4), the image signal of FIG. The right side image can be correctly displayed on the screen 10R as shown in FIG.

[0074]

[Equation 2]

The image signal of the distant view layer after the projective conversion obtained by the conversion processing unit 431 is supplied to the distant view pasting unit 441 of the image signal generation unit 44. Also, the conversion processing unit 4
The image signal of the middle-view layer after projective conversion obtained in 32 is supplied to the middle-view pasting unit 442 and the conversion processing unit 4
The image signal of the near view layer after the projective transformation obtained in 33 is supplied to the near view pasting unit 443. Furthermore, the image signal of the receding layer obtained by the conversion processing unit 434 is supplied to the reduction and pasting unit 444.

In the distant view pasting unit 441, the pasting unit 4
21 and the image signal supplied from the conversion processing unit 431, an image signal in which the image of the distant view layer is pasted on the image of the horizontal movement layer is generated and supplied to the middle-view pasting unit 442.

In the middle ground pasting unit 442, the conversion processing unit 4
32 and the image signal supplied from the distant view pasting unit 441, an image signal in which the image in the middle view layer is pasted to the image in which the image in the distant view layer is pasted is generated and supplied to the near view pasting unit 443. .

In the near-view pasting section 443, the conversion processing section 4
33 and the image signal supplied from the middle ground pasting unit 442, an image signal in which the image in the middle ground layer is pasted to the image in which the image in the middle ground layer is pasted is generated and supplied to the reduction pasting unit 444. To do.

In the reduction pasting unit 444, the conversion processing unit 4
34 and the image signal supplied from the near-field pasting unit 443, an image signal in which the image in the near-field layer is pasted with the image in the receding layer whose image shrinks with time is generated. The image signal generated by the reduction and pasting unit 444 becomes the image signal of the side image to which each layer from the horizontal movement layer to the reduction layer is pasted.

Therefore, by supplying the image signal generated by the reduction and pasting unit 444 to the projector 12R as the peripheral image signal SDR, the image on the right side which is continuous with the front image can be displayed on the screen 10R.

When the intermediate image signal stored in the intermediate image information storage area 32 is not subjected to the process of interpolating a portion having no image, or the reduction paste unit 4
When an image-free portion is generated in the image signal generated at 44, a pre-interpolation processing unit is provided to determine what layer image is combined in an area adjacent to the image-free portion, and Interpolation processing is performed using an image of a deep layer. For example, when there is no image in a part that is horizontally adjacent to the region where the images of the middle-view layer and the foreground layer are pasted together, interpolation processing can be performed using the image of the deep-view middle-layer. A good peripheral image signal SDR without image loss can be generated.

Further, by generating the peripheral image signal SDL in the same manner as the peripheral image signal SDR and supplying the same to the projector 12L, the image on the left side surface which is continuous with the front image can be displayed on the screen 10L.

Further, the storage unit 30 stores intermediate image information. Since this intermediate image information is an amount of motion and an intermediate image signal for each layer, when storing image signals of peripheral images. The amount of information can be reduced in comparison. Therefore, it is possible to display an image in a wide angle of view without using a large capacity recording medium or the like.

By the way, in the above-mentioned embodiment, from the motion vector MV of the detection area provided on the side edge side of the front image,
Layering images such as distant view, middle view, and near view and determining the amount of motion are performed to generate intermediate image information and store it in the intermediate image information storage area 32. Using the intermediate image information accumulated in the information accumulation area 32, the intermediate image signal GYv is read according to the motion amount MYv of each layer and the peripheral image signal is generated by pasting the image. The peripheral image signal can also be generated without using the motion vector MV. Next, as a second embodiment, the motion vector MV
An image signal processing device capable of generating a peripheral image signal without using the will be described.

In the image signal processing apparatus 55 of the second embodiment, a projection image in the lateral direction is generated from the front image by geometric transformation, and the projection image is subjected to integration processing to generate an integral image and the integration is performed. An image is divided into layers from images, a motion amount MYs for each layer and an intermediate image signal for each layer are generated and stored in the intermediate image information storage area 32 as intermediate image information.

FIG. 18 shows the structure of the image processing apparatus 55. The image signal SDC of the front image is sent to the geometric conversion unit 61.
Is supplied to. The geometric transformation unit 61 uses the image signal SDC of the front image to generate a projection image signal SDP of the image in which the front image is displayed on the projection plane that is the plane in the traveling direction by the geometric transformation process. The projected image signal SDP generated by the geometrical conversion unit 61 includes a motion determination image generation processing unit 63 of the motion detection block 62, an integral image generation processing unit 64, and an information generation processing unit 72 of the intermediate image information generation block 70.
Is supplied to.

The motion discrimination image generation processing unit 63 extracts the projection image signal SDP within a predetermined range from the end portion in contact with the front image for each frame from the projection image signal SDP supplied from the geometric conversion unit 61 and sequentially pastes it. The motion discrimination image signal SMD is generated by the attachment. This motion determination image signal SMD is supplied to the motion amount determination unit 65.

The integral image generation processing unit 64 shifts the position in the horizontal direction while projecting the image signal SDP for ms frames.
Is added and averaged to generate one integrated image signal. Also, ns integrated image signals are generated with the position shift amount being ns different values. The ns integrated image signals SA generated by the integrated image generation processing unit 64 are supplied to the motion amount determination unit 65.

The motion amount determining section 65 compares the ns integrated image signals SA with the motion determining image signal SMD to detect an image based on the integrated image signal SA that matches the image based on the motion determining image signal SMD. Then, the amount of movement of the target frame at a predetermined position is determined based on the detection result. Here, in the integral image, when the position shift amount and the image movement amount match, this subject appears in the integral image. Further, since ns integrated images are generated with different shift amounts, ns images with different movement amounts are indicated by the integrated image signal SA.
Therefore, the correlation between the ns integrated image signals SA and the motion determination image signal SMD is determined, and based on this determination result,
It is possible to determine the amount of movement of the subject forming the movement determination image, that is, the amount of movement of the target frame at a predetermined position. Here, since the movement amount is small when the image of the subject is a distant view, and the movement amount is large when the image of the subject is a close view,
This amount of movement indicates the depth of the image of the subject.
In this way, when the motion amount determining unit 65 determines the motion amount at each position of the projected image, the motion amount information MF indicating the determination result.
Is generated and supplied to the layer classification unit 71 and the information generation processing unit 72 of the intermediate image information generation block 70.

The layer classification unit 71 generates layer classification information LB indicating which of a plurality of layers the predetermined position of the frame of interest belongs to, based on the supplied motion amount information MF, and supplies it to the information generation processing unit 72. .

In the information generation processing unit 72, the average value is calculated for each frame using the movement amount of the preceding and following frames for each frame, and the intermediate image information accumulation of the accumulation unit 30 is performed as the movement amount MYs of the corresponding frame of each layer. Supply to the area 32. Further, based on the obtained motion amount MYs for each layer and the layer classification information LB supplied from the layer classification unit 71,
The image signal SDP supplied from the geometric conversion unit 61 is classified for each layer to generate an intermediate image signal GFs. That is, the image signals are read out in the horizontal direction by an amount corresponding to the amount of movement and are sequentially attached to the corresponding layers to generate the intermediate image signal GFs for each layer similar to FIG.

Further, when the moving directions of the images of the distant view layer, the middle view layer and the near view layer are used as the reference, the moving direction is opposite in the backward layer. Therefore, an intermediate image signal of the receding layer can be generated by reading the projective plane image signal from the end side which is in contact with the front image by the absolute value of the movement amount and performing horizontal reversal and pasting.

Further, interpolation processing is performed on the created intermediate image signal GFs for each layer so that an uncovered area without an image does not occur, and a portion without an image is embedded. The intermediate image signal GYs for each layer obtained by performing the interpolation processing in the information generation processing unit 72 is associated with the motion amount MYs of each layer and stored in the intermediate image information storage area 32 as intermediate image information. .

The peripheral image signal of the right side image can be generated by reading the intermediate image information thus generated from the intermediate image information storage area 32 and processing it in the same manner as in the peripheral image signal generation block 40 described above. .

Next, each block will be described in detail. FIG. 19 is a diagram for explaining the geometric conversion processing in the geometric conversion unit 61. In FIG. 19, the position O is the shooting position of the front image. The geometric transformation unit 61 projects the image at the position p of the front image onto the position q on a plane parallel to the traveling direction, and generates a projected image from the front image. Here, the relationship between the position xb in the traveling direction on the projection plane and the horizontal position Xb on the front image is as shown in Expression (5). Further, the relationship between the position yb in the height direction on the projection plane and the vertical position Yb on the front image is as shown in Expression (6).

[0096]

[Equation 3]

In the equations (5) and (6), "θg"
Indicates the angle at which the horizontal angle of view of the front image is halved,
When the horizontal angle of view of the camera that has photographed the front image is clear, the half angle of this horizontal angle of view is set to “θg”.
When the horizontal angle of view of the camera is not clear, a value preset according to the focal length of the lens used is used. “D” is a value obtained by halving the horizontal length of the front image (horizontal length of the image frame).

By using the equations (5) and (6) as described above, the image signals of the left and right projected images can be generated in frame units from the image signal of each frame of the front image.
When the input image is the front image of the vehicle-mounted forward system, the left and right projected images are lateral images, and the amount of movement of these projected images is proportional to the vehicle speed. In addition, the amount of movement of the projected image is inversely proportional to the depth, and when the projected image is a distant view image with a large depth, the amount of movement is small, and when the projected image is a close view image with a small depth. The amount of movement increases.

FIG. 20 is a diagram for explaining the operation of the motion discrimination image generation processing unit 63. In the motion determination image generation processing unit 63, as shown in FIG. 20A, a predetermined number of pixels wa (for example, 4 pixels) are horizontally set from the front image side end of the projection plane image (the right end in the right projection plane image) for each frame. 20B, the image signal SMD of the motion determination image is generated by cutting out the image signal with the width of) and pasting the cut out image signal for a predetermined number of frames in the frame order as shown in FIG. 20B. In the motion determination image, when the amount of movement of the subject is wa pixels / frame in the horizontal direction and is equal to the number of cut-out pixels wa of the image signal (house in FIG. 20), the subject image is continuous without discontinuity points. It becomes an image. In an image of a subject (mountain in FIG. 20) whose movement is slower than the cutout of the image signal, a portion that is repeatedly cut out occurs, and thus a discontinuity of the image occurs at a connection portion of the cutout image signals.
Further, in the image of a subject (tree in FIG. 20) that moves faster than the cutout of the image signal, a portion that is not cutout occurs because the movement is fast, and image discontinuity occurs at the connecting portion.

The integral image generation processing section 64 generates an integral image signal SA used for discriminating the motion amount of each pixel of the motion discrimination image. Here, the integral image will be described with reference to FIG. For example, when a subject Ja moving to the right and a subject Jb moving to the right at twice the speed of the subject Ja are photographed, the photographed image is as shown in FIG. 21A. Here, the image of the next frame F2 is shifted with respect to the first frame F1 by the pixel amount dx of the subject Ja in the direction opposite to the moving direction of the subject Ja. Similarly, the image of the next frame F3 is also shifted with respect to the frame F2 by the pixel amount dx of the subject Ja in the direction opposite to the moving direction of the subject Ja.
In this way, if an image obtained by adding up and averaging the shifted images of each frame is used as an integral image, the position of the subject Ja is constant and the position of the subject Jb moves, so that the signal of the image signal of the subject Jb is obtained. The level becomes smaller than that of the subject Ja, and only the subject Ja can be extracted by the integrated image as shown in FIG. 21C.

As shown in FIG. 21D, when the integrated image is generated by doubling (2dx) the image shift amount of each frame, the position of the subject Jb becomes constant and the position of the subject Ja moves. Therefore, as shown in FIG. 21E, only the subject Jb can be extracted by the integral image.

As described above, the image of each frame is moved by a predetermined amount, added and averaged to form an integral image, and a plurality of integral images are generated by changing the shift amount of the image of each frame. Then, by determining in which integral image the subject is correctly displayed, the amount of movement of each subject can be determined using the integral image even if one image includes subjects having different amounts of movement. That is,
Since the amount of motion of the subject forming the image of each frame can be determined, the amount of motion of the target frame at a predetermined position can be determined.

FIG. 22 is a diagram showing the structure of the integral image generation processing unit 64. The projection image signal SDP supplied from the geometric conversion unit 61 is calculated by the calculation processing unit 6 of the integral image generation processing unit 64.
41-1 to 641-ns.

In the arithmetic processing unit 641-1, the projection plane image signal moved horizontally by a preset number of pixels SH-1 from the previous frame is added by adding ms frames (for example, 10 frames). The signals are calculated sequentially. Further, a signal obtained by multiplying this addition signal by 1 / m and averaging is supplied to the pixel cutout unit 642-1.

In the pixel cutout unit 642-1, an image signal of a predetermined number of pixels wb is horizontally cut out from the supplied image signal for each frame from the front image side end (the right end in the right projection plane image). Then, the cut-out image signals are sequentially pasted and output as an integrated image signal SA-1. The integrated image signal SA-1 output from the image cutout unit 642-1 becomes an image signal indicating a subject having a motion amount of a preset number of pixels SH-1.

Similar to the arithmetic processing unit 641-1, the arithmetic processing units 641-2 to 641-ns also have a preset number of pixels SH-2 to
Projection plane image signals that have been moved in the horizontal direction by SH-ns more than the previous frame are added for ms frames to sequentially calculate addition signals, and the addition signals are multiplied by 1 / m to be averaged and then pixel cutout is performed. And supply to the units 642-2 to 642-ns. Furthermore, the pixel cutout units 642-2 to 642-ns cut out image signals of wb for a predetermined number of pixels, sequentially paste them, and output them as integrated image signals. This pixel cutout unit 642
-2 to 642-ns output integrated image signals SA-2 to S
A-ns is a preset number of pixels SH-2 to SH-
The image signal shows a subject having a movement amount of 3.

Assuming that the number of pixels cut out by the motion discriminating image generation processing unit 63 and the pixel clipping unit 642 of the integral image generating processing unit 64 are equal (wa = wb), the consistency of the position of the subject in the motion discriminating image and the integral image Can be taken.

FIG. 23 is a diagram showing the operation of the integral image generation processing unit 64. When the image shift amount is the number of pixels SH-r, the image is moved and added by the number of pixels SH-r (<wb) for each frame as shown in FIG. 23A.
As shown in B, the mountain image is extracted. When the image shift amount is set to the pixel number SH-s (= wb) larger than the pixel number SH-r, the image is moved and added by the pixel number SH-s for each frame as shown in FIG. 23C. As shown in FIG. 23D, an image of a house having a larger amount of movement than a mountain is extracted. Further, when the image shift amount is set to the pixel number SH-t (> wb) larger than the pixel number SH-s, the image is moved and added by the pixel number SH-t for each frame as shown in FIG. 23E. As a result, as shown in FIG. 23F, an image of a house having a larger amount of movement than the mountain is extracted.

Here, when the motion discriminant image generator 63 generates a motion discriminant image, the frame Fa, which was first clipped, is used as a reference for starting the segmentation, and a segment amount (wa = wb) equal to the motion discriminant image. When the images are cut out for the same number of frames to generate an integral image, the integral image and the motion discriminating image have the same horizontal image length, and the phase of the subject matches between the integral image and the motion discriminating image.

Further, in the motion discriminating image, when the clipping amount and the motion amount are not equal, the object has a discontinuous shape, but also in the case of the integral image, similarly, the pixel number SH-r is the pixel number w.
When the number is smaller than b, the images are cut out with an overlap, so that the images become discontinuous according to the amount of movement, as shown in FIG. 23G. Further, when the number of pixels SH-s is equal to the number of pixels wb, the images are continuous according to the amount of movement as shown in FIG. 23H. Further, when the number of pixels SH-t is larger than the number of pixels wb, a cutout portion of the image causes a missing portion, resulting in a discontinuous image corresponding to the amount of movement as shown in FIG. 23J. Therefore, the motion amount determining unit 65 described below performs matching between the motion determination image and the integral image, and determines the amount of movement of the subject based on the number of pixels SH, which is the image shift amount of the integral image determined to match. be able to.

The motion discriminating unit 65 obtains a correlation value between each pixel of the motion discriminating image and the integral image, and discriminates which integral image each pixel of the motion discriminating image is based on the correlation value. Then, the amount of movement of each pixel of the movement determination image is determined.

In calculating the correlation value, the pixels of the image A and the image B are
When calculating the correlation value with the pixel A, a predetermined range around the pixel of interest (pixel for which the correlation value is calculated) of the image A is set, and the correlation value of the pixel of interest is obtained using the image signal of this predetermined range. . For example, ± mvc pixels in the x direction centering on the pixel of interest,
A rectangular range of ± n vc pixels (for example, a rectangular range of 31 pixels in the horizontal direction and 5 pixels in the vertical direction around the pixel of interest) is set in the y direction, and the correlation value VC is calculated based on the equation (7).

[0113]

[Equation 4]

In the equation (7), the signal level of each pixel in the rectangular range of the image A is DAi (i = 1 to 1).
(mvc × nvc)), the signal level of each pixel in the rectangular area of image B is DBi (i = 1 to (mvc × nvc)),
D is the average value of the signal level of each pixel in the rectangular range
Aav and DBav.

For example, when it is determined based on the equation (7) that the mountain image in the motion determination image shown in FIG. 24A matches the integral image shown in FIG. 24B, each pixel forming the mountain image is determined. It can be determined that the motion amount is a motion amount corresponding to the number of pixels SH-r. Further, when it is determined that the house image in the motion determination image shown in FIG. 24A matches the integral image shown in FIG. 24C, the amount of movement of each pixel forming the image of the house becomes the number of pixels SH-s. It can be determined that the amount of movement corresponds. Furthermore, FIG.
When it is determined that the tree image in the motion determination image shown in FIG. 4A matches the integral image shown in FIG. 24D, the amount of movement of each pixel forming the image of this tree is the number of pixels S.
It can be determined that the amount of movement corresponds to H-t. In this way, when the motion amount determination unit 65 determines the motion amount for each pixel, the motion amount information MF indicating the determination result.
Is supplied to the layer classification unit 71 and the information generation processing unit 72.

Although not shown, the integral image generation processing unit 6
In FIG. 4, the image shifting directions are shown in FIGS. 23A, 23C, and 2.
If it is set so as to include the direction opposite to 3E (for example, the image shift amount SH is set to 48 pixels to -24 pixels), an image of an overtaking vehicle that moves in the opposite direction with respect to an image of a tree, a house, or a mountain is displayed. Can be extracted. Further, in order to determine the amount of movement of the image in the direction opposite to the direction of the distant view layer, as shown in FIG. 25A, the image of each frame is cut out, the image is horizontally flipped, and then pasted. Fig. 25
The motion determination image shown in B is generated. In this case, FIG.
In the motion determination image shown in (1), an image in which the moving direction is the reverse direction, such as an overtaking vehicle, is correctly displayed in the same direction as the distant view, middle view, and near view layers. Note that FIG. 25C is a motion determination image when the image is pasted without being horizontally reversed. Therefore, the image signal of the integral image in which the image shift direction is the reverse direction and the image signal of the motion determination image generated by inverting and pasting the cut out image are sequentially compared from the reference frame. Thus, the amount of movement of the subject moving in the opposite direction can be determined.

The layer classification unit 71 layers each pixel of the motion discrimination image in the depth direction based on the motion amount information MF. For example, assuming that the depth is layered in advance into three layers of a distant view, a middle view, and a near view and four layers of a layer having a movement in the opposite direction, layer classification is performed to determine which layer each pixel belongs to.

In this layer classification, the above-mentioned threshold setting unit 2
Similar to 64, a threshold value is set, and the threshold value is compared with the motion amount of each pixel to determine which layer each pixel belongs to and generate the layer classification information LB. The generated layer classification information LB is supplied to the information generation processing unit 72.

FIG. 26 shows the structure of the information generation processing section 72. Motion amount average value calculation unit 7 of the information generation processing unit 72
In 21, based on the motion amount information MF from the motion amount determining unit 65 and the layer classification information LB supplied from the layer classifying unit 71, the average value of the motion amounts of the preceding and following nz frames is calculated for each frame in each layer. , The speed of the frame of interest of the layer. The calculated motion amount MYs for each frame of each layer is supplied to the image reading unit 722 and stored in the intermediate image information storage area 32.

The image reading unit 722 extracts a signal for each layer from the image signal SDP of the projected image according to the motion amount MYs calculated by the motion amount average value calculation unit 721 based on the layer classification information LB, Intermediate image signal G for each
Generate Fs. The intermediate image signal GFs is supplied to the intermediate image signal interpolation unit 723. At this time, since the amount of motion of the distant view layer is small and the amount of motion of the near view layer is large, the intermediate image of the distant view layer has a short horizontal length, and the intermediate image of the near view layer has a long length, as in FIG. 14 described above. Further, since the movement is in the opposite direction in the backward layer, the intermediate image of the backward layer is created by reading the images from the side image by the absolute value of the amount of movement and reversing and sticking the images.

Similar to the intermediate image signal interpolating unit 273, the intermediate image signal interpolating unit 723 pastes the images of the respective layers having different motion amounts MYs in the order of distant view, middle view, and near view, regardless of the difference in motion amount. Uncovered without images
d) The intermediate image signal GFs is corrected so that no area is generated, and the corrected signal is stored in the intermediate image information storage area 32 as the intermediate image signal GYs.

When there is a backward layer such as an overtaking vehicle, the interpolation processing is not performed because it is a layer before the near view layer. When a horizontally moving layer indicating a building or the like that appears in the distant view when turning right or left is provided, the interpolation process is performed in the same manner as the distant view layer. It should be noted that in the determination of the subject in the backward layer or the horizontal movement layer, the motion pattern is determined based on the movement direction of the motion determination image of the right projection image and the left projection image in the same manner as in FIG. 10, and the determined motion pattern is determined. It can be determined from. For example, it is possible to determine that a right turn or a left turn is being performed when the left image and the right image have different movements in the upper part of the screen. Further, the left image and the right image show the same direction of movement of the upper part of the screen, and a subject whose movement is in the opposite direction can be discriminated as a backward layer.

Next, the operation of generating a peripheral image using the intermediate image information of each layer will be described. In generating the peripheral image, the peripheral image signal can be generated in the same manner as the above-described peripheral image signal generation block 40. That is, the image of each layer is read out by the amount of movement, and the distant view layer, the middle view layer, and the near view layer are attached in this order. Further, even when the horizontal movement layer or the backward layer is set, the peripheral image signal can be generated in the same manner as the peripheral image signal generation block 40.

In addition, in image pasting, for example, when a right side image is generated as a peripheral image, the same frame portion of each layer is pasted so as to overlap at the left end of the generated right side image. Also, the right end of the projected image converted from the front image first and the left end of the generated side image are made equal. After that, the image is moved for each layer based on the amount of movement of each layer, and the next image is pasted to this moved image for each layer, so as to be continuous with the projection plane image as shown in FIG. 27. A side image can be generated.

By the way, the image generated by pasting the intermediate image for each layer using the intermediate image information is an image on the same plane as the projection plane as shown in FIG. On the other hand, the screen 10R on which the peripheral image is displayed is installed with an inclination with respect to the projection plane. Therefore, when the image of each layer is pasted and displayed on the screen 10R, the image conversion process is performed so that the image moves in a correct shape.

Here, as shown in FIG. 28, 1/2 of the horizontal length of the front image is the distance d, and the position O which is the photographing position of the front image is on the normal line from the center of the screen 10R. When the screen 10R is tilted like this,
The image signal at the position (Xc, Yc) on the screen 10R is
By extracting the image signal at the image position (xc, yc) before conversion calculated based on the equations (8) and (9), the peripheral image signal SDR of the right side image after projective conversion can be easily generated. Also, by displaying the right side moving image on the screen 10R using this image signal SDR, the right side image having a high sense of presence can be displayed.

[0127]

[Equation 5]

The peripheral image signal SDL is also the peripheral image SDR.
Of course, it can be generated in the same manner as.

Further, the processing performed in each processing block described above may be realized not only by hardware but also by software. The structure in this case is shown in FIG.

As shown in FIG. 29, the computer 90 has a built-in CPU (Central Processing Unit) 901. The CPU 901 is provided with a ROM 9 via a bus 920.
02, RAM 903, hard disk drive 90
4, the input / output interface 905 is connected. Furthermore, an input unit 911, a recording medium drive 912, a communication unit 913, and an image input / output unit 914 are connected to the input / output interface 905.

When a command is input from the input unit 911 configured by using an operation input unit such as a keyboard or a mouse or a voice input unit such as a microphone, the command is supplied to the CPU 901 through the input / output interface 905. To be done.

The CPU 901 has a ROM 902 and a RAM.
The program stored in the hard disk drive 903 or the hard disk drive 904 is executed to perform the processing according to the supplied instruction. Furthermore, ROM902 and RAM90
3 or the hard disk drive 904 also stores a program for executing the processing in each block described above, and also generates a side image.

Here, the image processing program for generating the intermediate image information and for generating and outputting the peripheral image signal from the intermediate image information should be stored in advance in the ROM 902 or the hard disk drive 904 which is an information recording medium. You can

Alternatively, a removable information recording / transmission medium composed of a removable recording medium using magnetism or light or a semiconductor element, for example, an optical disk such as a floppy (R) disk or a CD-ROM, or an optical disk such as an MO disk. For recording the image generation program on a magnetic disk, tape cartridge, or semiconductor memory,
These removable information recording / transmission media are attached to the recording medium drive 912 to read the recorded image processing program, and the read program is read through the input / output interface 905 and the bus 920 to a hard disk drive.
It may be installed by storing it in the drive 904 or the like.

Further, a communication unit 913 is provided to receive an image processing program supplied using a wired or wireless transmission path, for example, a network such as LAN or the Internet, or satellite broadcast wave or terrestrial broadcast wave. , The received image processing program is input / output interface 905.
And hard disk drive 9 via bus 920
It can also be installed on 04 etc.

Here, when the image processing program is executed to generate the intermediate image information, the hard disk drive 904 or the recording medium drive 912.
The image signal SDC of the front image recorded on the recording medium mounted on the recording medium is read out, and the above-described processing is performed to output intermediate image information, for example, together with the image signal SDC to the hard disk drive 904 or the recording unit. Media drive 912
It is recorded on the recording medium mounted on the. When the output of the image signal of the peripheral image is requested, the intermediate image information is read to generate the image signals SDR and SDL of the left and right side images, and the image signals SDR and SDL are the image signals of the front image. It is output at the same timing as SDC.
Therefore, continuous images with a wide angle of view can be displayed using the front and left and right screens.

Thus, according to the above-mentioned embodiment,
It is possible to determine the movement of the front image and generate a peripheral image in a different direction based on the movement determined from the front image. Therefore, since moving images in different directions can be simultaneously generated from front moving images captured by a video camera or the like, multi-screen images with a high sense of presence and a wide angle of view can be presented.

Also, since images in different directions can be generated from one input image, it is not necessary to use a plurality of video cameras or cameras using wide-angle lenses, and shooting can be performed easily.

Further, since the image in the different direction is generated by using the real image, it is possible to display a more realistic and realistic image than the image in the three-dimensional virtual space by computer graphics, and Since the depth is expressed as a hierarchy of a two-dimensional plane, three-dimensional arithmetic processing is unnecessary and signal processing is easy.

Further, by using the signal stored as the input image signal, it is possible to display an image that enters the front image with the passage of time as a peripheral image before the front image is displayed. In this way, since an image that cannot be displayed when a real-time image signal is used can be displayed as a peripheral image, it is possible to perform more realistic and highly realistic image display with a wide angle of view. Further, by using an already existing huge image source as an input image signal, it is possible to enjoy these images with a high sense of realism and a wide angle of view.

In the above-described embodiment, a plurality of detection areas are provided at the side edges of the front image to obtain a motion vector for each detection area, or a motion discrimination image is generated from a projected image and compared with an integral image. By doing so, the intermediate position image information is generated by discriminating the movement of the target frame at the predetermined position, and the peripheral image signal is generated by using this intermediate image information. Any method can be used as long as it can be discriminated, and it goes without saying that the above method is an example and not a limitation.

[0142]

According to the present invention, by using the input image signals of a plurality of frames, the motion of the target frame in the input image signal at a predetermined position is detected, and the detected motion and the input image signals of the plurality of frames are detected. Intermediate image information is generated based on the information. Further, based on this intermediate image information, a peripheral image signal having an angle of view that does not exist in the frame of interest is generated. Therefore, by generating a peripheral image signal having an angle of view that does not exist in the frame of interest using the actually captured input image signal, this peripheral image can be displayed as a realistic and highly realistic image. Also, without using multiple video cameras or video cameras with special lenses,
A wide-angle image can be easily presented by generating peripheral images in different directions from two input images. Furthermore, it is also possible to present an image with a high sense of presence and a wide angle of view by utilizing an already existing huge image source.

Further, it is assumed that the layer classification is performed based on the motion of the predetermined position to classify the predetermined position into any of a plurality of layers, and the motion of each layer is calculated based on the classification result, the input image signal and the motion of the predetermined position. The amount and the image signal for each layer are used as intermediate image information. For this reason, it is possible to perform three-dimensional image presentation without performing three-dimensional processing by performing depth-direction hierarchization based on movement on a two-dimensional plane and pasting image signals of each hierarchy. .

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a display system.

FIG. 2 is a diagram showing an image projection state.

FIG. 3 is a diagram showing a configuration of an image processing apparatus.

FIG. 4 is a diagram showing a motion vector of each detection area.

FIG. 5 is a diagram for explaining a method of discriminating a motion vector.

FIG. 6 is a diagram for explaining a change in area size during zooming.

FIG. 7 is a diagram showing a method of determining an image reference position.

FIG. 8 is a diagram showing a configuration of a motion detection processing unit.

FIG. 9 is a diagram showing a configuration of a layer classification unit.

FIG. 10 is a diagram for explaining a relationship between a vector direction and a motion pattern.

FIG. 11 is a diagram for explaining a threshold setting method.

FIG. 12 is a diagram for explaining a generation operation of layer classification information.

FIG. 13 is a diagram illustrating a configuration of an information generation processing unit.

FIG. 14 is a diagram showing an intermediate image for each layer.

FIG. 15 is a diagram showing an intermediate image for each layer after interpolation processing.

FIG. 16 is a diagram showing a configuration of a peripheral image signal generation block.

FIG. 17 is a diagram for explaining image conversion processing.

FIG. 18 is a diagram showing another configuration of the image signal processing device.

FIG. 19 is a diagram for explaining a geometric conversion process.

FIG. 20 is a diagram for explaining the operation of the motion determination image generation unit.

FIG. 21 is a diagram for explaining an integrated image generating operation.

FIG. 22 is a diagram showing a configuration of an integral image generation unit.

FIG. 23 is a diagram for explaining the operation of the integral image generation unit.

FIG. 24 is a diagram for explaining the operation of the motion amount determination unit.

[Fig. 25] Fig. 25 is a diagram for describing an operation of generating a movement determination image of a backward layer.

FIG. 26 is a diagram showing a configuration of an information generation processing unit.

FIG. 27 is a diagram showing a projected surface image and a generated side surface image.

FIG. 28 is a diagram for explaining an image conversion process for displaying an image correctly on a screen.

FIG. 29 is a diagram showing a configuration when a computer is used.

[Explanation of symbols]

10L, 10C, 10R ... Screen, 12L, 12
C, 12R ... Projector, 15, 55 ... Image processing device, 20, 62 ... Motion detection block, 21 ...
・ Frame delay processing unit, 22 ... Motion detection processing unit, 2
5, 70 ... Intermediate image information generation block, 26, 71
... Layer classification unit, 27, 72 ... Information generation processing unit, 30 ... Storage unit, 31 ... Front image signal storage region, 32 ... Intermediate image information storage region, 40 ... Surroundings Image signal generation block, 41 ... Image generation control unit, 4
2 ... Layer image generation processing unit, 43 ... Projection conversion unit, 44 ... Image signal generation unit, 61 ... Geometric conversion unit, 63 ... Motion determination image generation processing unit, 64 ... Integral image generation processing unit, 65 ... Motion amount determination unit, 90 ...
・ Computer 221 ... Size conversion processing unit, 22
2 ... Error sum calculation unit, 223 ... Comparison processing unit, 22
4 ... data holding unit, 225 ... search control unit, 26
DESCRIPTION OF SYMBOLS 1 ... Pattern determination part, 262 ... Layer creation determination part, 263 ... Classification processing part, 264 ... Threshold setting part, 265 ... Classification correction part, 271, 721 ... Motion amount average Value calculation unit, 272, 722 ... Image reading unit, 273, 723 ... Intermediate image signal interpolation unit, 421
-1 to 421-5 ... Attached part, 422-1 to 422-5
.. 1 frame delay unit, 423-1 to 423-5 ... Image shift unit, 431 to 434 ... Conversion processing unit, 441
..Landscape pasting part, 442 ... Middle view pasting part, 4
43 ... near view pasting unit, 444 ... reduction pasting unit, 641-1 to 641-ns ... arithmetic processing unit, 642-1
~ 642-ns ... Pixel cutout unit, 901 ... CP
U, 902 ... ROM, 903 ... RAM, 904
... Hard disk drive, 905 ... I / O interface, 911 ... Input section, 912 ...
Recording medium drive, 913 ... Communication unit, 914 ...
Image input / output unit, 920 ... Bus

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 5/262 H04N 5/262 5L096 7/18 7/18 KV (72) Inventor Satoshi Kokubo Shinagawa, Tokyo 6-735 Kita-Shinagawa-ku, Sony Corporation (72) Inventor Daisuke Kikuchi 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo F-Term (reference) within Sony Corporation 5B057 AA20 CA12 CA16 CB12 CB16 CD05 CD06 CE08 CE10 DA07 DB02 DC34 5C023 AA11 BA11 5C054 AA01 AA04 FC13 FD02 5C061 AA23 AB12 5C082 AA27 BA12 BA20 BA42 BB25 CA52 CA54 CA55 CA84 DA61 DA87 MM10 5L096 CA02 DA02 FA34 FA69 GA08 GA19 GA51

Claims (40)

[Claims] 1. Using input image signals of a plurality of frames,
Motion detection means for detecting a movement of a target frame at a predetermined position in the input image signal, and intermediate image information generation for generating intermediate image information based on the movement detected by the movement detection means and the input image signals of the plurality of frames And a peripheral image signal generating means for generating a peripheral image signal having an angle of view that does not exist in the frame of interest based on the intermediate image information generated by the intermediate image information generating means. apparatus. 2. The intermediate image information generation means performs layer classification based on the movement of the predetermined position, and classifies the predetermined position into one of a plurality of layers, and the layer classification means 7. An information generation processing means for generating the amount of movement of each layer and the image signal of each layer as the intermediate image information based on the classification result, the input image signal, and the movement of the predetermined position. 1. The image processing device according to 1. 3. The layer classification means includes threshold setting means for determining a threshold used when performing the layer classification based on a motion amount at a predetermined position of a predetermined number of frames. The image processing device described. 4. The information generating means generates a signal for each layer by extracting a signal from the input image signal for each layer based on a motion amount for each layer. The image processing device described. 5. The information generation processing means, when there is a region where the image disappears when the images based on the image signals of the respective layers are combined in a predetermined order, the image signal of the peripheral region of the region where the image disappears. 3. The image processing apparatus according to claim 2, further comprising: an intermediate image interpolation processing unit that performs an interpolation process by using an image signal to create an image signal of a region where the image disappears. 6. The image processing apparatus according to claim 2, wherein the plurality of layers when the layer classification is performed by the layer classification means indicate depth. 7. The motion detecting means detects a correlation between an image at a predetermined position of the frame of interest and an image of a previous frame or a subsequent frame corresponding to the frame of interest, and finds the most correlation with the predetermined position of the frame of interest. The image processing apparatus according to claim 2, wherein the movement direction and the movement amount are detected from the position of the high image. 8. The layer classification means detects a motion pattern from the input image signals of the plurality of frames based on a motion direction of the predetermined position, and sets the plurality of layers according to a detection result. The image processing apparatus according to claim 7, which is characterized in that. 9. The image processing apparatus according to claim 8, wherein the layer classification unit performs classification into the plurality of layers based on the amount of movement of the predetermined position detected by the movement detection unit. 10. The geometric conversion means for generating an image signal of a projected image obtained by projecting an image based on an input image signal onto a predetermined surface, wherein the motion detection means uses the image signal of the projected image to generate the frame of interest. 7. The intermediate image information generating unit generates the intermediate image information based on the motion detected by the motion detecting unit and the image signal of the projected image. 1. The image processing device according to 1. 11. The motion detecting means extracts the image signal of a predetermined region of the projected image for each frame to generate an image signal of the motion determining image, and a position of the projected image in the frame. An average value image signal for a predetermined frame is sequentially generated using the image signals shifted for each frame, and an image signal of an integral image is generated by extracting the image signal for each frame from the average value image signal, and further the projection is performed. An integral image generating unit that generates a plurality of image signals of the integral image by changing the image shift amount, an image signal of the motion determining image generated by the motion determining image generating unit, and the image generating unit by the integral image generating unit. Detecting the correlation with the image signals of a plurality of integrated images and moving the predetermined position from the shift amount when the image signal of the integrated image having the highest correlation with the predetermined position is generated. 2. A motion amount discriminating means for discriminating an amount is provided.
The image processing device according to 0. 12. The image processing apparatus according to claim 11, wherein the layer classification unit performs layer classification according to the motion amount determined by the motion amount determination unit. 13. The peripheral image signal generating means reads the intermediate image information from the intermediate image information accumulating means and the peripheral image signal so that the peripheral image signal can be output in correspondence with the input image signal. The image processing apparatus according to claim 1, further comprising a control unit that controls generation of the image. 14. The peripheral image signal generation means includes image signal generation processing means for generating the peripheral image signal by pasting the image signals for each layer in a predetermined layer order. The image processing device described. 15. The peripheral image signal generation means, if the image signal obtained by pasting the image signals of each layer in a predetermined layer order has an image-free area, an image of a peripheral area of the image-free area. 15. A peripheral image interpolation processing means for performing an interpolation process using a signal to create an image signal of a region where the image signal is absent.
The image processing device described. 16. The peripheral image signal generation means moves the pasted image signal for each layer based on the movement amount for each layer, and then pastes a new image signal for each layer to the image. The image processing apparatus according to claim 14, further comprising layer image generation processing means for supplying the signal generation processing means. 17. The peripheral image signal generation means performs projective transformation of the image signal generated by the layer image generation processing means and supplies the image signal to the image signal generation processing means, or generates by the image signal generation processing means. The image processing apparatus according to claim 16, further comprising a projective transformation unit that performs projective transformation of the generated image signal and outputs the result. 18. The projective transformation means performs projective transformation in accordance with a positional relationship between an image display surface of an image based on the input image signal and an image display surface of an image based on the peripheral image signal to produce the peripheral image. The image processing device according to claim 17, wherein the image processing device generates a signal. 19. The image processing apparatus according to claim 1, wherein the input image signal is a stored signal. 20. A plurality of frames of input image signals are used to detect a movement of a target frame at a predetermined position in the input image signal, and an intermediate image based on the detected movements and the plurality of frames of input image signals. An image processing method, wherein information is generated, and a peripheral image signal having an angle of view that does not exist in the frame of interest is generated based on the generated intermediate image information. 21. Layer classification is performed based on the movement of the predetermined position to classify the predetermined position into one of a plurality of layers, and based on the classification result, the input image signal, and the movement of the predetermined position, 21. The image processing method according to claim 20, wherein a motion amount for each layer and an image signal for each layer are generated as the intermediate image information. 22. The image processing method according to claim 21, wherein a threshold value used when performing the layer classification is determined based on a motion amount of the predetermined position of a predetermined number of frames. 23. An image signal for each layer is generated by extracting a signal for each layer from the input image signal based on a motion amount for each layer.
The described image processing method. 24. When there is a region where the image disappears when the images based on the image signals of the respective layers are combined in a predetermined order, interpolation processing is performed using the image signal of the peripheral region of the region where the image disappears. 22. The image processing method according to claim 21, wherein an image signal of a region where the image disappears is created. 25. The image processing method according to claim 21, wherein the plurality of layers indicate depth. 26. A correlation between an image at a predetermined position of the frame of interest and an image of a previous frame or a subsequent frame with respect to the frame of interest is detected, and the image is moved from the position of the image having the highest correlation with the predetermined position of the frame of interest. 22. The image processing method according to claim 21, wherein the direction and the amount of movement are detected. 27. The motion pattern is detected from the input image signals of the plurality of frames based on the movement direction of the predetermined position, and the plurality of layers are set according to the detection result. The described image processing method. 28. The image processing method according to claim 27, wherein the classification into the plurality of layers is performed based on the amount of movement of the predetermined position detected by the movement detecting means. 29. An image signal of a projected image is generated by projecting an image based on an input image signal onto a predetermined surface, the movement of the target frame at a predetermined position is detected using the image signal of the projected image, and the detected image is detected. 21. The image processing method according to claim 20, wherein the intermediate image information is generated based on a motion and an image signal of the projected image. 30. An image signal of a motion discriminating image is generated by extracting an image signal of a predetermined area of the projection image for each frame, and a predetermined frame is generated using an image signal in which the position of the projection image is shifted for each frame. Of the average value image signal is sequentially generated, and the image signal of the integral image is generated by extracting the image signal from the average value image signal for each frame, and the image of the integral image is changed by changing the shift amount of the projection image. A plurality of signals are generated, and the correlation between the image signal of the motion determination image and the image signal of the plurality of integral images generated by the integral image generating means is detected, and the correlation is highest with the predetermined position. 30. The image processing method according to claim 29, wherein the amount of movement of the predetermined position is determined from the shift amount when the image signal of the integrated image is generated. 31. The image processing method according to claim 30, wherein layer classification is performed according to the determined motion amount. 32. The reading of the intermediate image information and the generation of the image signal of the peripheral image are controlled so that the image signal of the peripheral image can be output in correspondence with the input image signal. 20. The image processing method described in 20. 33. The image processing method according to claim 21, wherein the image signal of each layer is pasted in a predetermined layer order to generate the image signal of the peripheral image. 34. When the image signal obtained by pasting the image signals for each layer in a predetermined layer order has an image-free area, interpolation processing is performed using an image signal in a peripheral area of the image-free area. 34. The image processing method according to claim 33, wherein an image signal of a region without the image signal is created. 35. After moving the pasted image signal for each layer based on the movement amount for each layer, a new image signal is pasted for each layer, and the pasting is performed for each layer. 34. The image processing method according to claim 33, wherein the separated image signals are pasted in a predetermined layer order to generate an image signal of the peripheral image. 36. Projective conversion of image signals pasted for each layer, or projective conversion of image signals pasted for each layer in a predetermined layer order. The image processing apparatus according to claim 35, wherein: 37. The peripheral image signal is generated by performing projective transformation according to the positional relationship between the image display surface of the image based on the input image signal and the image display surface of the peripheral image. Item 36. The image processing method described in Item 36. 38. The image processing method according to claim 20, wherein the input image signal uses a stored signal. 39. A computer for processing an image signal, using a plurality of frames of input image signals, detecting a motion of a predetermined position of a frame of interest in the input image signals; The intermediate image information generation procedure for generating intermediate image information based on the motion and the input image signals of the plurality of frames, and the image not existing in the frame of interest based on the intermediate image information generated in the intermediate image information generation procedure. An image processing program for executing a peripheral image signal generation procedure for generating a peripheral image signal of a corner. 40. A computer for processing an image signal, using a plurality of frames of input image signals, a motion detection procedure for detecting a motion at a predetermined position of a frame of interest in the input image signal; The intermediate image information generation procedure for generating intermediate image information based on the motion and the input image signals of the plurality of frames, and the image not existing in the frame of interest based on the intermediate image information generated in the intermediate image information generation procedure. An information recording medium, characterized in that a program for executing a peripheral image signal generating procedure for generating a peripheral image signal of a corner is recorded.