JP2002271783A - Image processing unit, image signal generating method, image processing program and information recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing unit that uses an image signal of a reference moving image so as to generate an image signal of a surrounding moving image corresponding to the reference moving image at the same that as that of the image signal of the reference moving image. SOLUTION: A block 20 calculates a motion by each motion detection area set to an image and area of the reference moving image by using a reference image signal SDc that satisfies binding conditions with respect to a motion of an image in radial directions around a prescribed position and a signal SDcd delayed by one frame. A block 22 calculates a moving amount LV of each layer and layer identification information LS that denotes to which layer a motion detection area belongs. A block 24 moves a signal of each layer from a delay section 26 by the moving amount LV by each layer, and extracts the reference moving image by the moving amount LV from a position corresponding to the motion detection area and adds the extracted image to the signal. Pasting the sum signals from a remote scene layer to a near scene layer generates a moving image signal. Applying conversion processing in response to the direction of a projected face of a side face moving image is applied to the signal to generate side face moving image signals SDL, SDR.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus, an image signal generating method, an image processing program, and an information recording medium. More specifically, the input image signal of a plurality of frames is used to detect the movement of, for example, an image end region in an image based on the input image signal, and a frame of interest is detected based on the detected movement and the input image signal of the plurality of frames. And generates an image signal of a peripheral moving image having a different angle of view from the image of the frame of interest.

[0002]

2. Description of the Related Art In recent years, a multi-screen display system, a curved display, a wide-angle display, a head-mounted display, and the like have been put to practical use in order to display images with enhanced realism.

Here, in order to display an image with enhanced realism using the above-described display, a three-dimensional virtual space is constructed by, for example, computer graphics. In addition, a large number of video cameras are used, or a wide-angle lens is attached to the video camera to photograph a wide area, and this photographed image is converted into a flat or curved surface such as a multi-surface display or a head-mounted display and displayed. I have.

[0004]

In the case of displaying a moving image with enhanced realism using a multi-screen display or a wide-angle display as described above, it is necessary to construct a three-dimensional virtual space using computer graphics. Requires a high-speed computer to perform arithmetic processing, which requires a great deal of cost and time, and results in an image with less realism and realism than a real image.

[0005] In addition, in the case of using real images, in order to present a wide range of space, it is necessary to take a picture of the entire presented range without any gap, and many video cameras and video cameras having special lenses are used. A large-scale image capturing device is required, and a large cost is required.

Furthermore, when an image source shot by a general video camera already exists, a wide range of images cannot be presented unless the same scene is retaken by a plurality of video cameras.

Therefore, the present invention provides an image processing apparatus and an image signal generation apparatus that can generate a peripheral moving image at the same time as the moving image and corresponding to the moving image by using an image signal of the moving image as a reference. A method and an information transmission recording medium are provided.

[0008]

SUMMARY OF THE INVENTION An image processing apparatus according to the present invention comprises a motion detecting means for detecting a motion of a predetermined area in an image formed on a specific plane by using input image signals of a plurality of frames; A layering processing unit configured to classify a predetermined area based on the motion detected by the detection unit to generate information indicating a classification result; and a target unit in the input image signal based on the classification information and the input image signals of a plurality of frames. And peripheral image signal generating means for generating an image signal of a peripheral image having the same time as that of the frame and having a different angle of view from the image of the frame of interest.

In addition, the image processing apparatus uses an image signal of a reference moving image which generates a motion satisfying at least one constraint condition, and utilizes the constraint condition to provide an image edge in the screen for each screen of the reference moving image. A motion detection unit that detects a motion of the region, a layering processing unit that determines a layer of the predetermined region based on the motion detected by the motion detection unit and generates layer information, Using the layer information and the image signal of the image end area, performing signal processing according to the screen order to generate an image signal of the peripheral moving image at the same time as the reference moving image and corresponding to the reference moving image Peripheral image generating means.

An image signal generating method according to the present invention detects movement of a predetermined area in an image formed on a specific plane using input image signals of a plurality of frames,
Based on the detected motion, a predetermined area is classified and information indicating a classification result is generated. Based on the classification information and the input image signals of a plurality of frames, the time is the same as that of a target frame in the input image signal. At the same time, it generates an image signal of a peripheral image having a different angle of view from the image of the frame of interest.

The image signal generating method includes at least one
The motion of the image edge region is detected for each screen of the reference moving image by using the constraint condition using the reference motion image that generates the motion satisfying the two constraint conditions, and the image edge is determined based on the detected motion. The image signal of the peripheral area is generated by using the image signal of the partial area and performing signal processing according to the screen order, and at the same time as the reference moving image and corresponding to the reference moving image.

[0012] An image processing program according to the present invention provides a computer with a process for detecting a motion of a predetermined region in an image formed on a specific plane using a plurality of frames of input image signals, and a process for detecting the motion based on the detected motion. Processing for classifying a predetermined area to generate information indicating a classification result; and, based on the classification information and the input image signals of a plurality of frames, at the same time as the frame of interest in the input image signal and the image of the frame of interest. And a process of generating image signals of peripheral images having different angles of view.

Further, the image processing program causes the computer to use a reference moving image which generates a movement satisfying at least one constraint condition, and to use the constraint condition to change the movement of the image end region for each screen of the reference moving image. First to detect
Based on the detected motion, the image signal of the image edge region is used and the signal processing is performed according to the screen order, and the same time as the reference moving image and the peripheral moving image corresponding to the reference moving image And a second process of generating the image signal of the above.

An information recording medium according to the present invention provides a computer with a process for detecting a motion of a predetermined area in an image formed on a specific plane using input image signals of a plurality of frames, and a process for detecting the motion based on the detected motion. Processing for classifying a predetermined area to generate information indicating a classification result; and, based on the classification information and the input image signals of a plurality of frames, at the same time as the frame of interest in the input image signal and the image of the frame of interest. And a program for executing a process of generating an image signal of a peripheral image having a different angle of view.

[0015] Further, the information recording medium is a computer.
A first process of detecting a motion of an image end region for each screen of the reference moving image by using the constraint condition using a reference moving image which generates a motion satisfying at least one constraint condition; Based on the image signal, performs signal processing according to the screen order while using the image signal of the image end area, and generates an image signal of the peripheral moving image at the same time as the reference moving image and corresponding to the reference moving image. And a computer-readable recording program for executing the processing including the above processing.

In the present invention, for example, a plurality of motions are added to an image end region of the reference moving image with respect to a reference moving image in which the motion of the image satisfies a constraint condition that is a radial direction around a predetermined position. A detection area is set, an error value from the image of the previous screen is calculated while the image of the motion detection area is zoomed around a predetermined position, and a position where the calculated error value is minimum is detected. Thus, the motion of the motion detection area is determined.

Based on the detection result of the motion, the image end region is divided into layers, the amount of movement is set for each layer, and the image of the layered image end region is set to the amount of movement of the corresponding layer. Accordingly, an image is extracted from the reference moving image, and the images are pasted in order from a layer having a small moving amount to a layer having a large moving amount. The image of each layer that has already been pasted is moved according to the amount of movement set for each layer, and the extracted image is added to the moved image, and then the image is pasted in layers. Will be

Here, when a gap is generated in the image by the pasting process in units of layers, the image located behind the gap is extended in the moving direction of the layer to generate an image signal of the image of the gap. Is done. Further, a signal conversion process is performed on the image signal of the image pasted on a layer basis according to the direction of the projection plane of the peripheral moving image.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows the overall configuration of a display system using an image processing device according to the present invention. In this display system, for example, three screens are arranged on the front and both sides of the user, and the projectors 12 corresponding to the screens 10L, 10C, and 10R are provided.
By projecting images from L, 12C, and 12R, a wide range of images is presented. This projector 12L,
12C and 12R are image processing devices 15A (15B and 15C)
Is connected to

The image processing device 15A (15B, 15B)
In C), a motion of a predetermined area in an image formed on a specific plane is detected using input image signals of a plurality of frames, and the predetermined area is classified based on the detected motion, and a classification result is obtained. Generate the information shown. Further, based on the classification information and the input image signals of a plurality of frames, an image signal of the peripheral image having the same time as the target frame in the input image signal and having a different angle of view from the image of the target frame is generated.

The image processing device 15A receives an image signal captured while moving, for example, an image signal SDC of a moving image captured forward by a vehicle-mounted camera. Image processing device 15
A delays the input image signal SDC by the time required to generate peripheral image signals SDL and SDR of a side moving image, which will be described later, and then supplies the image signal SDC 'to the projector 12C. The projector 12C has an image signal SD
A front moving image UC based on C ′ is projected on a screen 10C located in the front as shown in FIG. Further, based on the image signal SDC of the front moving image UC, which is a reference moving image, the image processing device 15A performs a side moving image corresponding to the front moving image UC, that is, a visual field range (view angle range) of the vehicle-mounted camera.
, Generates peripheral image signals SDL and SDR of a moving image of a peripheral region adjacent to the image, for example, left and right side moving images. The peripheral image signal S of the generated left side moving image UL is supplied to the projector 12L.
DL is input, and the left side moving image UL is projected on the screen 10L located on the left side based on the peripheral image signal SDL. The generated right side moving image UR is displayed on the projector 12L.
Is input to the peripheral image signal SDR.
The right side moving image UR is projected on the screen 10R located on the right side based on DR.

FIG. 3 shows the configuration of an image processing apparatus 15A according to the first embodiment. The image signal SDC is transmitted to the delay unit 16 and the motion detection block 20 of the image processing device 15A.
And the peripheral image signal generation block 24. The motion detection block 20 sets a plurality of motion detection areas AR as shown in FIG. 4 at the image end area of the moving image, for example, on both side ends of the front moving image UC, and sets the image in each motion detection area AR. The motion amount MV is calculated and supplied to the layering processing block 22. The motion amount MV is calculated based on the image signal SDC of the motion detection area and the image signal SDd one frame before.

The layering processing block 22 determines whether each motion detection region is a region of a distant view image of a distant subject (a distant view layer) or a region of a near view image of a close subject (a near view layer). Alternatively, it is determined based on the motion amount MV whether or not the area is a middle view image area (middle view layer) located between the distant view image and the close view image. Further, based on the determination result, layer identification information LS indicating whether each motion detection area belongs to a distant view, a middle view, or a near view layer is generated and supplied to the peripheral image signal generation block 24. Further, a layer movement amount LV for each layer is calculated for each frame based on statistical information calculated using the movement amount of the motion detection area included in the layer, and the layer movement amount LV is supplied to the peripheral image signal generation block 24. I do.

The peripheral image signal generation block 24 extracts a front moving image from each position set in the image end area corresponding to the motion detection area by the layer moving amount LV of the layer corresponding to the motion detection area. Further, the peripheral image signal generation block 24 performs image synthesis by superimposing the extracted images in order from a distant view layer with a small moving amount to a near view layer with a large moving amount, and then performs a projection plane of the side moving image. , And generates and outputs peripheral image signals SDL and SDR of the side moving image at the same time as the front moving image.

The image signal S of the side moving image for each layer
By supplying EL and SER to the delay unit 26, the signals are delayed by one frame and then supplied to the peripheral image signal generation block 24. Here, the peripheral image signal generation block 24 moves the image based on the image signals SELd and SERd of the side moving image delayed by one frame for each layer according to the moving amount set for each layer. The peripheral image signal generation block 24 synthesizes images by adding an image extracted from the front moving image to the moved image and superimposing the images on a layer-by-layer basis to generate peripheral image signals SDL and SDR of the side moving image. Generate.

Further, when an image in which a distant view, a middle view, and a near view are superimposed in order to create an image, if there is a gap portion without an image due to a difference in the amount of layer movement, the peripheral image signal generation block 24 By supplementing the gap by interpolation processing, peripheral image signals SDL and SDR of the side moving image without the gap are generated.

Next, the configuration and operation of each block will be described in detail with reference to the drawings. In the following description, only the right side moving image will be described, and description of generation of the left side moving image will be omitted.

The motion detection block 20 determines the motion amount MV of the image for each motion detection area as described above. Here, as described above, if the image appears to come out from one point (hereinafter referred to as “image reference position”) CP by photographing the front with the onboard camera, the motion detection block 20 The amount of motion is determined by adding a constraint that the direction of motion is a radial direction from one point.
In this case, the front moving image at time T shown in FIG. 5A is, for example, at time T ′ after one frame time has elapsed, as shown in FIG. 5B, the front moving image at time T centered on the image reference position CP. This is almost the same as the zoomed image. Therefore, it is considered that the distance from the image generation position to the center of the motion detection area is proportional to the enlargement ratio in the zoom operation. Therefore, the motion detection block 20 performs, as shown in FIG.
A motion detection area is reduced to (1 / Z) by setting an enlargement factor Z centered on the image reference position CP, and the error sum of the reduced area and an image signal of a corresponding position in the image one frame before is set. Is calculated, and the amount of motion for each motion area is calculated by detecting an enlargement factor Z at which the sum of the errors becomes a minimum value.

Here, when the reduction processing is performed, there are pixels in which the coordinate values of the pixels in the area do not become integer values. on the other hand,
In the image one frame before the reduction process has not been performed, the coordinate values of the pixels in the region are integer values. Therefore, linear interpolation is performed on the reduced image to calculate a signal level at a position where the coordinate value is an integer value. For example, as shown in FIG. 6, when the motion detection area of Ka × Ka pixels is multiplied by (1 / Z) times and the area size becomes the size of Kb × Kb pixels, linear interpolation is performed to perform Ka × Ka pixel Is calculated by using the signal level of (1), where the number of pixels is “Kb × Kb”. By calculating the error sum using the calculated signal level, the motion amount can be determined with high accuracy.

FIG. 7 shows the structure of the motion detection block 20. The image signal SDc is supplied to the size changing unit 201, and the image signal SDcd one frame before is supplied to the error sum calculation unit 202. The size changing unit 201 sets a plurality of motion detection areas by dividing the right end portion of the front moving image in units of, for example, 16 × 16 pixels. In the following description, the operation of each unit for one motion detection area will be described.

The size changing unit 201 generates an image signal SDz of the motion detection area multiplied by (1 / Z) around the image reference position CP by using the enlargement ratio Z supplied from the enlargement ratio setting unit 205. Then, it is supplied to the error sum calculation unit 202. Further, the image signal SDz is converted to an error sum calculation unit 202.
At the center of the image reference position CP (1
/ Z), the coordinate value Pz of the center of the motion detection area moved by the multiplication is also supplied to the error sum calculation unit 202. In addition,
The image signal SDz is a signal converted to a signal level for each pixel position by the interpolation processing as described with reference to FIG.

The error sum calculator 202 calculates the image of the area corresponding to the motion detection area multiplied by (1 / Z) with respect to the coordinate value Pz indicated by the size changer 201 from the image signal SDd one frame before. Select a signal. The image signal SD of the motion detection area multiplied by (1 / Z) times the selected signal.
An error sum ET is calculated by calculating and adding an error for each pixel using z, and the result is notified to the comparing unit 203.

The comparison unit 203 compares the error sum minimum value ETL with the error sum ET calculated by the error sum calculation unit 202. Here, when the error sum minimum value ETL is not set, the error sum ET calculated first is replaced with the error sum minimum value ET.
Set as L. Further, the error sum minimum value ETL may be set in advance to a value larger than the calculated error sum. When the error sum ET is smaller than the error sum minimum value ETL as a result of the comparison between the error sum minimum value ETL and the error sum ET, the error sum ET is set as a new error sum minimum value ETL and the error sum minimum value is set. An update signal CH indicating that the ETL has been updated is notified to the data storage unit 204. Further, the comparison unit 203 notifies the enlargement ratio setting unit 205 of a signal CE indicating that the comparison between the error sum ET and the minimum error value ETL has been completed.

The data storage unit 204 has an enlargement ratio setting unit 2
The enlargement ratio Z is notified from 05, and the error sum minimum value ET
When the update signal CH indicates that L has been updated, the notified enlargement factor Z is stored. When the enlargement ratio is already stored, the stored enlargement ratio is updated with the notified enlargement ratio Z. The enlargement ratio setting unit 20
When the signal CHZ indicating the completion of the change processing of the enlargement ratio is supplied from 5, the stored enlargement ratio Z is supplied to the layering processing block 22 as the motion amount MV.

The enlargement ratio setting unit 205 sets a lower limit value and an upper limit value of the enlargement ratio in advance (for example, lower limit = 1.0, upper limit =
1.1) First, the lower limit value is notified to the size changing unit 201 and the data storage unit 204 as the enlargement factor Z. Thereafter, each time a signal CE indicating that the comparison between the error sum ET and the error sum minimum value ETL is completed is supplied from the comparing unit 203, the enlargement factor Z is sequentially increased by a predetermined amount (for example, a step of 0.005). ), And notifies the size change unit 201 and the data storage unit 204. Thereafter, when the enlargement ratio Z reaches the upper limit, the enlargement ratio setting unit 205 notifies the data storage unit 204 of a signal CHZ indicating the completion of the enlargement ratio change processing.

As described above, the image of the motion detection area is sequentially reduced by sequentially increasing the enlargement rate by the enlargement rate setting unit 205, and the image of the reduced area is compared with the image of one frame before. By storing the enlargement ratio when the error is minimized in the data storage unit 204, when the change processing of the enlargement ratio by the size change unit 201 is completed, the motion amount MV of the motion detection area is stored in the data storage unit 204. To the layering processing block 22.

The same processing is performed for all the set motion detection areas using the enlargement ratio Z set by the enlargement ratio setting unit 205. When the enlargement ratio setting process by the enlargement ratio setting unit 205 is completed, the motion amount MV of each motion detection area AR is determined as indicated by the length of the arrow in FIG. The amount MV can be notified from the data storage unit 204 to the layering processing block 22.

The layering processing block 22 determines which of the plurality of layers each motion detection area belongs to by setting a threshold value and comparing it with the amount of motion, and classifies the image end area into a plurality of layers. I do. Here, a distant view image has a small amount of motion, and a near view image has a large amount of motion. For this reason, the layering processing block 22 can determine whether each motion detection area belongs to a layer, for example, a distant view, a middle view, or a near view by setting a threshold value and comparing the motion amount with the motion amount.

FIG. 9 shows the configuration of the layering processing block 22. The motion amount MV supplied from the motion detection block 20 is supplied to the statistical information calculation unit 221 and the layer movement amount calculation unit 225. The statistical information calculation unit 221 obtains statistical information MS based on the supplied motion amount for each region, for example, an average value, a maximum value, and a minimum value of the motion amount, or a frequency distribution obtained by classifying the motion amount. This is supplied to the layer division threshold value setting unit 222.

The layer division threshold value setting section 222 determines a threshold value Th based on the statistical information MS obtained by the statistical information calculating section 221.
Is supplied to the layer determination unit 223 and the delay unit 224. Further, the threshold value Thd one frame before, which is the threshold value delayed by one frame by the delay unit 224, is supplied to the layer division threshold value setting unit 222.
Threshold value Th one frame before based on the statistical information MS obtained in
A new threshold value Th is set by correcting d.

Here, when the maximum value and the minimum value of the amount of motion are indicated as the statistical information MS, when n thresholds are set, the range of the maximum value and the minimum value, which is the appearance range of the amount of motion, is set to ( n-1) Divide equally and set a threshold. For example, as shown in FIG. 10A, the range from the maximum value MVmax to the minimum value MVmin is divided into three equal parts, and the boundary values are set to the threshold values Th-1 and Th-2. Alternatively, the threshold values Th-1 and Th-2 are set based on Expressions (1) and (2) using the average value MVavg.

[0042]

(Equation 1)

When the threshold value of the previous frame is supplied, a new threshold value is calculated by correcting the threshold value based on the statistical information obtained by the statistical information calculation unit 221. For example, the threshold of one frame before is “Thd−1”,
When “Thd−2” is set, thresholds Th−1 and Th−2 are calculated using Expressions (3) and (4). “Α” and “β” are coefficients.

[0044]

(Equation 2)

When the histogram as shown in FIG. 10B is displayed as statistical information, the minimum amount of movement is determined, and the threshold values Th-1 and Th-2 are determined based on the determined minimum amount of movement. Can also be set. Further, the threshold value Thd- already set based on the histogram
1, Thd-2 may be corrected based on the statistical information obtained by the statistical information calculation unit 221 in the same manner as described above to obtain new threshold values Th-1 and Th-2. As described above, the threshold value set by the layer division threshold value setting unit 222 is
The information is supplied to the layer determination unit 223.

The layer determination section 223 calculates the amount of motion MV of each motion detection area and the threshold T set by the layer division threshold setting section.
By comparing with h, it is determined which layer the image end region belongs to. Further, it generates layer identification information LS indicating the result and supplies it to the layer movement amount calculation unit 225 and the peripheral image signal generation block 24. The layer movement amount calculation unit 225 calculates an average value for each layer based on the movement amount of the motion detection area included in the layer, and calculates the layer movement amount LV.
To the peripheral image signal generation block 24.

When the motion detection area is, for example, a part of the blue sky, the motion detection block 20 calculates the sum of the errors by reducing the motion detection area, and the error sum is substantially independent of the enlargement ratio. There is a possibility that the amount of movement becomes constant and cannot be accurately determined. Therefore, when the motion amount MV cannot be determined, the motion detection block 20 supplies information indicating the color of the image in the motion detection area to the layering processing block 22. Further, the layering processing block 22 performs layer division using information indicating a color. For example, in a motion detection area where the amount of motion cannot be calculated, when the color of the image in this area is the color of the blue sky, the area is assigned to a distant view layer to generate the layer identification information LS.

Next, the peripheral image signal generation block 24
Based on the layer identification information LS and the layer moving amount LV, an image is extracted from the layered image end region by the corresponding layer moving amount. Further, the peripheral image signal generation block 24 performs image synthesis by moving the already generated one-frame-side side moving image for each layer, and then superimposing an image extracted by the layer moving amount for each layer. To generate a side moving image. Further, the directions are different between the projection plane of the front moving image and the projection plane of the right side moving image. For this reason, the peripheral image signal generation block 24 generates and outputs an image signal of the side moving image to be projected on the screen by converting the generated side moving image into an image corresponding to the projection plane.

FIG. 11 shows the peripheral image signal generation block 24.
Is shown. This peripheral image signal generation block 2
Reference numeral 4 denotes a configuration in a case where the image end region of the front moving image is divided into three layers of a distant view, a middle view, and a near view by the layering processing block 22.

The layer identification information LS and the layer moving amount LV generated in the layering processing block 22 are supplied to the signal extracting unit 241. The signal extraction unit 241 extracts an image signal by an amount corresponding to a layer movement amount for each layer from an image end area of the front moving image based on the image signal SDC of the front moving image. For example, as shown in FIG. 12A, for each position of the motion detection area, the image UE is extracted from the right end of the front moving image by a layer moving amount corresponding to the layer of the motion detection area. Here, when the motion detection area is determined to be a distant view layer, signals corresponding to the number of pixels corresponding to the layer movement amount of the distant view layer are extracted from the right end of the front moving image at a position corresponding to the motion detection area. . Further, when it is determined that the image is a middle view layer or a near view layer, a signal of the number of pixels corresponding to the layer movement amount of the middle view layer or the close view layer is extracted from the right end of the front moving image. If the image composition is performed by superimposing the extracted images of the respective layers, a right side moving image UR can be generated as shown in FIG. 12B.

As described above, the image signal SR-f of the distant view layer extracted by the signal extracting unit 241 is supplied to the adding unit 242. Further, the extracted image signal S of the middle ground layer
Rm is supplied to the addition unit 243, and the extracted near-view layer image signal SR-n is supplied to the addition unit 244.

The image shift unit 245 is notified of the amount of layer movement from the layering processing block 22, and the image signal SERd− for each layer one frame before from the delay unit 26.
f, SERd-m and SERd-n are supplied. The image shift unit 245 generates an image signal SRd-f obtained by moving the image of the distant view layer based on the image signal SERd-f by the distance of the distant view layer, and supplies the image signal SRd-f to the adder 242. Similarly,
The image shift unit 245 generates an image signal SRd-m obtained by moving the image of the middle ground layer based on the image signal SERd-m by the moving amount of the middle ground layer, and supplies the generated image signal SRd-m to the addition unit 243. Image signal S obtained by moving the image of the foreground layer based on SERd-n by the amount of the layer movement of the foreground layer
Rd-n is generated and supplied to the adder 244.

The addition section 242 generates an addition signal SRA-f of the image signal SR-f and the image signal SRd-f, and supplies the addition signal SRA-f to the interpolation section 246. Similarly, the addition unit 243 generates an addition signal SRA-m of the image signal SR-m and the image signal SRd-m, and
46, and the adder 244 outputs the image signal SR-
An addition signal SRA-n of n and the image signal SRd-n is generated and supplied to the interpolation unit 246.

Here, for example, the image signals SERd-f, SERd
When the right side moving image UR generated by superimposing -m and SERd-n is as shown in FIG. 12C, the image is shifted by the layer shift amounts MR1 and MR2 for each layer by the image shift unit 245. Therefore, the right side moving image UR generated by superimposing the image signals SRd-f, SRd-m, and SRd-n is as shown in FIG. 12D.
Further, since the image signals SR and SRd are added by the adders 242, 243, 244, the addition signal SRA-
f, SRA-m, and SRA-n are superimposed to synthesize an image UE, and the right-side moving image UR is as shown in FIG. 12E. The right-side moving image after one frame period elapses with respect to FIG. Can be generated. At this time, the image shift unit 24
5, the image is moved by the layer moving amount for each layer, and the image is extracted from the right end of the front moving image by the layer moving amount for each layer. Images are superimposed on the front moving image by an amount corresponding to each other, and each layer becomes a continuous image even if the image is moved.

The interpolation unit 246 includes addition units 242, 243,
Based on the addition signal supplied from the H.244, when the images are synthesized by superimposing the images of the layers having the smaller moving amounts in order, it is determined whether or not a gap portion having no image occurs due to the difference in the moving amount of the layers. When a gap is generated, an image of the gap is generated by interpolation. For example, when a stationary object such as a building is photographed by an in-vehicle camera, the moving amount of the distant view layer is small and the moving amount of the near view layer is large. Therefore, the image shift unit 2
When the distant view image, the middle view image, and the near view image of the right side moving image are moved by each layer moving amount by 45, for example, FIG.
In the right side moving image shown in FIG.
When the foreground layer image UR2 and the foreground layer image UR2 are adjacent to each other and the amount of movement of the foreground layer is different from the amount of movement of the middle view layer, when the image is moved for each layer, a gap region UN without an image is generated as shown in FIG. 13B. In some cases. The gap area can be determined by using a method such as setting a flag indicating the fact at the position where the images are overlapped.

For this reason, the interpolating unit 246 performs interpolation using the peripheral pixels of the gap area, creates an image of the gap area, and superimposes the image. For example, as shown in FIG. 13C, the gap area is interpolated by extending the image located on the rear side of the gap area to the gap area side, which is the moving direction of the layer. As described above, by performing the interpolation processing, it is possible to generate an image signal of the right side moving image without a gap area in the right side moving image. In addition, by performing interpolation by extending the image located on the rear side in the direction of movement of the layer, it is possible to prevent an image with a large amount of movement from becoming long in a trailing manner during movement. Further, the peripheral image may be shifted to the gap region side. Further, the interpolation unit 246 performs interpolation processing by creating a plurality of pixels and filling the gap area by weighted average of distances using peripheral image signals, that is, weighted average of pixels in adjacent areas that are in contact with the gap area. It is good to do.

The addition signal SRA-f after the interpolation processing by the interpolation section 246 is converted into an image signal SER-f by the middle-view synthesis section 24.
7 and to the delay unit 26. Similarly, the added signal SRA-m of the middle-ground layer is the image signal SER-m
Is supplied to the middle view synthesis section 247 and the delay section 26, and the sum signal SRA-n of the foreground layer is converted to the image signal SER-n
Is supplied to the foreground composition unit 248 and the delay unit 26.

Using the image signal SER-f and the image signal SER-m, the middle view synthesis section 247 generates an image signal SER-fm in which the middle view layer image is superimposed on the far view layer image to produce the near view synthesis. To the unit 248.

The foreground composition unit 248 performs image composition by superimposing a foreground image on an image obtained by superimposing a middle image on a distant image using the image signal SER-fm and the image signal SER-n. It is supplied to the image conversion unit 249 as an image signal SFR of the right side moving image.

The image conversion unit 249 converts the image signal SER of the right side moving image supplied from the foreground composition unit 248 into a peripheral image signal SDR for image projection. Here, the image based on the image signal SFR supplied from the foreground composition unit 248 moves the image one frame before according to the layer moving amount as described above, and extracts only the layer moving amount from the front moving image UC. The extracted image is superimposed on the moved image. Therefore, the image UR 'is indicated by a solid line in FIG. 14A, and is not an image that springs out from the image reference position CP as indicated by a broken line.
Further, as shown in FIG. 14B, a screen 10R for projecting a right side moving image and a screen 1R for projecting a front moving image.
The 0C projection planes are not in the same orientation. Therefore, the image conversion unit 249 generates the image UR 'based on the image signal SFR from the image reference position CP.
Is performed.

That is, the image conversion section 249 enlarges the image based on the image signal SFR in the vertical direction in proportion to the distance from the image reference position CPR corresponding to the right side moving image projected on the screen 10R as image conversion processing. I do. In the horizontal direction, the movement of the image is the image reference position CPR.
It expands in proportion to the distance from.

Here, as shown in FIG. 14C, the length from the front end of the right side moving image to the image reference position CPR is "L", the horizontal length of the right side moving image is "A", and When a proportionality constant γ based on the angle θs between the projection plane of the moving image and the projection plane of the right-side moving image is set, as shown in FIG. 14D, the position (x,
y) and a position (X,
The relationship of Y) can be approximately expressed by Expressions (5) and (6). The lengths “L” and “A” and the proportionality constant γ are set in advance before generating the side moving image.

[0063]

(Equation 3)

For this reason, the signal at the position (x, y) on the image based on the image signal SFR calculated based on the equations (5) and (6) is extracted as the signal at the position (X, Y) after the image conversion. For example, it is possible to easily generate an image-converted peripheral image signal SDR. Further, the peripheral image signal SDL of the left side moving image can be similarly generated. Therefore, the projector 12R projects the right-side moving image on the screen 10R by using the peripheral image signal SDR, so as to be able to project a more realistic right-side moving image as shown in FIG. 14E. Further, the projector 12L outputs the peripheral image signal S
By projecting the left side moving image on the screen 10L using the DL, it is possible to project a more realistic left side moving image.
Further, the projector 12C outputs the peripheral image signals SDL, S
By projecting the front moving image on the screen 10C based on the image signal SDC 'which is an image signal obtained by delaying the image signal SDC by the time required for generating the DR, the real-time image signal can be obtained only by using the image signal SDC of the front moving image. , A front moving image and a good left and right side moving image can be projected. In the above description, the side moving image is generated. However, the moving image on the upper surface side or the bottom surface side can be displayed in real time in the same manner. Note that the above-described first embodiment and a second
In addition, the layer division, the threshold setting, the number, the size, the position, and the like of the motion detection areas in the third embodiment are illustrative and not restrictive. For example, a block matching method or the like can be used to determine the motion of the motion detection area. However, this method is desirable depending on the zoom ratio of the forward image or the backward image. In addition, the threshold calculation in the layer division may be performed by any method as long as the layer can be appropriately determined based on the motion information (the motion amount / direction).

Further, the processing performed in each block described above may be realized not only by hardware but also by software. FIG. 15 shows the configuration in this case.

The computer 30 incorporates a CPU (Central Processing Unit) 301 as shown in FIG.
02, RAM 303, hard disk drive 30
4, the input / output interface 305 is connected. Further, an input unit 311, a recording medium drive 312, a communication unit 313, and an image input / output unit 314 are connected to the input / output interface 305.

When a command is input from an external device or from an input unit 311 configured using operation input means such as a keyboard or a mouse or voice input means such as a microphone, the command is input. Output interface 30
5 is supplied to the CPU 301.

The CPU 301 includes a ROM 302 and a RAM 3
03 or a program stored in the hard disk drive 304 to execute a process according to the supplied instruction. Furthermore, a ROM 302 and a RAM 303
Alternatively, the hard disk drive 304 previously stores an image processing program for causing the computer 30 to execute the same processing as the above-described image processing apparatus, and generates a side moving image as a peripheral image.

FIG. 16 is a flowchart showing the overall configuration of the image processing program according to the first embodiment. In the figure, in step ST1, a motion detection process is performed to detect a motion amount of an image. Next, in step ST2, layer information indicating which layer the image belongs to is generated by the layering process, and the amount of motion for each layer is calculated. In step ST3, a peripheral image generation process is performed, and a peripheral image is synthesized based on the layer information and the amount of motion for each layer.

FIG. 17 is a flowchart showing the motion amount detection processing in step ST1 of FIG. Step ST
At 11, a motion detection area is set for the front moving image, and the process proceeds to step ST12. In step ST12, the image of the motion detection area is reduced to calculate the sum of errors, and
Proceed to T13. In step ST13, the calculated sum of errors is compared to detect a motion amount in which the sum of errors is reduced, and the process proceeds to step ST14. In step ST14, image sizes within a preset range are set in step ST12,
It is determined whether or not the processing in ST13 has been completed. If the processing has not been completed, the process returns to step ST11, and if completed, the process proceeds to step ST15. In step ST15,
The amount of motion that minimizes the error sum is defined as the amount of motion in the motion detection area.

FIG. 18 is a flowchart showing the layering process in step ST2 of FIG. Step ST2
In step 1, statistical information, for example, an average value is calculated using the motion amount detected in the motion amount detection processing in step ST1, and the process proceeds to step ST22. In step ST22, based on the statistical information, a threshold is set for layering the image into, for example, a distant view, a middle view, and a near view. In step ST23, the set threshold value and the motion amount are compared to generate layer information indicating to which layer the motion detection area belongs. In step ST24, the motion amount for each layer is calculated based on the motion amount of the motion detection area included in the layer.

FIG. 19 is a flowchart showing the peripheral image generation processing of step ST3 in FIG. Step S
At T31, an image is extracted for each layer from the front moving image based on the amount of motion and the layer information for each layer generated in the layering process. In step ST32, the image of each extracted layer is synthesized at a predetermined position. Step ST
At 33, it is determined whether or not there is an imageless portion when the images are combined, and it is necessary to perform interpolation processing to supplement the image of this portion. Here, when a portion having no image occurs, the process proceeds to step ST34, and when a portion having no image occurs, the process proceeds to step ST35. In step ST34, an interpolation process is performed, and the process proceeds to step ST35. For example, an image around a portion having no image is expanded and interpolated. In step ST35, a geometric transformation process is performed in accordance with the orientation of the screen on which the synthesized image is projected, and the image signal of the image subjected to the geometric transformation is output as an image signal of the peripheral image in step ST36.

Such an image processing program is a removable information recording / transmission medium constituted by using a removable recording medium utilizing magnetism or light, a semiconductor element, or the like, for example, an optical disk such as a floppy (R) disk or CD-ROM. Or a magneto-optical disk such as an MO disk, a tape cartridge, or a semiconductor memory. In this case, the removable information recording transmission medium is mounted on the recording medium drive 312 to read the image processing program, and the read program is stored in the ROM 302, the hard disk drive 304, or the like via the input / output interface 305 or the bus 320. It may be installed by making it.

Further, the information recording transmission medium is a wired or wireless transmission path for transmitting an image processing program, for example, an LA.
Network such as N and the Internet, or satellite broadcast waves and terrestrial broadcast waves are also included. In this case, the information recording transmission medium is connected to the communication unit 313, and the image processing program transmitted via the information recording transmission medium is transmitted to the communication unit 31.
ROM 302 or a hard disk drive 304 via an input / output interface 3, an input / output interface 305 and a bus 320.
And the like, and the image processing program is installed.

Here, when the image signal of the front moving image is input to the image input / output unit 314 during the execution of the image processing program, the processing in the above-described image processing block is performed, and based on the image signal of the front moving image. An image signal of the side moving image is generated. The image signal shown in FIG. 2 can be displayed by outputting the generated image signal of the side moving image from the image input / output unit 314 and supplying it to the projectors 12L and 12R.

As described above, according to the above-described embodiment,
By simply inputting an image signal SDC of a motion that satisfies at least one constraint condition obtained by photographing with one vehicle-mounted camera or the like, a peripheral image that does not exist in a moving image based on the image signal SDC can be presented in real time.
Multi-directional image presentation with a high sense of reality can be realized.

Further, since an image of an originally nonexistent portion is created based on an input image, it is not necessary to use a large number of cameras or special cameras to obtain an image of an originally nonexistent portion. An image with a feeling can be obtained.

Furthermore, since an image is created using a real image, it is possible to present a more realistic image. Since the three-dimensional space is not reconstructed as if a virtual space is created by computer graphics, very simple processing can be performed. An image on a projection plane different from the input image can be obtained.

Further, even for an enormous existing image source, if it is a motion image satisfying at least one constraint condition, the above-described processing is performed to input images on both side surfaces and upper and lower surfaces. Images can be presented at the same time
It is more realistic and can express a wide range of images.

By the way, the above-mentioned image processing device 15A
By adding a constraint that the moving direction of the front moving image is a radial direction from one point, the front moving image, the left side moving image, and the right side moving image are displayed in real time based on the input image signal SDC. did. However, if the stored and freely readable image signal SDC is used, the front moving image, the left side moving image, and the right side moving image based on the image signal SDC can be used without adding the above-described constraint condition. Can be displayed.

The image processing apparatus 15B for performing such processing
Supplies the stored image signal SDC of the front moving image captured by, for example, an in-vehicle camera to the projector 12C, and displays the front moving image based on the image signal SDC on the screen 10C located at the front shown in FIG. Let it. Also,
The image processing device 15B generates an image signal indicating a peripheral image at an angle of view that is not included in the visual field range of the vehicle-mounted camera based on the accumulated image signal SDC of the front moving image, for example, left and right side surfaces that are continuous with the front moving image. Intermediate image information for generating peripheral image signals SDL and SDR, which are image signals indicating a moving image, is generated and stored. When the front moving image is displayed on the screen 10C, the peripheral image signal SD of the side moving image continuous with the front moving image using the accumulated intermediate image information.
L and SDR are generated, and the peripheral image signal SDL is supplied to the projector 12L and the peripheral image signal SDR is supplied to the projector 12R. For this reason, the left side moving image having continuity with the front moving image is displayed on the screen 10L located on the left side, and the screen 10L located on the right side is displayed.
A right side moving image having continuity with the front moving image is displayed on R, and an image with a wide angle of view can be presented. The storage of the image signal SDC and the intermediate image information is performed by the image processing device 15.
B, or may be stored in external storage means.

Next, as a second embodiment of the present invention, a case where the image processing device 15B stores the image signal SDC and the intermediate image information will be described. FIG. 20 shows a schematic configuration of the image processing device 15B. The image processing device 15B can display a front moving image, a left moving image, and a right moving image using the stored image signal SDC. The image signal SDC is stored in the front moving image signal storage area 51 of the storage unit 50, and the motion detection block 40
Are supplied to the delay unit 41, the motion detection unit 42, and the information generation unit 47 of the layering processing block 45.

The delay section 41 delays the image signal SDC by one frame and supplies it to the motion detecting section 42 as the image signal SDCa. The motion detection unit 42 sets a plurality of motion detection areas divided as indicated by broken lines in FIG. 21 on the side end side of the front moving image, and sets the image signal SDC of the frame of interest in the motion detection area and one frame. A comparison with the previous image signal SDCa is performed for each motion detection area, and a motion vector M indicating the motion of the image is calculated.
VE is determined for each motion detection area and supplied to the layer classification unit 46 of the layering processing block 45.

The layer classification unit 46 determines the motion pattern of the front moving image based on the motion vector MVE of each motion detection area, and performs layer setting when generating intermediate image information. For example, the layer classification unit 46 includes a layer of a distant view image of a distant subject, a layer of a near view image of a close subject, a layer of a middle view image located between the distant view image and the near view image, and a layer of the middle view image. The layer setting is performed by determining how to provide a layer different from the layer. Here, the front moving image obtained when the front moving image is captured by the vehicle-mounted camera is an image obtained by sequentially zooming in on the front moving image when traveling straight in one direction. Further, when the car is moving backward, the front moving image is sequentially zoomed out. Further, when the overtaking vehicle is photographed, the overtaking vehicle is displayed as a zoom-out image on the zoom-in image. Also, when you make a right or left turn,
For example, the upper side of the front moving image is moved in the horizontal direction, and the lower side is a zoomed-in image. For this reason, it is assumed that the layer classification unit 46 determines the motion pattern of the front moving image based on the motion vector MVE of each motion detection area, and performs layer setting based on the determined motion pattern.
For example, when it is determined that the pattern is a motion pattern that goes straight in one direction, a layer of each image of a distant view, a middle view, and a near view is generated, and when it is determined that the motion pattern is a right turn or left turn, a distant view, a middle view, A layer including an image that is moved in the horizontal direction as well as a layer of each image in the foreground is created.
Further, when the passing vehicle has a certain motion pattern, the layer setting is performed so as to create not only the layers of the distant view, the middle view, and the near view but also the layers including the zoom-out image.

The layer classifying section 46 includes the motion detecting section 4
Based on the motion vector MVE supplied from 2, layer classification is performed on which layer each motion detection area set on the side end of the front moving image belongs. This layer classification
This is performed using a layer set according to the motion pattern.
For example, the layer classifying unit 46 classifies the three layers into a distant view, a middle view, and a near view, and when a layer including an image moved in the horizontal direction or a layer including a zoom-out image is generated, Layer classification is performed including layers. By this layer classification, the layer classification unit 46 generates layer classification information LB indicating which motion detection area belongs to which layer, and supplies the generated layer classification information LB to the information generation unit 47.

The information generating section 47 classifies each motion detection area set in the front moving image based on the layer classification information LB from the layer classification section 46, and divides the image signal of this motion detection area into frames for each layer. An intermediate image signal for each layer is generated by using the layers in order. Further, an average value of the motion amount is calculated for each layer based on the motion amount of the motion vector MVE of the motion detection area divided into layers. The intermediate image signal GYv generated by the information generation unit 47 and the calculated motion amount (average value) MYv for each layer are stored in the intermediate image information storage area 52 of the storage unit 50 as the intermediate image information.

When displaying an image on the screens 10C, 10R and 10L, the projector 12C displays an image based on the stored image signal SDC on the screen 10C. Further, the image processing device 15B reads the intermediate image information stored in the intermediate image information storage area 52 by the read control signal RC from the peripheral image signal generation block 60 connected to the storage unit 50, and The image synthesis is performed by sequentially using the intermediate image signals GYv of the respective layers by an amount corresponding to the motion amount MYv of the respective layers, and superimposing the images of the respective layers in the order of distant view, middle view, and near view. Further, when layers that do not belong to the three layers of the distant view, the middle view, and the near view are provided, the image synthesis processing of these layers is also performed to generate the peripheral image signals SDL and SDR. Further, the image processing device 15B supplies the peripheral image signal SDL to the projector 12L at a timing corresponding to the image signal SDC of the front moving image, and also causes the peripheral image signal SDR to correspond to the image signal SDC of the front moving image. The timing is supplied to the projector 12R. The projector 12L displays a left-side moving image continuous with the front moving image on the screen 10L. Further, the projector 13R displays a right side moving image continuous with the front moving image on the screen 10R.

Next, each component of the motion detection block 40 will be described in detail. For the sake of simplicity, in the following description, only the right side will be described, and description of the left side will be omitted.

The motion detection block 40 determines the motion vector MVE for each motion detection area as described above. Here, when the front moving image has a moving center of the image, that is, when the front moving image is obtained by photographing the front with the onboard camera, the image at the time T shown in FIG. At time T ′, the image shown in FIG. 5B is obtained, which is substantially equal to the zoom-in operation image centered on the image reference position CP.

Here, as shown in FIG. 5C, the enlargement ratio Z is set to reduce the motion detection area of the frame of interest to (1 / Z), and the position of the reduced motion detection area is moved. The sum of the error with the image signal one frame before is calculated. Further, by changing the enlargement factor Z, the sum of errors is calculated while moving the position in the same manner. By detecting the position where the sum of the errors becomes the minimum value in this manner, the motion vector MVE of each motion detection area can be determined for each frame of interest. Further, the enlargement ratio Z when the sum of the errors becomes a minimum value is defined as a motion amount.

By the way, when the reduction processing is performed, there are pixels in which the coordinate values of the pixels in the area do not become integer values.
On the other hand, in the image one frame before the reduction process has not been performed, the coordinate values of the pixels in the region are integer values. For this reason,
Linear interpolation is performed on the reduced image to calculate a signal level at a position where the coordinate value becomes an integer value. For example, as shown in FIG. 6 described above, for an image of Ka × Ka pixels (1
/ Z) reduces the image size to K
When the size becomes b × Kb pixels, linear interpolation is performed to calculate the signal level of the image with the number of pixels being “Kb × Kb” using the signals of Ka × Ka pixels. By calculating the error sum of the calculated signal level and the signal level one frame before the position corresponding to the reduced image, the motion amount can be determined with high accuracy.

When the image reference position CP is not clear, as shown in FIG. 22, when the center of the motion detection area ARa is shifted from the position Pa1 to the position Pa2, the sum of the errors becomes the minimum value, and the center of the motion detection area ARb is shifted to the position. If the sum of the errors becomes the minimum value when the position is shifted from Pb1 to the position Pb2, the motion detection area ARa
Vector direction of motion vector MVE and motion detection area A
The image reference position CP can be detected by detecting a point where the vector direction of the motion vector of Rb intersects.

At the time of right turn or left turn, for example, the upper image of the front moving image moves horizontally. Therefore, when the minimum value of the error sum is not detected even when the zoom-in operation is performed around the image reference position CP, the image of the motion detection area is moved in the horizontal direction to determine the minimum value of the error sum. At this time, the amount of movement of the motion detection area until the sum of the errors becomes the minimum value can be set as the amount of motion of the motion vector MVE.

Next, at the time of retreat, it looks as if the image is sucked into the image reference position CP.
The image at the time of retreat is substantially equal to the zoom-out operation image centered on the image reference position CP. For this reason, the enlargement factor Z is set to “1” or less. That is, since the motion of the image at the time of backward movement is opposite to that at the time of forward movement, a plurality of motion detection areas are set on the side end of the previous frame, and
Z), and the sum of the error with the image of the frame of interest is calculated while moving the position of the reduced area.
Alternatively, each motion detection area of the frame of interest is (1 / Z)
And, while moving the position of the reduced area, calculate the error sum with the image one frame later. Further, by changing the enlargement factor Z, the sum of errors is calculated while moving the position in the same manner. By detecting the position where the sum of the errors becomes the minimum value in this manner, the motion vector MVE at the time of retreat can also be determined.

When the motion vector MVE is discriminated as described above, the amount of motion of the motion vector MVE becomes small because the image of the distant view has little motion, and the amount of motion of the motion vector MVE is small because the image of the near view has much motion. growing.

FIG. 23 shows the structure of the motion detecting section 42. The image signal SDC is supplied to the size converting section 421, and the image signal SDCa supplied from the delay section 41 is supplied to the error sum calculating section 422. Is done. Size conversion unit 421
Sets the motion detection area by dividing a side end portion of the front moving image into a plurality of motion detection areas, for example, a plurality of motion detection areas in units of 16 × 16 pixels. Further, the size conversion unit 421 generates an image signal FEz obtained by multiplying the image of the motion detection area by (1 / Z) using the enlargement ratio Z supplied from the search control unit 425 described later, and also generates the image reference position C
The coordinate value Qz converted by multiplying the image of the motion detection area by (1 / Z) based on P is set, and the image signal FEz and the coordinate value Qz are supplied to the error sum calculation unit 422. Note that the image signal FEz is a signal when the coordinate value is converted into an integer value by the interpolation process as shown in FIG. 6 described above, and the coordinate value Qz is the coordinate when the coordinate value is converted into the integer value by the interpolation process. The value, that is, “Kb
× Kb ”.

The error sum calculation section 422 is provided by the size conversion section 42
The signal at the position indicated by the coordinate value Qz from 1 is converted into an image signal S
While selecting from DCa, the sum of the error between the selected signal and the image signal FEz is calculated and notified to the comparing unit 423.

The comparing section 423 compares the minimum value of the error sum with the error sum calculated by the error sum calculating section 422. Here, when the error sum minimum value is not set, the comparing unit 423
The error sum calculated first is set as the error sum minimum value. When the calculated error sum is smaller than the error sum minimum value, the comparing unit 423 sets the error sum as a new error sum minimum value, and notifies the data storage unit 424 that the error sum minimum value has been updated. Notice. Also, the comparison unit 423
Notifies the search control unit 425 by a signal ES that the comparison between the minimum error sum and the error sum calculated by the error sum calculation unit 422 is completed. Note that the error sum minimum value may be set in advance to a value larger than the calculated error sum.

Data storage unit 424 stores enlargement factor Z notified by search control unit 425 when comparison unit 423 notifies that the minimum error sum has been updated. When the enlargement ratio is already stored, the data storage unit 4
24 updates the stored enlargement ratio with the notified enlargement ratio Z. When the search control unit 425 notifies the completion of the change processing of the enlargement ratio by the signal ER, the data storage unit 424 sets the stored enlargement ratio as the motion amount and sets the image reference position direction as the vector direction. The motion vector MVE is supplied to the layer classification unit 46.

The search control unit 425 sets the lower limit value of the enlargement ratio to “1” and also sets the upper limit value in advance.
21 and the data storage unit 424. Thereafter, each time the comparison unit 423 notifies that the comparison between the minimum sum of errors and the sum of errors calculated by the sum of errors calculation unit 422 is completed,
The enlargement ratio Z is sequentially increased and notified to the size conversion unit 421 and the data storage unit 424. Thereafter, when the enlargement ratio Z reaches the upper limit, the search control unit 425 notifies the data storage unit 424 of the completion of the enlargement ratio change processing.

When the minimum error sum obtained by the comparison unit 423 is not small, that is, when an image equal to the side edge of the front moving image cannot be detected, the search control unit 425
The enlargement factor Z is notified to the size conversion unit 421 as "1", and the image signal FEz of the motion detection area is calculated by the error sum calculation unit 42.
2 to be supplied. Further, the control signal RP is output to the error sum
The signal at the position where the motion detection area of the image signal FEz is moved by a predetermined amount in the horizontal direction is selected from the image signal SDCa. Thereafter, by moving the position selected from the image signal SDCa in the horizontal direction by the control signal RP and determining the minimum value of the error sum, the motion vector MVE of the image to be moved in the horizontal direction can also be obtained. Further, although not shown, the signal after one frame is calculated by the error sum calculation unit 42.
2 or a motion detection area is set in the image one frame before, and the signal of the image of the frame of interest is calculated by the error sum calculation unit 42.
2, the amount of motion of the image sucked into the image reference position CP can also be determined.

As described above, the search direction is set to the image reference position CP.
And the horizontal direction to detect the position of the image in the motion detection area and the image position of the other frame that minimizes the error sum. The motion vector MVE can be correctly obtained even for an image of a car or a person crossing or an image of a passing vehicle.

The layer classification unit 46 determines what kind of motion pattern the front moving image is based on the motion vector MVE of each motion detection area, and sets a layer based on the determined motion pattern. , And supplies layer classification information LB indicating which region belongs to which layer to the information generation unit 47.

FIG. 24 shows the configuration of the layer classification section 46. Motion pattern determination unit 461 of layer classification unit 46
Accumulates the motion vector MVE of each motion detection area supplied from the motion detection unit 42 in frame units, and determines a motion pattern based on the accumulated motion vector MVE.

Here, the motion pattern determination unit 461 determines that the vector direction of the motion vector MVE in each motion detection area is
When the image is in the radial direction from the image reference position CP, it is determined whether or not all images are in the zoom-in operation direction in which the image boils from the image reference position CP. It is determined that the vehicle is moving straight. For example, when the vector direction of the motion vector MVE is set to the radial direction from the image reference position CP as indicated by the arrow in FIG.

Next, when the motion pattern judging section 461 does not judge that the motion is the straight-ahead motion, the vector direction of the motion vector MVE in each motion detection area is opposite to the radial direction, and all the images are sucked into the image reference position CP. It is determined whether or not the zoom-out operation direction is to be performed. Here, when all the images are in the zoom-out operation direction in which the images are sucked into the image reference position CP, it is determined that the operation is the backward operation. For example, FIG.
The direction of the image reference position CP as shown by the arrow of
When it is detected that all the images are in the zoom-out operation direction sucked into the image reference position CP, it is determined to be the retreat operation.

When the movement pattern determination section 461 does not determine the straight-forward operation or the backward movement, it determines whether only a part is in the zoom-out operation direction and only a part is in the zoom-out operation direction. Sometimes, it is determined that there is an overtaking vehicle. For example, as shown in FIG. 25C, when it is detected that the motions in the motion detection areas on both left and right end portions are in the zoom-in operation direction and only a part is in the zoom-out operation direction, it is determined that there is an overtaking vehicle. Furthermore, when it is not determined that there is a straight-ahead operation, a reverse operation, and an overtaking vehicle, when the vector direction of the motion vector MVE is horizontal in the upper motion detection region of the front moving image as shown in FIG. 25D. Is determined to be a right or left turn operation according to the vector direction. Also, as shown in FIG. 25E, a motion vector M
When the vector direction of the VE is the horizontal direction, it is determined that the traversing operation is performed. The motion pattern MP determined in this way is notified to the layer determination unit 462. In addition, as a motion pattern, not only the above-mentioned pattern, but also a pattern in which a right turn or a left turn is performed while stopping or retreating the motion can be considered.

The layer determination section 462 determines whether or not the determined motion pattern continues for a predetermined number of frames or more based on the motion pattern MP determined by the motion pattern determination section 461. Here, when the motion pattern continues for a predetermined number of frames or more, the layer determination unit 462 generates layer pattern information LP corresponding to the determined motion pattern and notifies the classification unit 463 of the layer pattern information LP.

Here, when the layer determination unit 462 determines continuously that the motion pattern is such that the entire screen is enlarged, such as a forward motion, by a predetermined number of frames or more, for example, the layer of the distant view, middle view, and near view is determined. It generates layer pattern information LP for instructing creation and notifies the classifying unit 463 of it. Also,
When the layer determination unit 462 determines that the movement pattern in which the upper part moves in the horizontal direction, such as a right turn or left turn operation, is continuously determined by a predetermined number of frames or more, the layer determination unit 462 determines not only the distant, middle, and near layers but also the horizontal Layer pattern information L instructing creation of a layer including an image moving in the direction
P is generated and notified to the classification unit 463. Further, when it is determined that the motion pattern includes an image that shrinks with time, such as a passing vehicle or a retreat, continuously for a predetermined number of frames or more, only a distant view, a middle view, and a near view layer are determined. Instead, it generates layer pattern information LP for instructing the creation of a backward layer including an image to be reduced. Also,
Layer pattern information that instructs creation of a layer including an image that moves in the horizontal direction when it is continuously determined that a part of the pattern is a movement pattern that moves in the horizontal direction, such as a crossing object, in a predetermined number of frames. Generate LP. As described above, when the determined motion pattern continues for a predetermined number of frames or more, the layer pattern information LP corresponding to the determined motion pattern MP is generated. However, the layer pattern information LP corresponding to the correct motion pattern can be generated.

The threshold value setting section 464 uses the motion vector MVE in a predetermined time range (for example, 30 frames before and after the frame of interest), and calculates the motion amount of the motion vector MVE whose vector direction is the radial direction from the image reference position CP. The average value Vavg, the maximum value Vmax, and the minimum value Vmin are obtained, and the threshold value Th for classifying into the layer indicated by the layer pattern information LP is set based on the average value Vavg, the maximum value Vmax, and the minimum value Vmin. And supplies it to the classification unit 463.

For example, when the layer pattern information LP indicates that the image is to be divided into distant, middle, and near views,
The threshold value setting unit 464 calculates the threshold value Th-1 indicating the division position between the distant view layer and the middle view layer based on the above equation (1). Further, the threshold value Th-2 indicating the division position between the middle view layer and the near view layer is calculated based on the above-described equation (2).

The setting of the threshold value may be similar to the case shown in FIG. 10 described above, in which a histogram of the amount of motion is obtained, and the threshold values Th-1 and Th-2 are obtained by using the minimum value of the histogram. good. As described above, since the threshold value Th is dynamically changed according to the distribution of the motion amount, the image is not classified into only one layer, and the layer classification can be appropriately performed according to the distribution of the motion amount. It can be carried out.

The classification section 463 determines which motion detection area of each frame belongs to any of the layers instructed to be created by the layer pattern information LP based on the threshold value Th from the threshold value setting section 464 and the motion amount of the motion vector MVE. And layer classification is performed. In addition, the classification unit 463 assigns a motion detection area moving in the horizontal direction and a motion detection area in the zoom-out operation direction to the corresponding layers. When the classification unit 463 completes the layer classification of each motion detection area of the frame of interest, the classification unit 463 generates layer classification information LA indicating the result of the layer classification, and generates the classification correction unit 465.
To supply.

When creating three layers, for example, a distant view, a middle view, and a near view, the classifying section 463 uses the motion vector MVE from the motion detecting section 42 to generate a previous image for each motion detecting area.
The average value of the motion amounts for the frame and the subsequent n frames is calculated. For example, when the motion amount of the motion vector MVE changes as time elapses in the motion detection area provided at the right end of the front moving image as shown in FIG. 26A (the numbers in the diagram indicate the motion amount). The classification unit 463 calculates the motion amount of the motion detection region AR1 in the Fp frame, the motion amount of the motion detection region AR1 in the Fp-1 to Fp-m frames, and the motion amount of the motion detection region AR1 in the Fp frame. The average value of the motion amount is calculated using the motion amount of the motion detection area AR1 in the +1 to Fp + n frames. Further, by comparing the average value calculated as the amount of motion of the motion detection area AR1 with the threshold values Th-1 and Th-2 set as described above, the motion detection area AR1 in this Fp frame is displayed in a distant view. , Middle view or near view. Thus, the classification unit 463
Calculates the average value of the motion amount of the motion detection region AR1 of the frame of interest using the motion amount of the previous frame and the subsequent frame, and performs the layering of the motion detection region AR1 using the average value. Even if an error occurs in the motion amount of the detection area AR1, the layer of the motion detection area AR1 can be classified. Further, even if the movement amount of each frame changes due to the difference in the size of the subject or the difference in the distance to the subject, the influence of these changes can be prevented. Further, the classification unit 463
Performs the same processing on other areas and other frames, so that each motion detection area is set to a distant view, as shown in FIG. 26B.
The image is classified into one of the middle view and near view layers, and the layer classification information LA is notified to the classification correction unit 465. FIG. 26B
In FIG. 7, the motion detection area indicated by the cross hatch indicates that the image is classified as a near view, the hatched motion detection area indicates a middle view, and the other motion detection areas indicate a distant view.

The classification correction section 465 corrects an area in which the number of connected areas belonging to the same layer is smaller than a predetermined number by referring to the layer classification information LA of each motion detection area to a layer adapted to the surrounding area. For example, in FIG. 26B, the area AR4-g of the Fp + 4 frame and the area AR6-h of the Fp-2 frame are
Regions belonging to the same layer are not connected. For this reason,
The classifying and correcting unit 465 corrects these areas into areas that match the surroundings as shown in FIG. 26C, and layer classification information LB indicating which area is which layer.
Is supplied to the information generation unit 47.

When there is an overtaking vehicle, not only the distant, middle, and near views described above, but also a retreat layer including an image of the overtaking vehicle is generated as shown in FIG. 26D. Here, since the overtaking vehicle has a motion vector MVE whose vector direction is opposite to that of the images of the distant view, the middle view and the near view, the time axis direction of the backward layer is generated with the direction opposite to the other layers.

FIG. 27 shows the configuration of the information generating section 47. Layer classification information L supplied from the layer classification unit 46
B is supplied to the motion amount average value calculation unit 471 and the intermediate image signal generation unit 472.

The motion amount average value calculation section 471 calculates the average value of the motion amount for each layer for each frame using the motion amount of each motion detection area. For example, in the Fe frame, when the number of motion detection areas determined to be a distant view layer is ny, the motion amount average value calculation unit 471 calculates an average value using the motion amounts of the ny motion detection regions, and calculates an intermediate image. The calculated motion amount MYv is supplied to the signal generation unit 472 and stored in the intermediate image information storage area 52 of the storage unit 50 as the intermediate image information.

The intermediate image signal generating section 472 performs, on the basis of the layer classification information LB supplied from the layer classification section 46, a layer-by-layer basis from the side end in accordance with the motion amount MYv calculated by the motion average value calculating section 471. The image signal SDC is extracted, an intermediate image signal GFv for each layer is generated, and supplied to the intermediate image signal interpolation unit 473.

The intermediate image signal interpolating section 473 reads the image signal SDC corresponding to the signal amount corresponding to the motion amount MYv of each layer from the side end to generate an intermediate image signal. For example, an image based on the intermediate image signal is as shown in FIG. 28A shows an image based on the intermediate image signal of the distant view layer, FIG. 28B shows an image based on the intermediate image signal of the middle view layer, and FIG. 28C shows an image based on the intermediate image signal of the near view layer, which is hatched. The part is an area without an image.

The intermediate image signal interpolating unit 473 may generate an area without an image when performing image synthesis by superimposing images of respective layers having different motion amounts MYv in the order of distant view, middle view, and near view. So that there is no intermediate image signal GFv
And the corrected signal is stored in the intermediate image information storage area 52 as an intermediate image signal GYv. In this correction of the intermediate image signal, if all the layers are interpolated with respect to the near layer, the deep layer will be hidden. Therefore, the intermediate image signal interpolating unit 473 interpolates the entire gap for the innermost layer, and for the gap located at the intermediate layer, the gap generated by the region belonging to the layer before this layer. Only in this case, an interpolation process using signals adjacent in the horizontal direction is performed to generate an image signal in a gap region where no image exists, and the image signal is superimposed on the intermediate image signal. For example, when the middle layer is the middle view layer in FIG. 28B and the near layer is the near view layer in FIG. 28C, the gap is an area of a tree or a car in the middle view layer. By performing the interpolation processing in this manner, the image shown in FIG. 28A based on the intermediate image signal GFv is obtained by interpolating a portion having no image (a region shaded with oblique lines) using a signal adjacent in the horizontal direction. , The image shown in FIG. 29A. Similarly,
The image shown in FIG. 28B based on the intermediate image signal GFv is a corrected image as shown in FIG. 29B.
Note that, for example, when the foreground layer shown in FIG. 28C is the foremost layer, no area without an image is generated in the foreground layer, so that the interpolation processing for the foreground layer is unnecessary. . Therefore, the foreground layer shown in FIG. 29C is the same as that in FIG. 28C.

The intermediate image information storage area 52 of the storage unit 50
Stores, as the intermediate image information, a relation between the motion amount MYv for each layer generated by the layering processing block 45 and the intermediate image signal GYv for each layer for each frame. Here, since the intermediate image signal of the distant view layer stored in the intermediate image information storage area 52 has a small amount of motion, the signal amount of the image signal is small as shown in FIG. 29A. Further, in the intermediate image signal of the foreground layer, since the amount of motion is large, the amount of signal also increases as shown in FIG. 29C. In the middle view layer, FIG.
As shown in B, the signal amount is between the distant view and the middle view.
In the backward layer, since the motion direction is opposite, the frame direction of the intermediate image signal of the backward layer is opposite to that of FIGS. 29A to 29C.

The intermediate image information storage area 52, when instructed to create a horizontal movement layer based on the layer pattern information LP, stores the image signal of the motion detection area in which the direction of the motion vector is horizontal. Is set to the intermediate image signal of this horizontal movement layer. Here, the image moving in the horizontal direction includes an image moving outward from the front moving image and an image moving inside. For example, when making a right turn, an image enters from the right end of the front moving image and exits from the left end. For this reason, the horizontal movement layer of the image moving outside is set to have the same time axis direction as the distant view layer and the like. Further, the horizontal movement layer of the image that enters the inside is the same as the backward layer in the direction of the time axis that is opposite to that of the distant view layer or the like.

In this manner, the intermediate image information storage area 5
Reference numeral 2 generates and stores intermediate image signals of the distant, middle, and near layers and intermediate image signals of the backward layer and the horizontally moving layer according to the amount of motion. Further, the amount of motion of each layer is also stored as described above.

Next, a case where the peripheral image signal SDR of the right side image is generated using the intermediate image information stored in the intermediate image information storage area 52 of the storage section 50 will be described.

When generating the peripheral image signal SDR, the peripheral image signal generating block 60
2 to the motion amount MYv stored as intermediate image information
And the signal readout amount of the intermediate image signal GYv for each layer is determined based on the movement amount MYv. further,
Image synthesis is performed by superimposing in the order of a distant view layer, a middle view layer, and a near view layer using an intermediate image signal of a signal amount corresponding to the motion amount MYv for each layer. Further, when a layer different from the distant, middle, and near view layers is provided, the image synthesis processing of this layer is also performed, and the peripheral image signal SDR can be generated. By supplying the generated peripheral image signal SDR to the projector 12R in accordance with the output timing of the image signal SDC of the front moving image, not only the front moving image but also the image on the right side can be continuously displayed.

FIG. 30 shows the configuration of a peripheral image signal generation block 60 for reading out intermediate image information stored in the intermediate image information storage area 52 and generating a peripheral image signal. The image generation control unit 61 of the peripheral image signal generation block 60 reads out the motion amount MYv for each layer stored as intermediate image information in the intermediate image information storage area 52 in frame order, and based on the read motion amount MYv, The readout amount of the intermediate image signal for each is determined. Also,
The image generation control unit 61 reads the intermediate image signal GYv from the intermediate image information storage area 52 based on the determined read amount.
Is read out for each layer and supplied to the composing units 621-1 to 621-5 for each layer provided in the signal extracting unit 62.

Here, the intermediate image signal of the horizontal moving layer that moves in the horizontal direction when turning right and left is supplied to the synthesizing unit 621-1. The intermediate image signal of the distant view layer is supplied to the synthesizing unit 621-2.
And the intermediate image signal of the middle view layer and the intermediate image signal of the near view layer are supplied to the combining units 621-3 and 621-4. Further, the intermediate image signal of the backward layer including the image of the overtaking vehicle is supplied to the combining unit 621-5.

An image shift unit 623-1 described later is connected to the synthesizing unit 621-1.
Image synthesis is performed by superimposing the image signal of the corresponding layer read from the intermediate image information storage area 52 on the image signal supplied from the image shift unit 623-1. The image signal of the horizontal movement layer obtained by performing the image synthesis by the synthesis unit 621-1 is supplied to the delay unit 622-1 and the distant view synthesis unit 641 of the image signal generation unit 44.

The delay section 622-1 delays the image signal supplied from the synthesizing section 621-1 by one frame and supplies the delayed image signal to the image shift section 623-1. The image shift unit 623-1 is
The image based on the image signal supplied from the delay unit 622-1 is moved in the horizontal direction based on the motion amount MYv of the horizontal movement layer supplied from the image generation control unit 61. Further, the image signal of the image moved in the horizontal direction is
21-1. The synthesizing unit 621-1 superimposes an image based on the intermediate image signal GYv read by the motion amount MYv of each layer from the intermediate image information storage area 52 on the image moved in the horizontal direction. Is performed to generate an image signal in which the image sequentially moves.

Similarly, delay sections 622-2 to 622-5 generate image signals delayed by one frame, and image shift sections 623-2 to 623-5 generate image signals delayed by one frame. Is moved in the horizontal direction based on the motion amount MYv of each layer supplied from the image generation control unit 61.
Further, the synthesizing units 621-2 to 621-5 output the intermediate image signal GY read from the intermediate image information storage area 52 by the amount of movement of each layer for the image moved in the horizontal direction.
Image synthesis is performed by superimposing images based on v,
An image signal in which an image sequentially moves is generated for each layer.

Since the intermediate image signal is a signal read from the front moving image signal, an image based on this intermediate image signal is an image on the same plane as the screen 10C.
However, the screen 10R for displaying the peripheral image is provided to be inclined with respect to the front screen 10C. For this reason, the image of each layer is superimposed by using an image signal of a layer such as a distant view, a middle view, a near view layer, or a backward layer in which the direction of the motion vector is the direction of the image reference position CP or the reverse direction. Assuming that the image is synthesized and the peripheral image signal SDR is generated and the image is displayed,
The image displayed on the screen 10R is the screen 10C.
The shape may not be correct due to the inclination of the screen 10R with respect to. For this reason, the image signal of the layer in which the direction of the motion vector is the direction of the image reference position CP or the opposite direction is supplied to each of the conversion units 631 to 634 of the projection conversion unit 63, and the image of each layer is displayed on the screen 10L, When the image is displayed on 10R, projective transformation is performed so that the image moves correctly in the vector direction of the motion vector MVE. In the image of the horizontal movement layer, since the vector direction of the motion vector MVE is the horizontal direction, projective transformation such as a distant view, a middle view, a near view layer, and a backward layer is not performed.

Conversion units 631 to 634 of projection conversion unit 63
Performs projective transformation on the image signals supplied from the combining units 621-2 to 621-5 so that the image has a shape corresponding to the direction of the screen 10R. Here, when the images are sequentially read from the front moving image, the display surface 10 of the read image is displayed.
The CR and the screen 10R are the same as those shown in FIGS.
As shown in B, when the image read from the front moving image is displayed on the screen 10R, the image on the display surface 10CR is directly projected onto the surface of the screen 10R, and does not have a correct shape.

For this reason, the conversion units 631 to 634 enlarge the image in the vertical direction at a magnification ratio proportional to the distance from the center Czm of zooming in or zooming out.
Projection transformation is performed to enlarge the image in the horizontal direction at an enlargement factor such that the speed is proportional to the distance from the center Czm of zooming in or out. In this processing, as described above, the position (x, y) of the image read directly from the front screen edge and the position (X, Y) after the conversion processing so that the image can be displayed correctly on the right-side moving image are: For the vertical direction, the above equation (5) is used, and for the horizontal direction, the above equation (4) is used. For this reason, as shown in FIG. 14D, the (X, Y) image signal on the screen 10R is obtained by using the signal at the position (x, y) satisfying Expressions (5) and (6), as shown in FIG. Screen 1 as shown
The right side moving image can be displayed correctly on 0R.

The image signal of the distant view layer after the projective conversion obtained by the conversion section 631 is supplied to the distant view synthesis section 641 of the image signal synthesis section 64. Further, the image signal of the middle ground layer after the projective conversion obtained by the conversion unit 632 is
2 and the image signal of the foreground layer after the projective transformation obtained by the transformation unit 633 is supplied to the foreground composition unit 643. Further, the image signal of the backward layer obtained by the conversion unit 634 is supplied to the reduced image synthesis unit 644.

The distant view synthesizing unit 641 generates an image signal by superimposing the image of the distant view layer on the image of the horizontal moving layer based on the image signal supplied from the synthesizing unit 621 and the converting unit 631. And supplies it to the middle view synthesis section 642.

The middle scene combining section 642 superimposes the image of the middle scene layer on the image in which the image of the far scene layer is superimposed based on the image signal supplied from the conversion section 632 and the far scene combining section 641. To generate an image signal and supply it to the foreground composition unit 643.

The foreground composition section 643 superimposes the image of the foreground layer on the image on which the image of the middle layer is superimposed, based on the image signal supplied from the conversion section 633 and the intermediate scene composition section 642. To generate an image signal and supply it to the reduced image synthesizing unit 644.

[0139] Based on the image signals supplied from the conversion unit 634 and the foreground composition unit 643, the reduced image composition unit 644 adds the image of the retreat layer whose image is reduced with time to the image on which the image of the foreground layer is superimposed. Image superimposition is performed by superimposing to generate an image signal. This reduced image synthesizing unit 6
The image signal generated in 44 is an image signal of a side moving image on which image synthesis in which layers from the horizontal movement layer to the reduction layer are superimposed is performed.

Therefore, the peripheral image signal generation block 60
Supplies the image signal generated by the reduced image synthesizing unit 644 to the projector 12R as a peripheral image signal SDR, so that the projector 12R can display an image on the right side continuous with the front moving image on the screen 10R.

When the intermediate image signal stored in the intermediate image information storage area 52 has not been subjected to the process of interpolating the non-image portion, or the reduced image synthesizing unit 64
When an imageless portion occurs in the image signal generated in step 4, an interpolation unit is provided in the image signal combining unit 64, not shown,
It is assumed that the interpolation unit determines what layer image has been superimposed in a region adjacent to a portion having no image, and performs an interpolation process using the image of the deep layer. For example, when there is no image in a portion horizontally adjacent to a region where the images of the middle view layer and the near view layer are superimposed, the interpolation unit performs the interpolation process using the image of the middle view layer having a large depth. , It is possible to generate a good peripheral image signal SDR without image loss.

The peripheral image signal generation block 60
By generating the peripheral image signal SDL and supplying the generated peripheral image signal SDL to the projector 12L in the same manner as the peripheral image signal SDR, an image on the left side continuous with the front moving image can be displayed on the screen 10L by the projector 12L.

Further, the storage section 50 stores intermediate image information, and the intermediate image information is a motion amount and an intermediate image signal for each layer. For this reason, the amount of information stored in the storage unit 50 is smaller than that in the case where the image signal of the peripheral image is stored. Further, the image processing device 15B can perform image display at a wide angle of view without using a large-capacity recording medium or the like.

Further, the processing performed in each block described above may be realized by software. The configuration in this case is the same as that in FIG. 15 described above, and the description is omitted.

FIG. 31 is a flowchart showing the overall configuration of an image processing program according to the second embodiment. In the figure, in step ST41, a motion detection process is performed to detect a motion vector of an image. Next, step S
In T42, intermediate image information generation processing generates image information for each layer, for example, a distant view layer, a middle view layer, and a close view layer, based on the motion vector, and intermediate image information indicating the motion amount of each layer. In step ST43, a peripheral image generation process is performed, and a peripheral image is synthesized based on the intermediate image information.

FIG. 32 is a flowchart showing the motion detection processing in step ST41 of FIG. Step ST
At 51, a motion detection area is set for the front moving image, and the process proceeds to step ST52. In step ST52, the image of the motion detection area is reduced or the position is changed to calculate an error sum, and the process proceeds to step ST53. In step ST53, the calculated error sum is compared to detect an image position where the error sum is reduced, and the process proceeds to step ST54. Step S
At T54, it is determined whether or not the examination within the preset image size and position range has been completed. If not, the process returns to step ST51, and if completed, the process proceeds to step ST55. In step ST55, a motion vector indicating the motion of the motion detection area is determined based on the image position where the error sum is minimized.

FIG. 33 is a flowchart showing the intermediate image information generation processing of step ST42 in FIG. In step ST61, it is determined based on the motion vector detected in the motion detection processing in step ST41 whether the motion pattern is, for example, a straight-line operation, a backward operation, or a right-turn or left-turn operation. , And generates layer pattern information indicating what layer is provided. In step ST62, it is determined whether or not the determined motion pattern is an enlarged motion of the entire screen.
Here, when the movement is the enlargement movement, the process proceeds to step ST63, and when the movement is not the enlargement movement, the process proceeds to step ST65.
In step ST63, statistical information, for example, an average value is calculated using the motion amount of the motion vector, and the process proceeds to step ST64. In step ST64, based on the statistical information, a threshold value for layering the image into, for example, a distant view, a middle view, and a near view is set. In step ST65, the set threshold value and the amount of motion are compared to generate layer information indicating to which layer the motion detection area belongs. When the entire screen is not an enlarged motion, layer information is generated with the motion detection area as a layer according to the motion pattern.

In step ST66, the motion amount for each layer is calculated based on the motion amount of the motion detection area included in the layer. In step ST67, based on the layer information and the amount of motion for each layer, an image is extracted for each layer from the front moving image to generate an intermediate image. In step ST68, it is determined whether or not there is an isolated region in the intermediate image. If there is an isolated region, the process proceeds to step ST69. If there is no isolated region, the process proceeds to step ST70. In step ST69, interpolation processing is performed to remove an isolated region, and the process proceeds to step ST70. In step ST70, an intermediate image for each layer without an isolated area and the amount of motion for each layer are generated as intermediate image information.

FIG. 34 is a flowchart showing the peripheral image generation processing of step ST43 in FIG. In step ST81, an image is extracted for each layer from the intermediate image for each layer based on the motion amount for each layer indicated by the intermediate image information. In step ST82, the image of each extracted layer is synthesized at a predetermined position. Step ST
At 83, it is determined whether or not there is an imageless portion when the images are combined, and whether or not interpolation processing to supplement the image of this portion is necessary. Here, when there is a portion without an image, the process proceeds to step ST84, and when there is no portion without an image, the process proceeds to step ST85. In step ST84, an interpolation process is performed, and the process proceeds to step ST85. For example, an image around a portion having no image is expanded and interpolated. In step ST85, a geometric transformation process is performed according to the orientation of the screen on which the synthesized image is projected, and the image signal of the image subjected to the geometric transformation is output as an image signal of the peripheral image in step ST86.

As described above, such an image processing program may be recorded on a removable information recording / transmission medium or transmitted via an information recording / transmission medium.

In the above-described second embodiment,
When the image processing program is executed to generate the intermediate image information, the intermediate image information
Disk drive 304 or recording medium drive 3
12 is recorded on a recording medium attached to the recording medium 12. Further, when the output of the image signal of the peripheral image is requested, the intermediate image information is read out and the peripheral image signal SD of the left and right side moving image is read.
L and SDR are generated, and the peripheral image signal SD
L and SDR are output in synchronization with the image signal SDC of the front moving image. For this reason, a continuous image with a wide angle of view can be displayed using the front and left and right screens.

By the way, in the second embodiment described above, based on the motion vector MVE of the motion detection area provided on the side end side of the front moving image, layering and moving of images such as distant, middle, and near views are performed. The amount of intermediate image information is determined, the intermediate image information is generated and stored in the intermediate image information storage area 52, and the peripheral image is displayed using the intermediate image information stored in the intermediate image information storage area 52. Thus, the peripheral image signal is generated by reading out the intermediate image signal GYv in accordance with the motion amount MYv of each layer and superimposing the images. However, the peripheral image signal can be generated without using the motion vector MVE.

Next, an image processing apparatus according to a third embodiment of the present invention will be described. This image processing device 15C
The peripheral image signal is generated without using the motion vector MVE.

The image processing device 15C generates a projected image in the lateral direction based on the front moving image by geometric transformation, performs an integration process on the projected image to generate an integrated image, and uses the integrated image to generate a layer of an image. By performing the division, a motion amount MYs for each layer and an intermediate image signal for each layer are generated,
It is stored in the intermediate image information storage area 52 as intermediate image information.

FIG. 35 shows the configuration of the image processing device 15C. The image signal SDC of the front moving image is supplied to the geometric conversion unit 71. The geometric transformation unit 71 generates a projected image signal SDP of an image in which the front moving image is displayed on the projection plane, which is a plane in the traveling direction, by using the image signal SDC of the front moving image by a geometric transformation process. The projected image signal SDP generated by the geometric transformation unit 71 is
Are supplied to the motion discrimination image generation unit 73, the integral image generation unit 74, and the information generation unit 82 of the layering processing block 80.

The motion discrimination image generation unit 73
By using the projected image signal SDP supplied from No. 1 and extracting a projected image signal SDP of a predetermined range from an end portion in contact with the front moving image for each frame and sequentially superimposing the images, the image synthesis is performed to perform the motion discrimination image. Generate a signal SMD. The motion determining image signal SMD is supplied to the motion determining unit 75.

The integrated image generating section 74 generates one integrated image signal by adding and averaging the projected image signals SDP for ms frames while shifting the position in the horizontal direction. In addition, ns integrated image signals are generated with the shift amounts of ns different positions. The ns integrated image signals SA generated by the integrated image generating unit 74 are supplied to the motion determining unit 75.

The motion discriminating section 75 compares the ns integrated image signals SA with the motion discriminated image signal SMD, and detects an image based on the integrated image signal SA matching with an image based on the motion discriminated image signal SMD. . Further, the motion determining unit 75 determines the amount of motion at a predetermined position of the frame of interest based on the detection result. Here, the integrated image is an image in which this subject appears as an image when the amount of position shift matches the amount of image movement. Also, since only ns integrated images are generated with different shift amounts, the integrated image signal SA indicates ns images with different moving amounts. For this reason, the motion determining unit 75
The correlation between the ns integrated image signals SA and the motion discrimination image signal SMD is discriminated, and based on the discrimination result, the movement amount of the subject constituting the motion discrimination image, that is, the movement amount of a predetermined position of the frame of interest is discriminated. Here, when the image of the subject is a distant view, the moving amount is small. Also, when the image of the subject is in the foreground, the movement amount is large. That is, this movement amount indicates the depth of the image of the subject. In this manner, the motion determining unit 75 determines the amount of motion at each position of the projected image, generates the amount of motion information MF indicating the determination result, and generates the amount of motion information MF indicating the determination result. 82.

The layer classification section 81 generates layer classification information LB indicating which of the plurality of layers the predetermined position of the frame of interest belongs to, based on the supplied motion amount information MF, and supplies it to the information generation section 82.

The information generating section 82 calculates an average value for each layer using the moving amount of the preceding and succeeding frames for each frame, and obtains the average image information storing area of the storing section 50 as the moving amount MYs of the corresponding frame of each layer. 52. Further, the information generation unit 82 calculates the motion amount MYs for each layer obtained.
And the layer classification information L supplied from the layer classification unit 81
B, the image signal S supplied from the geometric transformation unit 71.
DP is classified for each layer to generate an intermediate image signal GFs. In other words, the information generation unit 82 performs image synthesis by reading image signals in the horizontal direction by the amount of motion and superimposing the image signals sequentially for each corresponding layer, thereby generating an intermediate image signal GFs for each layer similar to FIG. Generate.

When the moving directions of the images of the distant view layer, the middle view layer, and the near view layer are set as references, the moving direction of the backward layer is reverse. Therefore, the information generation unit 82
Generates an intermediate image signal of the backward layer by reading the projection plane image signal from the end side in contact with the front moving image by the absolute value of the motion amount, performing left-right inversion and superimposing the images, and performing image synthesis.

Further, the information generating section 82 performs an interpolation process on the created intermediate image signal GFs for each layer so as not to generate a region without an image, and embeds a portion without an image. The intermediate image signal GYs for each layer obtained by performing the interpolation processing in the information generation unit 82 is stored in the intermediate image information storage area 52 as intermediate image information in association with the motion amount MYs of each layer.

The peripheral image signal generation block 60 can generate the peripheral image signal of the right side moving image by reading the intermediate image information from the intermediate image information storage area 52 and processing it in the same manner as described above.

Next, each block will be described in detail. FIG. 36 is a diagram for describing the geometric transformation processing in the geometric transformation unit 71. In FIG. 36, a position O is a photographing position of the front moving image UC. Ideally, the photographing position and the viewer position should be the same. Here, it is assumed that the photographing position and the viewer position are the same. The geometric conversion unit 71
The image at the position p of the front moving image is projected onto the projection plane F parallel to the traveling direction.
The projection image UI is projected on the position q on I, and a projection image UI is generated based on the front moving image. Here, the relationship between the position xb in the traveling direction on the projection plane and the horizontal position Xb on the front moving image is as shown in Expression (7). Also, the position yb in the height direction on the projection plane
And the vertical position Yb on the front moving image is as shown in Expression (8).

[0165]

(Equation 4)

Note that in equations (7) and (8), “θg”
Indicates an angle obtained by halving the horizontal angle of view of the front moving image. When the horizontal angle of view of the camera that has captured the front moving image is clear, the angle of half of this horizontal angle of view is referred to as “θg”. If the horizontal angle of view of the camera is not clear, a value set in advance according to the focal length of the used lens is used. “D” is a value obtained by halving the horizontal length of the front moving image (the horizontal length of the image frame).

By using the equations (7) and (8), the geometric transformation unit 71 can generate the image signals of the left and right projected images on a frame basis based on the image signals of each frame of the front moving image. In addition, when the input image is a front moving image of an in-vehicle forward system, the left and right projected images are side images,
The motion amount of the projected image is proportional to the vehicle speed. The amount of movement of the projected image is inversely proportional to the depth. Therefore, when the projected image is a distant view image and has a large depth, the motion amount is small. Also, when the projected image is a close-up image and the depth is small, the amount of motion increases. The size of the projected image only needs to be large enough for each subsequent process. For example, the size is preferably the same as that of the peripheral image to be presented.

FIG. 37 is a diagram for explaining the operation of the motion discrimination image generation unit 73. Motion determination image generation unit 73
As shown in FIG. 37A, the image signal has a width of a predetermined number of pixels wa (for example, 4 pixels) in the horizontal direction from the front moving image side end (the right end in the right projection plane image) of the projection plane image for each frame. And the extracted image signal is shown in FIG.
As shown in FIG. 7B, image synthesis is performed by overlapping by a predetermined frame in the frame order, and a motion discrimination image signal SMD of the motion discrimination image Up is generated. The motion discrimination image U
p is when the amount of movement of the subject is wa pixels / frame in the horizontal direction and is equal to the number wa of cutout pixels of the image signal (FIG. 3).
7, the image of the subject is a continuous image having no discontinuity. An image of a subject (mountain in FIG. 37) whose movement is slower than the extraction of the image signal has a portion that is repeatedly extracted, so that a discontinuity of the image occurs at a connection portion of the extracted image signals. In addition, in an image of a subject (a tree in FIG. 37) that moves faster than the clipping of the image signal, a portion that is not clipped occurs because the movement is fast, and discontinuity of the image occurs at a joint portion.

The integral image generating section 74 integrates the integral image signal SA used to determine the motion amount of each pixel of the motion discrimination image.
Generate Here, the principle of the motion amount determination operation will be described with reference to FIG. For example, if the subject Ja moving rightward and the subject Jb moving rightward at twice the speed of the subject Ja are photographed, the photographed image is as shown in FIG. 38A. Here, the image of the next frame F2 is moved from the first frame F1 in the direction opposite to the moving direction of the subject Ja.
Shift by the pixel amount dx of a. Similarly, the image of the next frame F3 is shifted relative to the frame F2 by the pixel amount dx of the subject Ja in the direction opposite to the moving direction of the subject Ja. As described above, when the average value image Um obtained by adding and averaging the images of the respective frames shifted by a fixed amount between the respective frames is used, the position of the subject Jb is constant and the position of the subject Jb is moved. The signal level of the image signal of the subject Jb is smaller than that of the subject Ja,
As shown in FIG. 38C, the subject Ja is represented by the average value image Um.
Only can be extracted.

As shown in FIG. 38D, the shift amount of the image of each frame is doubled (2dx) and the average value image U
When m is generated, the position of the subject Jb becomes constant and the position of the subject Ja moves, so that only the subject Jb can be extracted from the average image Um as shown in FIG. 38E.

As described above, the average value image obtained by moving the image of each frame by a predetermined amount, adding and averaging is generated, and a plurality of images are obtained by changing the shift amount of the image of each frame. Generate an average image. Then, by determining which average value image shows the subject correctly, even if one image includes subjects having different amounts of motion, the amount of motion of each subject can be determined using the average value image. . Here, similarly to the motion discriminating image generating unit 73, the integrated image generating unit 74 cuts out an image signal of a predetermined number of pixels in the horizontal direction for each frame based on the plurality of average images, and generates an integrated image. The integrated image and the motion discrimination image are compared to determine which integrated image has an image equal to the subject constituting the motion discrimination image, and the motion amount at a predetermined position in each frame is determined.

FIG. 39 is a diagram showing the configuration of the integral image generating section 74. The projection image signal SDP supplied from the geometric transformation unit 71 is added to the addition processing units 741-1 to 741-1 of the integral image generation unit 74.
741-ns.

The addition processing unit 741-1 adds the projecting plane image signal, which has been shifted in the horizontal direction from the previous frame by a predetermined number of pixels SH-1, for the number of ms frames (for example, 10 frames) and adds the added image. The signals are calculated sequentially. Further, the addition processing unit 741-1 multiplies the added signal by 1 / m and supplies the image signal of the average image to the area selection unit 742-1.

The area selecting section 742-1 outputs a predetermined number of pixels wb in the horizontal direction from the front moving image side end (the right end in the right projection plane image) for each frame from the supplied average value image signal. The image signal is cut out, and the cut-out image signal is sequentially superimposed and output to the motion determining unit 75 as an integrated image signal SA-1. This area selection unit 742
The integrated image signal SA-1 output from -1 is an image signal indicating a subject having a motion amount of a predetermined number of pixels SH-1.

Similarly to addition processing section 741-1, addition processing sections 741-2 to 741-ns also have a preset number of pixels SH-2 to
A projection plane image signal shifted in the horizontal direction from the previous frame by SH-ns is added for ms frames, an addition signal is sequentially calculated, the added signal is multiplied by 1 / m, averaged, and area selection is performed. To the units 742-2 to 742-ns. Further, the region selecting units 742-2 to 742-ns cut out signals of the predetermined number of pixels wb from the supplied image signals, superimpose them sequentially, and output the resultant to the motion determining unit 75 as an integrated image signal. The integrated image signals SA-2 to SA-ns output from the area selection units 742-2 to 742-ns are images indicating the subject having the motion amount of the predetermined number of pixels SH-2 to SH-3, respectively. Signal.

Here, assuming that the number of pixels to be cut out by the area discriminating section 742 of the motion discriminating image generating section 73 and that of the integral image generating section 74 are equal (wa = wb), the matching of the position of the subject between the motion discriminating image and the integral image is performed. Sex can be taken.

FIG. 40 is a diagram showing the operation of the integral image generating section 74. When the image shift amount is the number of pixels SH-r, the integrated image generation unit 74 moves and adds the image by the number of pixels SH-r (<wb) for each frame as shown in FIG. 40A. At this time, the added image shows a mountain image as shown in FIG. 40B. When the image shift amount is set to the number of pixels SH-s (= wb) larger than the number of pixels SH-r, as shown in FIG.
Move by H-s and add. At this time, the added image indicates an image of a house having a larger motion amount than the mountain as shown in FIG. 40D. Further, the image shift amount is represented by the number of pixels S.
When the number of pixels SH-t (> wb) is larger than H-s, the image is moved and added by the number of pixels SH-t for each frame as shown in FIG. 40E. At this time, the added image shows an image of a house having a larger motion amount than the mountain as shown in FIG. 40F.

Here, the integral image generating section 74 starts clipping based on the frame Fa that was first clipped when the motion discriminating image generating section 73 generates the motion discriminating image, and is equal to the motion discriminating image. Cutout amount (wa
= Wb), an image is cut out by the same number of frames to generate an integral image. At this time, the integral image and the motion determination image have the same horizontal image length, and the phase of the subject matches between the integration image and the motion determination image.

In the motion discrimination image, the subject has a discontinuous shape when the cutout amount and the motion amount are not equal. Similarly, when the number of pixels SH-r is smaller than the number of pixels wb, the image becomes inconsistent. Is performed with overlap. For this reason, the integral image Ub-r becomes a discontinuous image according to the amount of motion as shown in FIG. 40G. When the number of pixels SH-s is equal to the number of pixels wb, the integrated image Ub-s is a continuous image according to the amount of motion as shown in FIG. 40H. Further, when the number of pixels SH-t is larger than the number of pixels wb, a missing portion is generated in the cutout of the image, and the integrated image Ub-t becomes a discontinuous image according to the motion amount as shown in FIG. 40J. For this reason, the motion determination unit 75 described below performs matching between the motion determination image and the integrated image, and determines the motion amount of the subject based on the number of pixels SH that is the image shift amount of the integrated image determined to be matched. .

The motion discriminating section 75 obtains a correlation value between each pixel of the motion discrimination image and the integrated image of each image shift amount, and based on the correlation value, each pixel of the motion discrimination image is equal to any of the integrated images. Then, the amount of motion of each pixel of the motion determination image is determined based on the determination result.

In the calculation of the correlation value, the pixels of the image A and the image B
When calculating the correlation value with, a predetermined range centering on the target pixel (pixel for calculating the correlation value) of the image A is set, and the correlation value of the target pixel is obtained using the image signal in the predetermined range. . For example, ± mvc pixels in the x direction with respect to the pixel of interest,
A rectangular range of ± nvc pixels (for example, a rectangular range of 31 horizontal pixels and 5 vertical pixels centering on the target pixel) is set in the y direction, and the correlation value VC is calculated based on Expression (9).

[0182]

(Equation 5)

In equation (9), the signal level of each pixel within the rectangular range of image A is represented by DAi (i = 1 to 1).
(mvc × nvc)), the signal level of each pixel within the rectangular range of the image B is set to DBi (i = 1 to (mvc × nvc)),
The average value of the signal level of each pixel within the rectangular range is D
Aav and DBav.

For example, when the motion judging section 75 judges that the image of the mountain in the motion image shown in FIG. 41A matches the integral image shown in FIG. 41B based on the equation (9), the image of this mountain is judged. It is determined that the motion amount of each of the constituent pixels is a motion amount corresponding to the pixel number SH-r. The image of the house in the motion determination image shown in FIG.
When it is determined that the image matches the integral image shown in FIG.
It is determined that the motion amount corresponds to H-s. Further, FIG.
When it is determined that the tree image in the motion discrimination image shown in FIG. 1A matches the integral image shown in FIG. 41D, the motion amount of each pixel forming the tree image is a motion amount corresponding to the number of pixels SH-t. It is determined that there is. As described above, the motion determining unit 75 determines the amount of motion for each pixel, generates motion amount information MF indicating the result of the determination, and supplies the generated information to the layer classifying unit 81 and the information generating unit 82.

Although not shown, the integral image generating section 74
Indicates the direction in which the image is shifted, as shown in FIGS. 41B, 41C, and 41D.
(For example, the image shift amount S
By setting H to 48 pixels to -24 pixels), an image of a passing vehicle or the like moving in the opposite direction to an image of a tree, a house, or a mountain can be extracted. Further, in order to determine the amount of movement of an image whose movement direction is opposite to that of the distant view layer or the like, as shown in FIG. 42A, an image is cut out from the image of each frame, and the images are horizontally inverted and then superimposed. FIG. 4
The motion discrimination image Up shown in FIG. 2B is generated. In this case, in the motion discrimination image Up shown in FIG. 42B, an image in which the motion direction is reverse, such as a passing vehicle, is correctly displayed in the same direction as the distant, middle, and near view layers. FIG. 42C shows a motion discrimination image Up in a case where the images are superimposed without performing left-right inversion. As described above, the image signal of the integral image in which the image is shifted in the reverse direction and the image signal SMD of the motion discrimination image generated by inverting and superimposing the cut out image are sequentially compared from the reference frame. By doing so, the amount of movement of the subject moving in the opposite direction can be determined.

The layer classifying section 81 layers each pixel of the motion discrimination image in the depth direction based on the motion amount information MF. For example, assuming that the depth is previously layered into four layers of three layers of distant, middle, and near views and a layer having movement in the opposite direction,
Layer classification is performed on which layer each pixel belongs to.

The layer classification unit 81 sets a threshold value in the same manner as the above-described threshold value setting unit 464, and compares this threshold value with the amount of motion of each pixel to determine which layer each pixel belongs to. The layer classification information L corresponding to each pixel
Generate B. The generated layer classification information LB is supplied to the information generation unit 82.

FIG. 43 shows the structure of the information generating unit 82. The motion amount average value calculation unit 821 of the information generation unit 82
The motion amount information MF from the motion determining unit 75 and the layer classifying unit 8
On the basis of the layer classification information LB supplied from No. 1, the average value of the amount of motion of the preceding and succeeding nz frames is obtained for each frame of interest in each layer, and is set as the speed of the frame of interest in that layer. The calculated motion amount MYs for each frame of each layer is supplied to the intermediate image signal generation unit 822 and stored in the intermediate image information storage area 52.

The intermediate image signal generator 822 extracts a signal for each layer from the image signal SDP of the projected image according to the motion amount MYs calculated by the motion average value calculator 821 based on the layer classification information LB. , Generates an intermediate image signal GFs for each layer. This intermediate image signal GFs is supplied to the intermediate image signal interpolation unit 823. At this time, since the motion amount of the distant view layer is small and the motion amount of the foreground layer is large, the intermediate image of the distant view layer is short in the horizontal direction and long in the intermediate image of the foreground layer, as in FIG. In addition, since the movement is in the reverse direction in the backward layer, the intermediate image signal generation unit 822 reads the image from the side moving image by the absolute value of the amount of movement and inverts left and right to superimpose, so that the intermediate layer in the backward layer Create an image.

Similarly to the intermediate image signal interpolating unit 473, the intermediate image signal interpolating unit 823 performs image synthesis by superimposing images of layers having different motion amounts MYs in the order of distant view, middle view, and near view. The intermediate image signal GFs is used so that an imageless area does not occur regardless of the difference in the amount of motion.
, And the corrected signal is stored in the intermediate image information storage area 52 as the intermediate image signal GYs.

Further, when there is a backward layer such as a passing vehicle, the intermediate image signal interpolation unit 823 does not perform the interpolation processing because this backward layer is a layer before the foreground layer. When a horizontal movement layer indicating a building or the like appearing in a distant view when turning left or right is provided, interpolation processing is performed in the same manner as in the distant view layer. It should be noted that the subject of the backward layer or the horizontally moving layer is determined by determining the motion pattern based on the motion direction of the right-projected image and the left-projected image in the same manner as in FIG. Can be determined based on the For example, when the movement of the upper part of the screen of the left image and the upper part of the screen of the right image indicate different directions, it is possible to determine that the vehicle is turning right or left. In addition, the movement of the upper part of the screen of the left image and the movement of the upper part of the screen of the right image indicate the same direction.

Next, an operation of generating a peripheral image using the intermediate image information of each layer will be described. This peripheral image can be generated in the same manner as the above-described peripheral image signal generation block 60. That is, the image of each layer is read by the amount of the motion, and the images are synthesized by superimposing the distant view layer, the middle view layer, and the close view layer in this order. Also, when the horizontal movement layer and the backward layer are set, the peripheral image signal can be generated in the same manner as in the peripheral image signal generation block 60.

In the image synthesis, for example, when a right-side moving image is generated as a peripheral image, the same frame portion of each layer is overlapped so as to overlap at the left end of the right-side moving image to be generated. Do. Also, the right end of the projected image obtained by first converting the front moving image and the left end of the generated side moving image are made equal. afterwards,
For each layer, an image is moved based on the amount of motion for each layer, and the next image is superimposed on the moved image for each layer to perform image synthesis. As shown in FIG. Can be generated.

By the way, the image generated by superimposing the intermediate images for each layer using the intermediate image information is an image on the plane equal to the projection plane FI as shown in FIG. On the other hand, the screen 10R on which the peripheral image is displayed is the projection plane FI.
Is installed with an inclination. For this reason, when the images of the respective layers are superimposed on each other and the images are synthesized and displayed on the screen 10R, the image conversion processing is performed so that the images move in a correct shape.

Here, as shown in FIG. 45, a half of the horizontal length of the front moving image UC is defined as a distance d, and the position O, which is the photographing position of the front moving image, is positioned from the center of the screen 10R. Consider a case where the screen 10R is installed at an angle so as to be on the normal line. In this case, the image signal at the position (Xc, Yc) on the screen 10R is calculated based on the position (xc, Yc) on the image URI before conversion calculated based on the equations (10) and (11).
By extracting the image signal yc), the peripheral image signal SDR of the right-side moving image after the projective transformation can be easily generated. Also,
By displaying the right-side moving image on the screen 10R using the peripheral image signal SDR, it is possible to display the right-side moving image with high sense of realism.

[0196]

(Equation 6)

It is needless to say that the peripheral image signal SDL can be generated in the same manner as the peripheral image signal SDR.

Further, the processing performed in each block described above may be realized not only by hardware but also by software. In this case, the configuration shown in FIG. 15 is used.

FIG. 46 is a flowchart showing the overall configuration of the image processing program according to the third embodiment. In the figure, in step ST91, a motion detection process is performed to generate a projection plane image, and the amount of movement of each pixel of the projection plane image is detected. Next, in step ST92, pixels are divided into layers based on the amount of motion, and intermediate image information indicating the amount of motion of each layer is generated. In step ST93, a peripheral image generation process is performed, and a peripheral image is synthesized based on the intermediate image information.

FIG. 47 is a flowchart showing the motion detection processing in step ST91 of FIG. Step ST
In 101, a projected image of a side surface is created based on a front moving image by geometric transformation. In step ST102, an integrated image for each motion amount is created based on the projected image,
The cut-out image is created by cutting out the image from the projected image. In step ST103, matching between the integrated image and the cut-out image for each motion amount is performed. In step ST104, the amount of motion for each pixel is determined by detecting the integrated image having the highest correlation value based on the result of the matching.

FIG. 48 is a flowchart showing the intermediate image information generation processing of step ST92 in FIG. In step ST111, the statistic of the motion amount is calculated based on the motion amount of the pixel detected in the motion detection process in step ST101. In step ST112, a threshold for performing layer division based on the statistics is set. Step ST
In 113, the set threshold value and the amount of motion are compared to generate layer classification information indicating to which layer each pixel of the projected image belongs. In step ST114, the motion amount for each layer is calculated based on the motion amount of the pixels included in the layer. In step ST115, based on the layer classification information and the amount of motion for each layer, an image is extracted for each layer from the projected image to generate an intermediate image. In step ST116, it is determined whether or not there is an isolated region in the intermediate image. If there is an isolated region, the process proceeds to step ST117. If there is no isolated region, the process proceeds to step ST118. Step S
In T117, an interpolation process is performed to remove an isolated region, and the process proceeds to Step ST118. In step ST118, an intermediate image for each layer without an isolated region and the amount of motion for each layer are generated as intermediate image information.

FIG. 49 is a flowchart showing the peripheral image generation processing in step ST93 of FIG. In step ST121, an image is extracted for each layer using the intermediate image for each layer based on the motion amount for each layer indicated by the intermediate image information. In step ST122, the image of each extracted layer is synthesized at a predetermined position. In step ST123, a portion without an image is generated when the images are combined, and it is determined whether or not an interpolation process for supplementing the image in this portion is necessary. Here, when there is a portion having no image, the process proceeds to step ST124, and when there is no portion having no image, the process proceeds to step ST125. In step ST124, interpolation processing is performed, and step ST125 is performed.
Proceed to. For example, an image around a portion having no image is expanded and interpolated. In step ST125, since the synthesized image is an image on the same plane as the projection plane, an image conversion process is performed in accordance with the direction of the screen on which the image is projected. Output as an image signal of the image.

[0203] Such an image processing program may be recorded on a removable information recording transmission medium as described above, or transmitted via an information recording transmission medium.

In the third embodiment described above,
When the image processing program is executed to generate the intermediate image information, the intermediate image information
Disk drive 304 or recording medium drive 3
12 is recorded on a recording medium attached to the recording medium 12. Further, when the output of the image signal of the peripheral image is requested, the intermediate image information is read out and the peripheral image signal SD of the left and right side moving image is read.
L and SDR are generated, and the peripheral image signal SD
L and SDR are output in synchronization with the image signal SDC of the front moving image. For this reason, a continuous image with a wide angle of view can be displayed using the front and left and right screens.

In each of the above-described embodiments, the case where an image signal of a peripheral image is generated using an image signal obtained by a vehicle-mounted camera has been described. An image signal of a peripheral image can be generated by performing similar processing even when an image or an image signal of a moving image captured while a person is walking is used.

As described above, according to the above-described first to third embodiments, the movement of the front moving image can be determined, and a peripheral image in a different direction can be generated based on the determined movement. For this reason, since moving images in different directions can be generated in a simultaneous sequence based on a front moving image captured by a video camera or the like, a multi-screen image with a high sense of reality and a wide angle of view can be presented.

Further, since an image in a different direction can be generated based on one input image, it is not necessary to use a plurality of video cameras or cameras using a wide-angle lens, and the photographing can be easily performed.

Further, since an image in a different direction is generated by using a real image, an image that is more realistic and more realistic than an image in a three-dimensional virtual space by computer graphics can be displayed, and the depth can be increased. Is expressed as a two-dimensional plane hierarchy, three-dimensional arithmetic processing is not required, and signal processing is easy.

Further, the image processing apparatus and the computer use the signal stored as the input image signal,
An image that enters the front moving image with the passage of time can be displayed as a peripheral image before being displayed as the front moving image. As described above, since an image that cannot be displayed when a real-time image signal is used can be displayed as a peripheral image, a more realistic and realistic image display with a wide angle of view can be performed. Also, by inputting a huge image source that already exists into a computer that executes the image processing apparatus and the image signal generation method of the present invention,
These images can be enjoyed with high realism and a wide angle of view.

In the above-described embodiment, a plurality of motion detection areas are provided at the side end of the front moving image to obtain a motion vector for each motion detection area, or a motion discrimination image is generated using a projected image. By comparing with the integral image, the motion of the predetermined position of the frame of interest is determined to generate intermediate image information, and the peripheral image signal is generated using the intermediate image information. It is needless to say that the motion at the predetermined position can be determined.

[0211]

According to the present invention, the motion of a predetermined area in an image based on the input image signal is detected using the input image signal of the plurality of frames, and the detected motion and the input image signal of the plurality of frames are detected. Based on this, an image signal of a peripheral moving image having the same time as the target frame and having a different angle of view from the image of the target frame is generated. Therefore, an image having a wide display range and a sense of depth can be obtained at low cost without using a large number of cameras or special cameras.

Further, since a plurality of motion detection areas are set in the image end area and motion is detected for each motion detection area, the reference moving image based on the input image signal includes a subject such as a distant view or a near view. , The movement of each subject can be detected. Further, when the constraint condition that the motion of the image is in the radial direction about the predetermined position is satisfied, the motion is easily calculated by calculating an error value from the image of the previous screen while zooming the image around the predetermined position. Can be determined.

Further, based on the detection result of the motion, the image end region is divided into layers, and signal processing is performed according to the screen order for each layer. Therefore, the reference moving image includes a subject such as a distant view or a near view. However, it is possible to generate an image signal of a correct peripheral moving image with a sense of depth by performing signal processing according to each subject.

Further, when a gap region occurs when the image of each layer is pasted, the image signal of the image of the gap portion is generated by the interpolation processing, so that a good image signal of the side moving image can be generated. Further, since the signal conversion process is performed according to the direction of the projection plane of the peripheral moving image, a peripheral moving image having a sense of depth can be displayed realistically.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a display system.

FIG. 2 is a diagram showing an image projection state.

FIG. 3 is a diagram illustrating a configuration of an image processing apparatus according to the first embodiment.

FIG. 4 is a diagram for explaining setting of a motion detection area.

FIG. 5 is a diagram for explaining a method of calculating a motion amount.

FIG. 6 is a diagram illustrating a change in an area size during zooming.

FIG. 7 is a diagram illustrating a configuration of a motion amount calculation block.

FIG. 8 is a diagram illustrating a motion amount of each motion detection area.

FIG. 9 is a diagram showing a configuration of a layering processing block.

FIG. 10 is a diagram for explaining setting of a threshold.

FIG. 11 is a diagram illustrating a configuration of a peripheral image signal generation block.

FIG. 12 is a diagram illustrating an operation of generating a right side moving image.

FIG. 13 is a diagram for explaining generation of a gap area and interpolation processing.

FIG. 14 is a diagram illustrating an image conversion process.

FIG. 15 is a diagram showing a configuration using a computer.

FIG. 16 is a flowchart illustrating an overall configuration of an image processing program according to the first embodiment.

FIG. 17 is a flowchart illustrating motion detection according to the first embodiment.

FIG. 18 is a flowchart illustrating a layering process according to the first embodiment.

FIG. 19 is a flowchart illustrating generation of a peripheral image signal according to the first embodiment.

FIG. 20 is a diagram illustrating a configuration of an image processing apparatus according to a second embodiment.

FIG. 21 is a diagram showing a motion vector of each motion detection area.

FIG. 22 is a diagram illustrating a method of determining an image reference position.

FIG. 23 is a diagram illustrating a configuration of a motion detection unit.

FIG. 24 is a diagram illustrating a configuration of a layer classification unit.

FIG. 25 is a diagram for explaining a relationship between a vector direction and a motion pattern.

FIG. 26 is a diagram illustrating an operation of generating layer classification information.

FIG. 27 is a diagram illustrating a configuration of an information generation unit.

FIG. 28 is a diagram showing an intermediate image for each layer.

FIG. 29 is a diagram showing an intermediate image for each layer after the interpolation processing.

FIG. 30 is a diagram illustrating a configuration of a peripheral image signal generation block.

FIG. 31 is a flowchart illustrating an overall configuration of an image processing program according to the second embodiment.

FIG. 32 is a flowchart illustrating motion detection according to the second embodiment.

FIG. 33 is a flowchart illustrating a layering process according to the second embodiment.

FIG. 34 is a flowchart showing peripheral image signal generation according to the second embodiment.

FIG. 35 is a diagram illustrating a configuration of an image processing apparatus according to a third embodiment.

FIG. 36 is a diagram for describing a geometric conversion process.

FIG. 37 is a diagram for explaining the operation of the motion determination image generation unit.

FIG. 38 is a diagram for explaining the principle of the motion amount determination operation.

FIG. 39 is a diagram illustrating a configuration of an integral image generation unit.

FIG. 40 is a diagram for explaining the operation of the integral image generation unit.

FIG. 41 is a diagram illustrating the operation of a motion amount determination unit.

FIG. 42 is a diagram for describing an operation of generating a motion discrimination image of a backward layer.

FIG. 43 is a diagram illustrating a configuration of an information generation unit.

FIG. 44 is a diagram showing a projection plane image and a generated side moving image.

FIG. 45 is a diagram for describing image conversion processing.

FIG. 46 is a flowchart illustrating an overall configuration of an image processing program according to the third embodiment.

FIG. 47 is a flowchart illustrating a motion detection process according to the third embodiment.

FIG. 48 is a flowchart illustrating a layering process according to the third embodiment.

FIG. 49 is a flowchart illustrating peripheral image signal generation according to the third embodiment.

[Explanation of symbols]

10L, 10C, 10R ... screen, 12L, 12
C, 12R ... projector, 15A, 15B, 15C
... Image processing devices, 16, 26, 41, 224, and 62
2-1 to 622-5: delay unit, 20, 40: motion amount calculation block, 22, 45, 80: layer division processing block, 24, 60, 72 ... peripheral image signal generation block , 30 ... computer, 42 ... movement detector, 44 ... image signal generator, 46 ... layer classifier, 47 ... information generator, 50 ... accumulator, 51 ...
..Front moving image signal storage area, 52 ... intermediate image information storage area, 61 ... image generation control section, 62 ... signal extraction section, 63 ... projection conversion section, 64 ... image signal Synthesizing unit, 71: Geometric conversion unit, 73: Discrimination image generation unit, 74: Integration image generation unit, 75: Motion judgment unit, 81: Layer classification unit, 82: Information Generator,
201: size changing unit, 202, 422: error sum calculating unit, 203, 423: comparing unit, 204, 42
4 data storage unit, 205 enlargement ratio setting unit, 2
21: statistical information calculation unit, 222: layer division threshold setting unit, 223: layer determination unit, 225: layer movement amount calculation unit, 241: signal extraction unit, 242
243, 244 addition section, 245 image shift section, 246 interpolation section, 247 middle view pasting section, 248 near view pasting section, 249 image conversion section , 421: size conversion unit, 425: search control unit, 461: pattern determination unit, 462: layer determination unit, 463: classification unit, 464: threshold setting unit, 465 ··· Classification correction unit, 471,821 ··· Motion amount average value calculation unit, 472,822 ··· Intermediate image signal generation unit, 473,823 ··· Intermediate image signal interpolation unit, 6
21-1 to 621-5 ... Synthesis unit, 623-1 to 623-5
..Image shift section, 631-634... Conversion section, 64
1 ... distant view synthesis section, 642 ... middle view synthesis section, 643
... foreground composition unit, 644 ... reduced image composition unit, 741
-1 to 741-ns ... Addition processing unit, 742-1 to 742-ns
... Area selection unit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Tetsushi Kokubo 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Daisuke Kikuchi 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F term (reference) 5C023 AA10 AA16 AA38 BA04 BA11 CA03 EA03 EA05 5C054 AA01 AA05 CA04 CC02 EA05 FC01 FC04 FC13 FE17 HA17 HA26 5L096 AA06 CA02 DA01 EA03 GA08 HA04

Claims (40)

[Claims] 1. A motion detecting means for detecting a motion of a predetermined area in an image formed on a specific plane using an input image signal of a plurality of frames, and based on the motion detected by the motion detecting means, Layering processing means for classifying a predetermined area to generate information indicating a classification result, based on the classification information and the input image signals of the plurality of frames, and at the same time as the frame of interest in the input image signal, An image processing apparatus comprising: a peripheral image signal generating unit configured to generate an image signal of a peripheral image having a different angle of view from the image of the frame of interest. 2. The image processing apparatus according to claim 1, wherein the image based on the input image signal and the image formed on the specific plane are images on the same plane. 3. A motion of an image edge region in a screen of the reference moving image for each screen of the reference moving image by using an image signal of a reference moving image that generates a motion satisfying at least one constraint condition. A layering processing unit that determines a layer of the predetermined area based on the motion detected by the motion detecting unit and generates layer information; Using the layer information and the image signal of the image end area, performing signal processing according to the screen order, and at the same time as the reference moving image, and the image signal of the peripheral moving image corresponding to the reference moving image An image processing apparatus comprising: a peripheral image generating unit configured to generate a peripheral image. 4. The image processing apparatus according to claim 3, wherein the motion detection means sets a plurality of motion detection areas in the image end area and detects a motion for each of the motion detection areas. . 5. The image processing apparatus according to claim 4, wherein the one constraint condition is that the movement of the image is a radial direction centered on a predetermined position. 6. The motion detection means calculates an error value with respect to an image of a previous screen while zooming the image of the motion detection area around the predetermined position, and performs a motion based on the calculated error value. 6. The method according to claim 5, wherein
The image processing apparatus according to any one of the preceding claims. 7. The peripheral image generating means performs signal processing in accordance with a screen order for each layer, based on a detection result of the motion detecting means, wherein the edge area of the image is divided into layers. The image processing apparatus according to claim 3, wherein the image processing apparatus generates an image signal of the image. 8. The peripheral image generating means sets a moving amount for each layer based on a detection result of the motion detecting means, and converts an image of the layered image end area into a corresponding layer. 8. The image processing apparatus according to claim 7, wherein signal processing is performed to extract the image according to the amount of movement and to paste the extracted image on a layer basis. 9. The image according to claim 8, wherein the peripheral image generating means performs the pasting in order from the layer having a small moving amount to the layer having a large moving amount, as the pasting process for each layer. Processing equipment. 10. The peripheral image generating means moves an image of each layer that has been pasted according to a moving amount set for each layer, and adds the extracted image to the moved image. 9. The image processing apparatus according to claim 8, wherein, after the addition, the process of pasting in units of layers is performed in a screen order. 11. The image processing apparatus according to claim 1, wherein the peripheral image generating means generates an image signal of the image of the gap portion by an interpolation process when a gap portion is generated in the image by the pasting process in the layer unit. 9. The image processing apparatus according to 8. 12. The peripheral image generating means, as the interpolation processing, generates an image signal of an image of the gap area by extending an image located behind the gap portion in a layer moving direction. The image processing apparatus according to claim 11, wherein: 13. The peripheral image generating means converts a signal obtained by performing signal processing for pasting the extracted image on a layer basis according to a direction of a projection plane of the peripheral moving image. The image processing apparatus according to claim 8, wherein the processing is performed. 14. A motion of a predetermined area in an image formed on a specific plane using input image signals of a plurality of frames is detected, and the predetermined area is classified based on the detected motion. Based on the classification information and the input image signal of the plurality of frames, information indicating the classification result is generated, and at the same time as the frame of interest in the input image signal, and the image of the frame of interest and the angle of view An image signal generation method characterized by generating image signals of different peripheral images. 15. The image signal generating method according to claim 14, wherein the image based on the input image signal and the image formed on the specific plane are images on the same plane. 16. Using a reference moving image that generates a movement satisfying at least one constraint condition, a motion of an image end region is detected for each screen of the reference moving image by using the constraint condition; Based on the detected motion, signal processing is performed according to the screen order while using the image signal of the image end area, and at the same time as the reference moving image, and of the peripheral moving image corresponding to the reference moving image. An image signal generation method characterized by generating an image signal. 17. The image signal generation method according to claim 16, wherein a plurality of motion detection areas are set in the image end area, and motion is detected for each of the motion detection areas. 18. The image signal generating method according to claim 17, wherein the one constraint condition is that the movement of the image is a radial direction centering on a predetermined position. 19. An image processing apparatus according to claim 19, further comprising: calculating an error value with respect to an image of a previous screen while zooming the image of the motion detection area around the predetermined position, and determining a motion based on the calculated error value. 19. The image signal generating method according to claim 18, wherein 20. A method for generating an image signal of a peripheral moving image by performing signal processing in accordance with a screen order for each layer, wherein the image end region is divided into layers based on the motion detection result. 17. The image signal generating method according to claim 16, wherein: 21. A moving amount is set for each of the layers based on the detection result of the motion, and an image of the layered image end region is extracted according to the moving amount of a corresponding layer. 21. The image signal generation method according to claim 20, wherein signal processing for pasting the extracted image in layers is performed. 22. The image signal generation method according to claim 21, wherein, in the layer-by-layer pasting processing, pasting is performed in order from a layer having a small moving amount to a layer having a large moving amount. 23. The image of each layer that has already been pasted is moved in accordance with the amount of movement set for each layer, and the extracted image is added to the image after the movement, and then, in units of layers. 22. The image signal generation method according to claim 21, wherein the signal processing for pasting is performed in the order of screens. 24. The image signal generation method according to claim 21, wherein when a gap portion is generated in the image by the layer-by-layer pasting process, an image signal of the image of the gap portion is generated by an interpolation process. Method. 25. The image processing apparatus according to claim 24, wherein, as said interpolation processing, an image signal of an image of said gap region is generated by extending an image located on the back side of said gap portion in a moving direction of a layer. Image signal generation method. 26. A signal conversion process according to a direction of a projection plane of the peripheral moving image is performed on a signal obtained by performing a signal process of pasting the extracted image on a layer basis. The image signal generation method according to claim 21, wherein 27. A computer, comprising: a process for detecting a motion of a predetermined region in an image formed on a specific plane using input image signals of a plurality of frames; and a process for detecting the predetermined region based on the detected motion. Classifying and generating information indicating a classification result, based on the classification information and the input image signals of the plurality of frames, at the same time as the frame of interest in the input image signal, and the image of the frame of interest Generating an image signal of a peripheral image having a different angle of view. 28. A computer, comprising: detecting a motion of an image edge region for each screen of the reference moving image by using the reference moving image which generates a motion satisfying at least one constraint condition and utilizing the constraint condition; A first process, based on the detected motion, using the image signal of the image edge region and performing signal processing according to the screen order, and at the same time as the reference moving image, And a second process for generating an image signal of a corresponding peripheral moving image. 29. The first processing includes a first step of setting a plurality of motion detection areas in the image end area, and a second step of detecting a motion for each of the motion detection areas. 29. The image processing program according to claim 28, wherein: 30. The image processing program according to claim 29, wherein in the first processing, the one constraint condition is that the movement of the image is in a radial direction about a predetermined position. . 31. A second step of calculating an error value from an image of a previous screen while zooming the image of the motion detection area around the predetermined position, and calculating the calculated error. 4th to determine movement based on value
31. The image processing program according to claim 30, further comprising the steps of: 32. The second processing includes a fifth step of dividing the image end region into layers based on a detection result in the first processing, and a signal processing according to a screen order for each layer. Performing a step of generating an image signal of a peripheral moving image.
Image processing program as described. 33. The sixth step, wherein a moving amount is set for each of the layers based on the detection result of the movement.
A step of extracting an image of the layered image end region according to the amount of movement of a corresponding layer, and an image extracted in the eighth step in units of layers. 33. The image processing program according to claim 32, further comprising a ninth step of pasting. 34. The image processing program according to claim 33, wherein in the ninth step, images are pasted in order from a layer having a small moving amount to a layer having a large moving amount. 35. The sixth step includes a tenth step of moving an image of each layer that has already been pasted according to the amount of movement of each layer set in the seventh step. 34. The computer-readable storage medium according to claim 33, wherein in the ninth step, a process of adding the extracted image to the image after the movement and pasting the image in units of layers is performed in screen order. 36. In the sixth step, when a gap is formed in the image pasted in the layer unit in the ninth step, an image signal of the image of the gap is interpolated by interpolation processing. The image processing program according to claim 34, further comprising an eleventh step of generating. 37. In the eleventh step, an image signal of an image of the gap area is generated by extending an image located on the rear side of the gap portion in a moving direction of a layer as an interpolation process. The image processing program according to claim 36, wherein 38. A signal conversion process according to a direction of a projection plane of the peripheral moving image on the image signal pasted on a layer basis in the ninth step. The image processing program according to claim 33, further comprising a twelfth step of performing. 39. A computer, comprising: a process for detecting a motion of a predetermined area in an image formed on a specific plane using input image signals of a plurality of frames; and detecting the predetermined area based on the detected motion. A process of classifying and generating information indicating a classification result, based on the classification information and the input image signals of the plurality of frames, at the same time as the frame of interest in the input image signal, and an image of the frame of interest. An information recording medium, wherein a program for executing a process of generating image signals of peripheral images having different angles of view is recorded in a computer-readable manner. 40. A computer detects a movement of an image end area for each screen of the reference moving image by using a reference moving image which generates a movement satisfying at least one constraint condition and utilizing the constraint condition. A first process, based on the detected motion, using the image signal of the image edge region and performing signal processing according to the screen order, and at the same time as the reference moving image, An information recording medium, wherein a program for executing a process including a second process of generating an image signal of a corresponding peripheral moving image is recorded in a computer-readable manner.