JPH10255071A - Image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten time required for finding out an optical flow and to improve accuracy by acquiring the moving image of a road, generating a three- dimensional(3D) model, extracting information to be required in displaying a 3D moving image from the generated 3D model, transmitting the extracted information, and displaying the transmitted information on a remote image processor. SOLUTION: A 3D model information acquiring device 2 comprises a frame memory 21, an optical flow generation part 22, a depth information calculation part 23, and a 3D model generation part 24. In the device 2, the generation part 22 executes optical flow generation processing for generating the optical flow of each frame or plural frames of a moving image. The calculation part 23 executes depth information calculation processing for calculating depth information based on the optical flow generated by the generation part 22 and the generation part 24 executes 3D model generation processing for a road based on the optical flow and the depth information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for obtaining three-dimensional image information from a two-dimensional moving image obtained from a video camera or a recorded video image, and transmitting the obtained three-dimensional information to a remote image processing apparatus. The present invention relates to a three-dimensional image generation system for transmitting and displaying a two-dimensional, three-dimensional moving image, or three-dimensional model, and more particularly to three-dimensional information from a two-dimensional road moving image obtained from a video camera or a recorded video image. The present invention relates to a technology that is effective when applied to a three-dimensional image generation system that generates a two-dimensional, three-dimensional moving image or three-dimensional model of a road by transmitting the obtained image data to a remote image processing device.

[0002]

2. Description of the Related Art Heretofore, as one for obtaining three-dimensional information from a two-dimensional image obtained from a video camera or a recorded video image, for example, one camera that moves a static scene at a constant speed has been used. There is a method called motion stereoscopic vision for estimating a depth distribution from an optical flow field obtained from a moving image. There is also a method called dynamic stereoscopic vision in which a dynamic scene is analyzed using two cameras.

[0003] In the transmission of three-dimensional information, a method is used in which a surface and the properties (texture) and solid of the surface are defined as a three-dimensional model in a specific coordinate system, and the viewpoint for viewing the three-dimensional model and the movement information of the viewpoint are transmitted. There is.

[0004]

In the above-mentioned conventional method, since the search is performed in all directions with respect to the search direction of the block for obtaining the correlation in order to obtain the optical flow, the time required for the search is long, and the three-dimensional search is performed. There is a problem that it takes time to obtain information.

In addition, since the apparent change in magnitude is not taken into account when calculating the correlation, there is a problem that the degree of the obtained correlation is low and the accuracy of the obtained optical flow is low.

Further, in transmission of a two-dimensional moving image,
Conventionally, the amount of information transmitted has been reduced by separating the still part and the moving part in the image and sending only the moving part.However, for an image where the entire screen is moving, It is not an effective method. Further, in the transmission of three-dimensional information, there is a problem that the transmission rate and the transmission time are increased because the data capacity is much larger than that of the two-dimensional information.

An object of the present invention is to provide a technique capable of reducing the time required for obtaining an optical flow and improving the accuracy.

Another object of the present invention is to provide a technique capable of reducing a transmission rate of information for displaying a moving image or a three-dimensional model.

[0009]

SUMMARY OF THE INVENTION Among the inventions disclosed in the present application, the outline of a representative one will be briefly described.
It is as follows.

[0010] An image processing system comprising three-dimensional moving image information acquiring means for acquiring three-dimensional moving image information based on a two-dimensional moving image acquired from a video camera or a video player, The moving image information obtaining means reads and stores the obtained two-dimensional moving image for each frame, and obtains a position of an infinity point of a motion of each frame stored in the frame memory, and obtains an infinity of the motion. For a frame group having the same point position, the direction radiating from the point at infinity of the movement is defined as the search direction of the same image portion, and a search block for searching the same image portion is moved in the search direction by a predetermined length. Then, the image of the search block is correlated between the respective frames, the portion having the highest correlation is detected as the same image portion, and the infinity point of the motion of the same image portion is detected. Optical flow generation means for generating an optical flow of the obtained two-dimensional moving image, and obtaining depth information of the two-dimensional moving image obtained from the generated optical flow. Depth information calculating means, type-recognizing the components of the acquired two-dimensional moving image, and arranging three-dimensional component models corresponding to the type-recognized components based on the optical flow and the depth information, respectively. And a three-dimensional model generating means for generating a three-dimensional model of the acquired two-dimensional moving image.

The optical flow generating means moves a search block for searching for the same image portion in a direction radially expanding from an infinity point of the movement by a predetermined length,
When correlating between frames, multiply the distance from the infinity point of motion in the current frame to be correlated to the search block by the reciprocal of the distance from the infinity point of motion in the target frame to be correlated to the search block. Obtain the proportionality constant obtained by combining them, enlarge the size of the image in the search block in the target frame to be correlated by the proportionality constant times, and respectively increase the image between the search block of the current frame and the previous frame. It has a means to take correlation.

Also, information for displaying a three-dimensional moving image is obtained from a three-dimensional model in which a plurality of three-dimensional part models are arranged and transmitted to a second image processing apparatus via a network. A first image processing apparatus having a three-dimensional video transmission unit, and a second image processor that outputs a three-dimensional video based on information of the three-dimensional video transmitted by the three-dimensional video transmission unit. An image processing system having an image processing device, wherein the three-dimensional moving image transmission unit acquires, from a second image processing device, viewpoint information indicating a position at which a three-dimensional moving image is output, and the viewpoint information Based on
Means for sequentially acquiring a two-dimensional projected image of the three-dimensional model, means for determining an infinite point of the motion of the two-dimensional projected image, and A means for obtaining vector information indicating an apparent moving direction and its magnitude; a means for comparing the obtained vector information with a vector of the original optical flow, correcting the vector to improve accuracy, and Means for transmitting information of the three-dimensional part model and its vector information, and information of the point at infinity of motion, the second image processing apparatus, the information of each of the three-dimensional part model and its vector information, and The apparatus includes means for constructing the three-dimensional model based on information on a point at infinity of motion, and means for displaying a two-dimensional projected image of the constructed three-dimensional model.

Further, information for displaying a three-dimensional moving image is obtained from a three-dimensional model in which a plurality of three-dimensional component models are arranged and transmitted to a second image processing device via a network. A first image processing apparatus having a three-dimensional video image transmitting means, and information on the three-dimensional video image transmitted by the three-dimensional video image transmitting means, having the plurality of three-dimensional part model information. And a second image processing device that outputs a three-dimensional moving image based on the three-dimensional moving image transmitting means, wherein the three-dimensional moving image transmitting unit transmits viewpoint information indicating a position at which the three-dimensional moving image is output to the second image processing device. Means for acquiring from the image processing device, means for sequentially extracting the three-dimensional part model present within the field of view of the viewpoint information, and means for recognizing the type of each extracted three-dimensional part model, Each of the extracted Means for acquiring arrangement information of a three-dimensional part model, means for transmitting type information of the recognized three-dimensional part model and arrangement information of the acquired three-dimensional part model, the second image processing apparatus Means for extracting a target three-dimensional part model based on the transmitted type information of each three-dimensional part model, and extracting the extracted three-dimensional part model based on the transmitted arrangement information. It comprises means for arranging and constructing a three-dimensional model, and means for displaying a two-dimensional projected image of the constructed three-dimensional model.

Further, information for displaying a three-dimensional moving image is obtained from a three-dimensional model in which a plurality of three-dimensional part models are arranged and transmitted to a second image processing device via a network. A first image processing apparatus having a three-dimensional moving image transmitting unit, and a plurality of parametric three-dimensional part model information, and a three-dimensional moving image transmitted by the three-dimensional moving image transmitting unit. And a second image processing device that outputs a three-dimensional video based on the information of the three-dimensional video, the three-dimensional video transmission means, the viewpoint information indicating the position to output the three-dimensional video, Means for acquiring from the second image processing device, means for sequentially extracting the three-dimensional component models present within the field of view of the viewpoint information, and the type and size of each of the extracted three-dimensional component models. Recognition Means for acquiring the arrangement information of each of the extracted three-dimensional part models, parameter information indicating the type and size of the recognized three-dimensional part model, and arrangement of the acquired three-dimensional part model Means for transmitting information and the second image processing apparatus, based on the transmitted type information of each of the three-dimensional component models, means for extracting a target three-dimensional component model, For the three-dimensional part model, based on the transmitted parameter information, based on the means for acquiring the actual size of the three-dimensional part model, based on the transmitted arrangement information,
The apparatus includes means for arranging the obtained three-dimensional part model to construct a three-dimensional model, and means for displaying a two-dimensional projected image of the constructed three-dimensional model.

Further, a two-dimensional moving image display information acquiring means for acquiring information for displaying a two-dimensional moving image from a video camera or a video player, and transmitting the information to a second image processing apparatus via a network. A first image processing apparatus having a transmitting means for transmitting the two-dimensional moving image transmitted from the transmitting means, and outputting a two-dimensional moving image based on the received display information. And a second image processing apparatus having a two-dimensional moving image output unit, wherein the two-dimensional moving image display information acquiring unit is a video camera, or a two-dimensional moving image acquired from a video player. A frame memory for reading and storing a moving image for each frame and a position of an infinity point of the motion of each frame stored in the frame memory are obtained. Means for generating the optical flow of the acquired two-dimensional moving image, and calculating the moving direction and the ratio of the moving with respect to the infinite point of the motion, and the continuous of the generated optical flow. The partial two-dimensional image is divided and extracted as a partial two-dimensional image, and the partial two-dimensional image is extracted from the rate of movement of each frame with respect to the point at infinity in the partial two-dimensional image. The entire moving speed is calculated as a normalized speed, and the appearance position, appearance time, and existence time of the partial two-dimensional image are extracted, and the transmission means transmits the partial two-dimensional image, the moving direction and the standard. Means for transferring display information consisting of vector information consisting of the conversion speed and the appearance position, appearance time, and existence time to the transmission means, and the two-dimensional video output means Based on the transmitted display information, a paste position and a size of each of the partial two-dimensional images for each frame to be output are obtained, and the partial position and the size are obtained based on the obtained paste position and the size. A means is provided for pasting a two-dimensional image to each frame and outputting the pasted frame.

Further, a three-dimensional moving image / three-dimensional model display information obtaining means for obtaining display information for displaying a three-dimensional moving image or a three-dimensional model from a video camera or a video player, and a network for transmitting the display information to the network. A first image processing apparatus having transmission means for transmitting to a second image processing apparatus via the first image processing apparatus; a means for receiving display information of a moving image transmitted by the transmission means; A three-dimensional video / three-dimensional model output means for outputting a three-dimensional video or three-dimensional model to a second image processing apparatus, wherein the three-dimensional video / three-dimensional A model display information acquisition unit, a frame memory for reading and storing a two-dimensional moving image acquired from a video camera or a video player for each frame, and the frame memory; The position of the stored infinite point of the motion of each frame is obtained, and the moving direction and the ratio of the moving to the infinite point of the motion are obtained for the frame group in which the position of the infinite point of the motion is the same. Generating means for generating an optical flow of the obtained two-dimensional moving image, and each of the generated optical flows is divided into continuous regions and extracted as partial two-dimensional images, The moving speed of the entire partial two-dimensional image is calculated as a normalized speed from the ratio of the movement of the frame in the partial two-dimensional image with respect to the point at infinity, and the transmission unit transmits the partial two-dimensional image. Means for transferring display information consisting of vector information comprising the moving direction and the normalized speed, the appearance position thereof, and the point at infinity of movement to the transmission means, The original moving image / three-dimensional model output means obtains depth information of each of the two-dimensional images based on the transmitted display information, and performs type recognition of the component parts of each of the two-dimensional images. Generate a three-dimensional model by arranging a three-dimensional part model corresponding to each component part based on the depth information,
Means for displaying a three-dimensional moving image based on the three-dimensional model or for displaying the three-dimensional model as it is.

[0017] Alternatively, a first image processing apparatus comprising: image transmission means for acquiring information for displaying a two-dimensional or three-dimensional image and transmitting the information to a second image processing apparatus via a network. An image processing system having an apparatus and a second image processing apparatus that outputs an image transmitted by the image transmission unit, wherein the first image processing apparatus is configured to perform an image transmission before transmission. Means for simulating data processing and data processing after transmission, and obtaining a difference between an expected output image as a result of the simulation and an original image before data processing before transmission, and obtaining an error signal thereof Means, and means for acquiring a correction signal for the acquired error signal, and means for transmitting the acquired correction signal and an image signal that has been subjected to data processing before transmission, the The image processing apparatus includes: a unit that performs post-transmission data processing on a transmitted image signal; and a unit that combines the transmitted correction signal with the transmitted data processed image signal and outputs the combined signal. Prepare.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION

(First Embodiment) FIG. 1 is a diagram for explaining a configuration of a moving image processing system according to a first embodiment of the present invention. In the first embodiment, a moving image of a road is acquired,
A moving image of a road that generates a three-dimensional model, extracts information necessary for displaying a three-dimensional moving image from the generated three-dimensional model, transmits the extracted information, and displays the information on a remote image processing device. The processing system will be described.

As shown in FIG. 1, the road moving image processing system according to the first embodiment includes a video camera or a video player 1 and a video recorder 11 for recording an image input from the video camera or the video player 1. A three-dimensional model information acquiring device 2 for acquiring three-dimensional model information from a moving image recorded on the video recording device 11, a display device 3 for displaying the acquired three-dimensional model information, A three-dimensional model information recording device 4 for recording;
A three-dimensional moving image display information generation unit 51 and a transmission unit 52 are provided to transmit three-dimensional moving image display information necessary for three-dimensional display of moving images from the recorded three-dimensional model information to a network 6.
(A LAN, a network, etc.), a three-dimensional moving image display information transmitting apparatus 5, a receiving unit 71, and a three-dimensional moving image display unit 72, which process the transmitted three-dimensional moving image display information. It comprises a three-dimensional moving image display information processing device 7 for displaying a three-dimensional moving image, an input device 8, a display device 9, and a data recording device 10 connected to the three-dimensional moving image display information processing device 7.

The above-described three-dimensional model information acquiring apparatus 2 generates a frame memory 21 for reading and storing a recorded moving image for each frame, and generates an optical flow for each frame or a plurality of frames of the stored moving image. Optical flow generating unit 22 for performing depth information calculating unit 23 for obtaining depth information from the generated optical flow
And a three-dimensional model generation unit 24 that generates a three-dimensional model of a road from optical flow and depth information of each frame or a plurality of frames.

Next, the processing of each unit of the above-described three-dimensional model information acquiring device 2 will be described.

In the three-dimensional model information acquisition device 2, first,
Depth information for performing optical flow generation processing for generating an optical flow for each frame of a moving image or for a plurality of frames in the optical flow generation unit 22 and calculating depth information in the depth information calculation unit 23 based on the generated optical flow The calculation process is performed, and the three-dimensional model generation unit 24 performs a three-dimensional model generation process of generating a three-dimensional model of the road based on the optical flow and the depth information.

Hereinafter, each of the above-described processes will be described in detail.

FIG. 2 is a flowchart for explaining the optical flow generation processing.

In the optical flow generation processing, as shown in FIG. 2, first, a motion vector is obtained by correlating the coordinates of the infinite point of the motion in each frame at two or more points in the image. It is determined as an intersection of vectors (step 201).

The technique for obtaining the infinity point of the movement is disclosed, for example, in "Robot Vision" (published by Shokodo by Masahiko Yauchida, pp. 227-231).

Then, in order to detect the same image portion corresponding to a frame in which the infinite point of the motion can be regarded as the same position, the search direction is set to the direction radiating from the infinite point of the motion as shown in FIG. In order to limit the search direction to the vicinity and determine the search direction, an angle θ1 connecting the infinite point of the motion and the search direction is obtained (step 202).

Thereafter, the position of an area (hereinafter, referred to as a search block) to be searched for by a predetermined length that changes in proportion to the distance from the infinity point of the object on the extension line (search direction) of the obtained angle is shifted, and , A proportional constant indicating a shape enlargement ratio (a dimensional change ratio caused by the distance of the display) by a predetermined length movement is obtained, a correlation is obtained by a search block multiplied by the obtained proportional constant, and a correlation coefficient is obtained with images of a plurality of frames. Is obtained (step 203).

Next, the maximum point of the correlation coefficient within a predetermined search range is extracted (step 204).

Then, an optical flow is obtained from the positional relationship between the original position of the comparison and the maximum point of the correlation coefficient (step 205).

For example, in the case shown in FIG.
When searching for the position of d2 from the position of d1, the search is performed with a unit length of d1 / c proportional to the distance of d1, and d2
When searching for the position of d3 from the position of, the search is performed with a unit length of d2 / c proportional to the distance of d2. Here, c is a constant and varies depending on the definition of the image to be handled.

Next, the proportionality constant is represented by the ratio of the distance from each search block to be correlated to the point at infinity of motion, and in the case of FIG. 4B, d2 / d1 and d4 respectively.
/ D3. Here, d1 and d3 indicate the distances between the objects a and b and the infinite point of the movement at a certain time, and d2 and d
Reference numeral 4 denotes the distance between the objects a and b and the infinite point of the movement after a predetermined time has elapsed from a certain time.

As described above, the size of the objects a and b is
Since the magnitude is different between the case where the object is near and the case where the object is distant, if the correlation is directly obtained as in the related art, the accuracy is reduced. Therefore, a high correlation can be obtained by multiplying a search block for searching for a distantly visible object by the obtained proportionality constant to make the sizes of the blocks to be searched uniform.

As shown in FIG. 5, a search range is determined from the infinity point of the motion of the actual road image to the radial direction indicated by the dotted line and its vicinity, and the search for the correlation is performed. The case where the block is a lighting part of a streetlight will be described specifically.

FIG. 5A shows a photographed road image showing a search block A for performing correlation, and FIG.
(A) shows a road photographed image after a predetermined time, and a portion corresponding to a search block A for performing correlation is a search block B.

FIG. 6 shows search blocks A and B at this time.

As shown in FIG. 6, the lighting part of the street lamp in the search block A (FIG. 6A) and the search block B
The lighting part of the streetlight (FIG. 6B) in the inside has the same dimensions as the actual thing, but the size differs when displayed.

In the prior art, the correlation was obtained without correcting this size, so that an accurate correlation was not always obtained. However, in the present invention, the above-mentioned proportionality constant is multiplied so as to obtain a correlation with the same size. 6 (a) and 6
The dimensions in the search block shown in (b) are matched with each other to increase the degree of correlation.

The technique of the conventional correlation and generation of the optical flow is disclosed in “Moving Image Processing by Personal Computer” (published by Morikita Publishing Co., Ltd.), pp. 149-169.

In order to obtain a more accurate optical flow, the optical flow may be corrected as follows.

A three-dimensional image is generated from the obtained optical flow, and the three-dimensional image is again subjected to a perspective transformation to return to a two-dimensional image. The two-dimensional image is compared with the original image to correct the optical flow. An optical flow with high precision is obtained by repeating.

Next, a depth information generation process for generating depth information based on the generated optical flow information will be described.

The depth information D (m) can be expressed by the following equation (1).

D (x, y) = α / v (x, y) (1) α: camera constant v: size of optical flow (pixel / frame) The coordinate position can be obtained.

As described above, when extracting a motion vector of an object from a two-dimensional moving image input from the video camera 1 or the like, most of the objects in the image are objects fixed to the ground and stationary coordinates. It can be regarded as a system, and when the car moves, the viewpoint moves, and the entire stationary coordinate system appears to move.

Since the mutual positional relationship between the objects on the stationary coordinate system does not change, the movement has the following rule.

A moving image in which the viewpoint moves appears as if all the objects gush from an infinite point of movement. The direction of this source is that all objects are radial from the infinity point of the video to the outside, and the speed of the source is proportional to the distance from the infinity point of the motion. Is proportional to the distance of the movement from the point at infinity.

That is, the apparent movement of all objects is
It changes while increasing its speed while expanding so that it radially springs out at the point at infinity of movement, so when taking a correlation, pay attention to only the moving direction,
The other directions are negligible and can be processed much more efficiently than conventional correlations in all directions.

Since the size of the object to be searched is proportional to the distance from the point at infinity of the motion, the size of the search block is set to a size proportional to the distance from the point at infinity of the motion. , The accuracy of the correlation can be improved.

Next, the three-dimensional model generation processing will be described.

FIG. 7 is a flowchart for explaining the three-dimensional model generation processing.

As shown in FIG. 7, in the three-dimensional model generation processing of this embodiment, a continuous portion of the optical flow generated by the above-described optical flow generation processing is determined based on the depth information obtained above. The regions are extracted as one region (step 701), and the types of components constituting each extracted region are recognized by performing pattern recognition (step 702).

The recognized part type can be represented by a shared three-dimensional part model (a pre-registered three-dimensional part model such as a road or a streetlight) or a unique three-dimensional part model (a scene that is not registered in advance, It is determined whether it is necessary to represent the building with a three-dimensional model (step 703).

If the determination result is a shared three-dimensional part model, parameters are determined from the shape of the three-dimensional part model and a size obtained based on a reference portion (for example, a center line 6 m of a road) in an image. Then, a three-dimensional part model corresponding to the area is generated (step 704).

If the result of the determination is a unique three-dimensional part model, the feature points on the image in that region are extracted, and the size is determined based on a reference portion (for example, a road center line 6 m) in the image. Then, a simple three-dimensional part model corresponding to the area is generated (step 705).

Thereafter, the local coordinates and global coordinates of the generated shared, unique three-dimensional part model are obtained from the information of the motion information recording device 11 and the position of the intersection of the road (step 706). The coordinates in this case are obtained for the reference points of the shared and unique three-dimensional part models.

Then, based on the global coordinates, the generated shared and unique three-dimensional part model is pasted, a three-dimensional model of the road is generated (step 707), and stored in the three-dimensional model information recording device 4 (step 707). Step 708).

As described above, when a search block for searching for the same image portion is moved by a predetermined length in a direction radially expanding from an infinity point of the motion, and a correlation is obtained between frames, a current frame to be correlated is obtained. The distance from the infinity point of the motion in the search block to the search block is calculated by multiplying the inverse of the distance from the infinity point of the motion in the target frame to the search block in the target frame to be correlated. Reduces the time required to find the optical flow by expanding the size of the image in the search block by a multiple of its proportional constant and correlating the image between the search block of the current frame and the previous frame, respectively. And the accuracy can be improved.

Next, the three-dimensional moving image display information transmitting apparatus 5 shown in FIG. 1 for transmitting the moving image moving on the stored three-dimensional model or the pseudo three-dimensional moving image is transmitted. Processing of the moving image display information generation unit 51 and processing of the three-dimensional moving image display unit 72 of the three-dimensional moving image display information processing device 7 that receives the transmitted three-dimensional moving image display information and displays the three-dimensional moving image Will be described. In the first embodiment, three transmission methods will be described.

First, a first processing example of the three-dimensional video display information transmitting device 5 and the three-dimensional video display information processing device 7 will be described.

The first processing example is a processing for transmitting information on each part model, its vector data, and information on the point at infinity of motion from the three-dimensional video display information transmitting apparatus 5 and displaying a three-dimensional video. It is.

FIG. 8 shows a three-dimensional moving image display information generator 51.
5 is a flowchart showing a first processing example of the first embodiment.

As shown in FIG. 8, the processing of the three-dimensional moving image display information generating unit 51 first obtains viewpoint information of a moving image to be displayed (step 801), and, based on the viewpoint information, obtains a three-dimensional model. The perspective conversion is performed on the three-dimensional model stored in the information recording device 4 to project a two-dimensional image (step 802).

The infinity point of the motion in the projected two-dimensional image is obtained in the same manner as described above (step 803), and the moving direction of the reference point of each part model with respect to the infinity point of the motion is obtained (step 804).

Then, depth information for each part model is obtained based on the global coordinates of each part model (step 8).
05), the apparent movement speed of each part model is obtained from the viewpoint movement speed in the viewpoint information (step 806), and the vector data of each part model is generated from the movement direction and movement speed of each part model (step 807). The information of each part model, its vector data, and the information on the point at infinity of movement are transferred to the transmission unit 52 (step 808). In the present embodiment, for simplicity of description, pixels on the screen are associated with the coordinates on a one-to-one basis.

The infinity point information of the movement may be transferred only when the infinity point of the movement changes. The data is transmitted by the transmission unit 52 via the network 6.

Specifically, the vector data of the streetlight and the information on the infinity point of the movement as shown in FIG. 9 are transmitted.

Then, the three-dimensional moving image display information processing device 7
In the first processing example, as shown in FIG. 10, the information of each part model transmitted by the receiving unit 71 and its vector data,
Then, the infinite point information of the motion is transferred to the three-dimensional moving image display unit 72 (step 1001), and the three-dimensional moving image display unit 72 uses the information of each part model, its vector data, and the infinite point information of the motion. A three-dimensional model is generated (step 1002), and the generated three-dimensional model is two-dimensionally projected by the above-described perspective transformation to display a moving image (step 1003).

As a result, the information transmitted when displaying a three-dimensional video is only the information of each part model and its vector data, and the information on the point at infinity of the motion. It can be performed at a lower transmission rate than transmitting all.

Next, a second processing example of the three-dimensional moving image display information transmitting device 5 and the three-dimensional moving image display information processing device 7 will be described.

This second processing is performed by storing information of each component model in advance in the data recording device 10 of the three-dimensional moving image display information processing device 7 and transmitting only the placement information of each component model. This is a process for displaying the original moving image.

FIG. 11 shows a three-dimensional moving image display information generator 5.
6 is a flowchart illustrating a second example of the first process.

In the second processing example of the three-dimensional moving image display information generation unit 51, first, viewpoint information is acquired by the input device 8 (step 1101), and a part model within the field of view of the viewpoint information is converted into a three-dimensional model. It is extracted from the information recording device 4 (step 1102), and global coordinates at the reference point of each component model are extracted as arrangement information (step 11).
03), the type of each part model and its global coordinates are transferred to the transmission unit 52 (step 1104).

Then, similarly, the data is transmitted by the transmission unit 52 via the network 6.

Specifically, taking the streetlights shown in FIG. 12 as an example, the type of the streetlights and the position information of each streetlight indicated by “+” are transmitted.

Then, the three-dimensional moving image display information processing device 7
In the second processing example, as shown in FIG. 13, the information of each component model type and its global coordinates transmitted by the receiving unit 71 are transferred to the three-dimensional moving image display unit 72 (step 13).
01), the three-dimensional video display unit 72 extracts a part model stored in the data recording device 10 in advance according to the type of each part model (step 1302), and generates a three-dimensional model based on the global coordinates of the part model. A three-dimensional model is generated (step 1303), and the generated three-dimensional model is two-dimensionally projected to display a moving image (step 1304).

By recording a three-dimensional part model in advance, the information transmitted when displaying a three-dimensional moving image is only the type of each part model and its global coordinates. This can be performed at a lower transmission rate than transmitting all of the moving image data.

Next, a third processing example of the three-dimensional moving image display information transmitting device 5 and the three-dimensional moving image display information processing device 7 will be described.

In the third process, the information of the parametric three-dimensional part model is stored in the data recording device 10 of the three-dimensional moving image display information processing device 7 in advance, and the parameters of each part model and the arrangement information thereof are transmitted. Is a process of displaying a three-dimensional moving image.

In this way, the storage amount of the three-dimensional part model can be reduced as compared with the second processing.

FIG. 14 shows a three-dimensional video display information generating unit 5.
11 is a flowchart illustrating a third processing example of No. 1;

In the third processing example of the three-dimensional moving image display information generating section 51, first, as shown in FIG. 14, viewpoint information is acquired by the input device 8 (step 1401), and the viewpoint information is set within the view range of the viewpoint information. Is extracted from the three-dimensional model information recording device 4 (step 1402), and the parameters of the part model are obtained based on, for example, a center line length serving as a reference (step 1403), and the reference point of each part model is obtained. Are extracted as arrangement information (step 1103), and the type and parameter of each part model and their global coordinates are transferred to the transmission unit 52 (step 1104).

Then, similarly, the data is transmitted by the transmission unit 52 via the network 6.

Specifically, taking the road shown in FIG. 15 as an example, parameters L1, W1, W
2, W3, D1, and the position information of each road are transmitted.

Then, the three-dimensional moving image display information processing device 7
In the third processing example, as shown in FIG. 16, each component model type, parameter information and its global coordinates transmitted by the receiving unit 71 are transferred to the three-dimensional video display unit 72 (step 1601). The three-dimensional moving image display unit 72 extracts a part model stored in the data recording device 10 in advance according to the type of each part model (step 1602).
A part model having an actual shape and size is obtained based on the parameters (step 1603), and a three-dimensional model is generated based on the global coordinates of the part model (step 160).
4) The generated three-dimensional model is two-dimensionally projected to display a moving image (step 1605).

Thus, the information transmitted when displaying a three-dimensional moving image includes the type and parameter of each part model,
And only the global coordinates thereof, the transmission can be performed at a lower transmission rate than when all the data of the three-dimensional moving image is transmitted.

(Embodiment 2) FIG. 17 is a diagram for explaining the configuration of a moving image processing system according to Embodiment 2 of the present invention.

In the second embodiment, a moving image of a road is acquired,
After generating an optical flow, without generating a 3D model, extract and transmit 2D image display information with a vector required for displaying a 3D moving image or a 2D moving image, and transmit a remote image. A road moving image processing system that performs processing to be displayed on the processing device will be described.

The road moving image processing system according to the present embodiment
As shown in FIG. 17, a video camera 1 mounted on a car, a video recording device 11 for recording an image input from the video camera 1, and two-dimensional and three-dimensional moving images recorded on the video recording device 11. A moving image display information acquisition / transmission device 170 that acquires and transmits two-dimensional image information with a vector for displaying a moving image or a three-dimensional model, and a display device 3 that displays the acquired two-dimensional image information with a vector
A moving image display information processing device 171 for processing the transmitted two-dimensional image information with a vector and displaying a two-dimensional, three-dimensional moving image, and a three-dimensional model, comprising a receiving unit 1711 and a moving image display unit 1712; It comprises an input device 8, a display device 9, and a data recording device 10 connected to the moving image display information processing device 171.

The above-described moving picture display information acquiring / transmitting apparatus 170 includes a frame memory 21 for reading and storing a recorded moving picture for each frame, and an optical flow for each or a plurality of frames of the stored moving picture. The optical flow generation unit 22 to generate, extracts a continuous area portion of the optical flow as a partial two-dimensional image, and generates two-dimensional image information with a vector based on a moving direction and a moving speed of the two-dimensional image. And a transmission unit 1702 for transmitting the generated two-dimensional image information with a vector via the network 6 (LAN, network, etc.). In the second embodiment, the units having the same reference numerals as in the first embodiment perform the same processes as those in the first embodiment, and a description thereof will be omitted.

Next, the processing performed by the vector-added two-dimensional image information generation unit 1701 and the transmission unit 1702 will be described.

First, as shown in FIG. 18, the process of the vector-added two-dimensional image information generating unit 1701 is to convert a continuous area of the optical flow generated by the optical flow generating unit 23 into a partial two-dimensional image. (Step 1801), and calculates the normalized speed of the partial two-dimensional image based on the vector information of the optical flow (Step 1802), the infinity point of the motion, the appearance position of the two-dimensional image, and the appearance Time and existence time information are obtained (step 1803). A two-dimensional image with a vector composed of a point at infinity of motion, a normalized vector and a two-dimensional image, and a vector composed of its appearance position, appearance time, and existence time information are attached. The two-dimensional image information is transferred to the transmission unit 1702 (Step 1804).

The normalized speed in step 1802 is calculated by defining as follows.

The distance from the infinity point of the motion to the object is d, the magnitude of the vector generated by the optical flow (the apparent speed of the object on the screen) is v, and the movement of the object on the screen Assuming that the distance is s, the apparent speed vector v is v = s / △ t.

Here, the normalized speed V is defined as follows.

V = v / d = s / (△ t · d) If the size of the vector to be transmitted at the time of transmission is the standardized speed V, it is not necessary to transmit v that changes for each frame, and the amount of transmission Is less.

When the moving image display information processing device 171 to be described later displays a two-dimensional image, the transmission unit 1702 transmits the infinite point of the motion, the two-dimensional image with the vector, its appearance position, appearance time, and existence time. A process of transmitting two-dimensional image information with a vector including information to the moving image display information processing device 171 via the network 6 is performed, and the moving image display information processing device 171 described later displays a three-dimensional moving image or a three-dimensional model. At this time, a process of transmitting the infinity point of the motion, the two-dimensional image with the vector, and the appearance position to the moving image display information processing device 171 via the network 6 is performed.

Next, the moving image display unit 1712 of the moving image display information processing device 171 that receives the transmitted two-dimensional image information with a vector and displays a two-dimensional moving image, a three-dimensional moving image, or a three-dimensional model is displayed. The processing will be described. In addition,
In the second embodiment, a two-dimensional moving image display, a three-dimensional moving image and a three-dimensional model display, and two transmission methods will be described.

First, the two-dimensional moving image display processing in the moving image display unit 1712 will be described.

As shown in FIG. 19, the two-dimensional image display processing in the moving image display unit 1712 includes the infinite point of the motion received by the reception unit 1711, the two-dimensional image with the vector,
Based on the appearance position, appearance time, and existence time information, a paste position of the two-dimensional image in each display frame is determined (step 1901), and a point at infinity of motion, a two-dimensional image with a vector, the appearance position, The size of the two-dimensional image of each frame is determined based on the appearance time and existence time information (step 1902), and the determined two-dimensional image is pasted on each frame and displayed as a two-dimensional moving image (step 1903). .

As a result, the direction of motion at all points of the two-dimensional image is designated by only a small amount of information, that is, two-dimensional image information with a normalized vector taking into account the vector information of the optical flow. It is very convenient for recording.

When the car moves and the viewpoint changes, by adding and transmitting image information and a vector of only a part of the image newly appearing due to the movement, as long as the image is visible, All old images are used without being thrown away, and
As long as the vector does not change, it is not necessary to transmit all the frames. If the initial image and the vectors of all points in the image are sent, the subsequent motion can be predicted.

That is, this vectored two-dimensional image information can be transmitted at a much higher efficiency than a conventional two-dimensional moving image.

Next, a process of displaying a three-dimensional image and a three-dimensional model in the moving image display unit 1712 will be described.

As shown in FIG. 20, the processing of displaying the three-dimensional image and the three-dimensional model in the moving image display unit 1712 is performed by patterning the types of the parts constituting the two-dimensional image with vector received by the receiving unit 1711. Recognition is performed (step 2001).

It is determined whether the recognized part type can be represented by a shared three-dimensional part model or must be represented by a unique three-dimensional part model (step 200).
2).

If the judgment result is a shared three-dimensional part model, the three-dimensional part model corresponding to the two-dimensional image is determined from the shape of the three-dimensional part model and the size obtained based on the reference part in the image. Generate (Step 200
3).

If the result of the determination is a unique three-dimensional part model, the feature points on the two-dimensional image are extracted, the size is determined based on a reference portion in the image, and the simple point corresponding to the two-dimensional image is obtained. Generate a three-dimensional part model (Step 20)
04).

Thereafter, depth information is obtained from the appearance position of the two-dimensional image, the normalized vector, and the infinite point of the movement, as in Embodiment 1, and the global coordinates of the generated shared, unique three-dimensional part model are obtained. (Step 2005).
The coordinates in this case are obtained for the reference points of the shared and unique three-dimensional part models.

Then, based on the global coordinates, the generated shared and unique three-dimensional part model is pasted, and a three-dimensional model is generated (step 2006). Is displayed (step 2007), and when a three-dimensional image is displayed, the three-dimensional model is perspectively transformed from a certain viewpoint and two-dimensionally projected and displayed (step 2008).

By transmitting only a small amount of information, that is, two-dimensional image information with a vector, three-dimensional coordinates and a three-dimensional part model can be extracted at the transmission destination, and a three-dimensional image or a three-dimensional model can be displayed based on this. Because you can
Extremely efficient transmission becomes possible.

In this embodiment, a two-dimensional image is a two-dimensional image whose viewpoint can be moved only forward and backward, and a three-dimensional image is a two-dimensional image whose viewpoint can be freely moved.

(Embodiment 3) In Embodiment 3, a description will be given of processing in the case where the amount of transmission information is further reduced by performing preprocessing such as compression on the amount of transmission information in the above-described Embodiment 1 or 2. . In this case, it is necessary to correct deterioration of image information caused by pre-processing such as compression or post-processing such as decompression.

Here, for example, the two-dimensional projected image of the three-dimensional model obtained by the three-dimensional model information recording device 4 shown in the first embodiment is used by the three-dimensional moving image display information transmitting device 5 to transmit the image. And performs pre-processing such as compression to reduce the amount of transmission, and transmits the processed data. The transmission of a correction signal-added image in which a received portion is image corrected and transmitted will be described.

FIG. 21 is a flowchart showing a process of transmitting a correction signal-added image in the three-dimensional moving image display information generating section 5 shown in the first embodiment.

The processing of transmitting the correction signal-added image by the three-dimensional moving image display information generating section 51 first obtains a two-dimensional projected image from the three-dimensional model of the three-dimensional model information recording device 4 as shown in FIG. Then, the pre-processing such as compression and post-processing such as decompression are simulated for the obtained two-dimensional projection image (post-processing such as how the image data is degraded after the compression and decompression of the image data) (step 2101). (Step 2102), an expected output image as a simulated result is obtained (Step 2103), and a difference between the expected output image and the two-dimensional projected image as an original image is obtained (Step 2104), and an error signal is obtained (Step 2104). Step 210
5).

Then, a correction signal is obtained for the error signal (step 2106), a two-dimensional projected image as an original image is synthesized with the correction signal (step 2107), and the simulation is performed again on the synthesized image. (Step 2108) to obtain an expected output image (Step 210).
9), a difference is obtained between the obtained predicted output image and the two-dimensional projected image as an original image (step 2110), and an error signal is obtained (step 2111). If the error signal is smaller than a predetermined value, (Step 2112), the correction signal and the image signal are preprocessed (compressed, etc.) and transferred to the transmission unit 52 (Step 2113). If it is larger, the process returns to step 2106.

Then, similarly, the data is transmitted by the transmission unit 52 via the network 6.

Next, as shown in FIG. 22, the processing of transmitting the correction signal-added image by the three-dimensional moving image display information processing apparatus 7 is performed by combining the correction signal and the image signal transmitted by the receiving unit 71 with the three-dimensional moving image. The image data is transferred to the display unit 72 (step 2201), and the three-dimensional moving image display unit 72 performs post-processing (decompression or the like) on the correction signal and the image signal (step 220).
2), combine them and display a moving image (step 2)
203).

As a result, it is possible to reduce deterioration in image quality caused by pre-processing such as compression performed to reduce the transmission rate and decompression, thereby enabling high-quality and highly accurate three-dimensional information transmission.

Although the third embodiment has been described as preprocessing such as compression to reduce information transmission,
The present invention is not limited to this. For example, when information is recorded on a recording device or the like, it can be replaced with pre-processing such as compression to reduce the amount of information to be recorded. Various adaptations are possible for those that perform processing.

As described above, the invention made by the present inventors is described below.
Although specifically described based on the embodiment, the present invention
It is needless to say that the present invention is not limited to the above-described embodiment, but can be variously modified without departing from the scope of the invention.

[0123]

The effects obtained by the representative ones of the inventions disclosed in the present application will be briefly described as follows.

[0124] 1. It is possible to shorten the time required for obtaining the optical flow and improve the accuracy.

[0125] 2. It is possible to reduce the transmission rate of information for displaying a moving image.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining a configuration of a moving image processing system according to a first embodiment of the present invention;

FIG. 2 is a flowchart illustrating an optical flow generation process.

FIG. 3 is a diagram illustrating a search direction of a correlation.

FIG. 4 is a diagram for explaining a positional deviation in a search direction when a correlation search is performed and a size of a search block at that time.

FIG. 5 is a diagram for explaining a difference in the size of an object at the time of correlation using road photographed images having different photographing times.

FIG. 6 is a diagram for explaining in detail a correlation in a search block multiplied by a proportional constant;

FIG. 7 is a flowchart illustrating a three-dimensional model generation process.

FIG. 8 is a flowchart illustrating a process of a three-dimensional moving image display information generation unit 51;

FIG. 9 shows a transmitted three-dimensional part model and its vector,
FIG. 3 is a diagram specifically showing the infinity point of the movement.

FIG. 10 is a flowchart illustrating a first processing example of the three-dimensional moving image display information processing apparatus 7.

FIG. 11 is a flowchart illustrating a second processing example of the three-dimensional moving image display information generation unit 51;

FIG. 12 is a diagram specifically showing an arrangement position to be transmitted;

FIG. 13 is a flowchart illustrating a second processing example of the three-dimensional moving image display information processing apparatus 7.

FIG. 14 is a flowchart illustrating a third processing example of the three-dimensional moving image display information generation unit 51;

FIG. 15 is a diagram showing a specific example of a transmitted parametric three-dimensional model part.

FIG. 16 is a flowchart illustrating a third processing example of the three-dimensional video display information processing apparatus 7;

FIG. 17 is a diagram for explaining a configuration of a moving image processing system according to a second embodiment of the present invention;

FIG. 18 is a two-dimensional image information generation unit with vector 1701.
6 is a flowchart showing the processing of FIG.

FIG. 19 is a flowchart illustrating a two-dimensional moving image display process in the moving image display unit 1712.

FIG. 20 shows a three-dimensional image in the moving image display unit 1712,
6 is a flowchart showing a display process of a three-dimensional model.

FIG. 21 is a flowchart illustrating a process of transmitting a correction signal-added image by the three-dimensional moving image display information generation unit 51.

FIG. 22 is a flowchart showing a process of transmitting a correction signal-added image by the three-dimensional video display information processing apparatus 7.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Video camera or video reproducing device, 2 ... 3D model information acquisition part, 3 ... Display device, 4 ... 3D model information recording device, 5 ... 3D moving image display information transmission device, 6 ... Network, 7 ... Tertiary Original moving image display information processing device, 8 ... input device, 9 ... display device, 10 ... data recording device, 11 ...
Video recording device, 21: Frame memory, 22: Optical flow generator, 23: Depth information calculator, 24: 3D model generator, 51: 3D video display information generator, 52: Transmission unit, 71: Reception Unit, 72 ... three-dimensional moving image display unit.

Claims (9)

[Claims] 1. An image processing system comprising a three-dimensional moving image information acquiring unit for acquiring information of a three-dimensional moving image based on a two-dimensional moving image acquired from a video camera or a video player. The three-dimensional moving image information obtaining means reads and stores the obtained two-dimensional moving image for each frame, and obtains the position of an infinite point of the motion of each frame stored in the frame memory, and For a frame group in which the position of the point at infinity is the same, a direction in which the movement radiates from the point at infinity is determined as a search direction of the same image portion, and a search block for searching the same image portion in the search direction is set to a predetermined length. The search block image is correlated between each frame, the portion having the highest correlation is detected as the same image portion, and the movement of the same image portion at infinity is detected. Optical flow generating means for generating the optical flow of the obtained two-dimensional moving image, and calculating the depth information of the two-dimensional moving image obtained from the generated optical flow. Depth information calculation means, Type recognition of the components of the obtained two-dimensional moving image,
A three-dimensional model generating means for arranging a three-dimensional part model corresponding to each component whose type has been recognized based on the optical flow and the depth information to generate a three-dimensional model of the acquired two-dimensional moving image; An image processing system comprising: 2. The image processing system according to claim 1, wherein the optical flow generation means moves a search block for searching for the same image portion in a direction radially expanding from an infinity point of the movement by a predetermined length. When calculating the correlation between each frame, the reciprocal of the distance from the infinity point of the motion in the target frame to be correlated to the search block is set to the distance from the infinity point of the motion in the current frame to be correlated to the search block. The proportionality constant obtained by the multiplication is obtained, and the size of the image in the search block in the target frame to be correlated is enlarged by the proportionality constant times, and the image between the search block of the current frame and the search block of the immediately preceding frame is obtained. An image processing system comprising means for calculating the correlation of 3. The image processing system according to claim 1, wherein the predetermined length of movement of the search block of the optical flow generation means is a distance from an infinite point of the movement to the search block. An image processing system, characterized in that the unit length is proportional to. 4. Acquiring information for displaying a three-dimensional moving image from a three-dimensional model in which a plurality of three-dimensional component models are arranged and transmitting the information to a second image processing device via a network. A first image processing apparatus having a three-dimensional video transmission unit, and a second image processor that outputs a three-dimensional video based on information of the three-dimensional video transmitted by the three-dimensional video transmission unit. An image processing system having an image processing device, wherein the three-dimensional moving image transmitting unit acquires, from the second image processing device, viewpoint information indicating a position at which a three-dimensional moving image is output, and the viewpoint information A means for sequentially acquiring a two-dimensional projected image of the three-dimensional model, a means for obtaining an infinite point of the motion of the two-dimensional projected image, and a three-dimensional image for the infinite point of the obtained motion. Part model look Means for obtaining vector information indicating the direction of movement and the magnitude thereof, and means for transmitting information on each of the three-dimensional part models and their vector information, and information on the infinite point of movement, Means for constructing the three-dimensional model based on the information of each of the three-dimensional part models and its vector information, and information on the point at infinity of motion; and a two-dimensional model of the constructed three-dimensional model. An image processing system comprising means for displaying a projection image. 5. Acquiring information for displaying a three-dimensional moving image from a three-dimensional model in which a plurality of three-dimensional part models are arranged and transmitting the information to a second image processing device via a network. A first image processing apparatus having a three-dimensional video image transmitting means, and information on the three-dimensional video image transmitted by the three-dimensional video image transmitting means, having the plurality of three-dimensional part model information. And a second image processing apparatus that outputs a three-dimensional video based on the three-dimensional video transmission means, wherein the three-dimensional video transmission means transmits viewpoint information indicating a position at which the three-dimensional video is output to the second Means for acquiring from the image processing device, means for sequentially extracting the three-dimensional component models present within the field of view of the viewpoint information, and means for recognizing the type of each of the extracted three-dimensional component models, Each extracted three Means for acquiring arrangement information of the original part model; means for transmitting the type information of the recognized three-dimensional part model and the acquired arrangement information of the three-dimensional part model, the second image processing apparatus Means for extracting a target three-dimensional part model based on the transmitted type information of each of the three-dimensional part models; anda means for extracting the extracted three-dimensional part model based on the transmitted arrangement information. An image processing system comprising: means for arranging and constructing a three-dimensional model; and means for displaying a two-dimensional projected image of the constructed three-dimensional model. 6. Acquiring information for displaying a three-dimensional moving image from a three-dimensional model in which a plurality of three-dimensional part models are arranged and transmitting the information to a second image processing apparatus via a network. A first image processing apparatus having a three-dimensional moving image transmitting unit, and a plurality of parametric three-dimensional part model information, and a three-dimensional moving image transmitted by the three-dimensional moving image transmitting unit. And a second image processing device that outputs a three-dimensional moving image based on the information of the three-dimensional moving image, wherein the three-dimensional moving image transmitting unit outputs viewpoint information indicating a position at which the three-dimensional moving image is output. Means for acquiring from the second image processing apparatus, means for sequentially extracting the three-dimensional component models present within the field of view of the viewpoint information, and the type and size of each of the extracted three-dimensional component models. Recognize Means for acquiring arrangement information of each of the extracted three-dimensional part models; parameter information indicating the type and size of the recognized three-dimensional part model; and arrangement of the acquired three-dimensional part model. And means for transmitting information, based on the transmitted type information of each of the three-dimensional component models, means for extracting a target three-dimensional component model, and the extracted Means for acquiring the three-dimensional part model of the actual size based on the transmitted parameter information for the three-dimensional part model transmitted, and the acquired three-dimensional part model based on the transmitted arrangement information. An image processing system, comprising: means for constructing a three-dimensional model by arranging the three-dimensional model; and means for displaying a two-dimensional projected image of the constructed three-dimensional model. 7. A two-dimensional moving image display information acquiring means for acquiring information for displaying a two-dimensional moving image from a video camera or a video player, and transmitting the information to a second image processing apparatus via a network. A first image processing apparatus having a transmitting means for transmitting the two-dimensional moving image transmitted from the transmitting means, and outputting a two-dimensional moving image based on the received display information. And a second image processing apparatus having a two-dimensional moving image output unit, wherein the two-dimensional moving image display information obtaining unit is a video camera,
Or, a frame memory that reads and stores a two-dimensional moving image obtained from a video player for each frame, and obtains the position of the infinite point of the motion of each frame stored in the frame memory, and calculates the position of the infinite point of the motion. Means for generating the optical flow of the obtained two-dimensional moving image by calculating the moving direction and the moving ratio of the motion with respect to the point at infinity for the frame group having the same position, and the generated optical flow Are divided into continuous area portions and extracted as partial two-dimensional images, and the partial two-dimensional images are extracted from the ratio of the movement of each frame in the partial two-dimensional image with respect to the point at infinity. The moving speed of the entire two-dimensional image is calculated as a normalized speed, the appearance position, appearance time, and existence time of the partial two-dimensional image are extracted, and the transmission is performed. Means for transferring to the transmission means display information consisting of the partial two-dimensional image, vector information comprising the moving direction and the normalized speed, and the appearance position, appearance time, and existence time, The two-dimensional moving image output means obtains, based on the transmitted display information, a pasting position and a size of the partial two-dimensional image for each frame to be output, and the obtained pasting position. An image processing system comprising: a means for pasting the partial two-dimensional image to each of the frames with a size and outputting the pasted frames. 8. A three-dimensional moving image / three-dimensional model display information acquiring means for acquiring display information for displaying a three-dimensional moving image or a three-dimensional model from a video camera or a video player, and a network for transmitting the display information. A first image processing apparatus having transmission means for transmitting to a second image processing apparatus via the first image processing apparatus; a means for receiving display information of a moving image transmitted by the transmission means; And a second image processing apparatus having a three-dimensional video / three-dimensional model output unit that outputs a three-dimensional video or three-dimensional model to the three-dimensional video / three-dimensional model. The model display information acquisition means is:
A frame memory for reading and storing a two-dimensional moving image obtained from a video camera or a video player for each frame, and obtaining a position of an infinity point of a motion of each frame stored in the frame memory; Generating means for obtaining a moving direction and a moving ratio of the movement with respect to the infinite point of the movement with respect to a frame group in which the position of the far point is the same, and generating an optical flow of the obtained two-dimensional moving image; The optical flow is divided into continuous area portions and extracted as partial two-dimensional images, and the movement rate of each frame in the partial two-dimensional image relative to the point at infinity is calculated as follows. The moving speed of the whole of the partial two-dimensional image is calculated as a normalized speed, and the transmitting means transmits the partial two-dimensional image and the moving direction thereof. Means for transferring display information consisting of vector information consisting of normalized speed, its appearance position, and the point at infinity of movement to a transmission means, the three-dimensional video / three-dimensional model output means, Based on the transmitted display information, determine the depth information of each of the two-dimensional images, type recognition of the components of each of the two-dimensional images,
A three-dimensional model is generated by arranging a three-dimensional part model corresponding to each of the component parts whose type is recognized based on the depth information, and displaying a three-dimensional moving image based on the three-dimensional model. And a means for displaying the three-dimensional model as it is. 9. A first image processing apparatus comprising: an image transmission unit that acquires information for displaying a two-dimensional or three-dimensional image and transmits the information to a second image processing device via a network. An image processing system comprising: a device; and a second image processing device that outputs an image transmitted by the image transmission unit, wherein the first image processing device performs transmission on an image to be transmitted. Means for simulating data processing and data processing after transmission, and obtaining a difference between an expected output image as a result of the simulation and an original image before performing data processing before transmission and obtaining an error signal thereof Means for acquiring a correction signal for the acquired error signal, and means for transmitting the acquired correction signal and an image signal that has been subjected to data processing before transmission, the second The image processing apparatus includes means for performing post-transmission data processing on the transmitted image signal, and means for combining the transmitted correction signal with the transmitted image signal after data transmission and outputting the combined signal. An image processing system characterized in that: