KR101383997B1 - Real-time video merging method and system, visual surveillance system and virtual visual tour system using the real-time video merging - Google Patents

Abstract

The present invention relates to a method and a system for real-time video merging. The system for real-time video merging according to the present invention includes: a background three-dimensional coordinate calculating unit for calculating the three-dimensional coordinates of a background, based on videos received from a plurality of cameras, and storing the background and the three-dimensional coordinates thereof; a real-time background texture updating unit for updating the texture of the background in real time; an inflow object detecting unit for detecting an external object flowing into the background; an inflow object three-dimensional coordinate calculating unit for calculating the three-dimensional coordinates of the detected external object; an object transforming unit for transforming the three-dimensional coordinates of the external object to be suited to the background and mapping the texture to the object; a background generating unit for generating a final background; a final image generating unit for inserting the texture-mapped object in the final background, merging the final background with the background of an adjacent camera to generate a final image; and a control unit for controlling at least one among the components. According to the present invention, the system can enable a user to easily grasp the whole situation in real time by reconstructing a plurality of screens recorded by each camera into one integrated screen and displaying it, and can enable a person, who cannot directly visit a surveillance area, to indirectly experience the surveillance area in real time by reconstructing a video independently recorded by cameras and displaying the video, according to the vision of a virtual experient, when the virtual experient moves along a path in the surveillance area. [Reference numerals] (400) Real-time background texture updating unit; (410) Background three-dimensional coordinate calculating unit; (420) Background generating unit; (430) Inflow object detecting unit; (44) User terminal; (440) Inflow object three-dimensional coordinate calculating unit; (450) Object transforming unit; (46) Network; (460) Final image generating unit; (470) Control unit

Description

Real-time video merging method and system, visual surveillance system and virtual visual tour system using the real-time video merging}

The present invention relates to a method and system for merging video in real time, and more particularly, to a video surveillance system and a virtual video tour system using real-time video merging.

Current control systems independently display screens taken by each camera on multiple displays. One controller is responsible for a small number of display screens. In the case of a large number of display screens in a control center, multiple control personnel are required. In addition, in the current control system, since each display screen shows an independent situation, it is difficult to integrate the entire situation of the monitoring facility and the area.

In addition, in the case of large events such as the Olympics and Expo, most people can't participate in the scene and can only grasp the atmosphere of the scene only through the broadcast. There was no way to check whether there was a risk in the path of entry and exit in real time using a smartphone. In addition, it was not possible to monitor comprehensively in places where people are difficult to access, such as nuclear power plants, there was a problem that it is not easy to investigate a newly visited area in advance, such as travel due to problems such as cost and time.

Accordingly, in order to solve the above problems, a technique of merging a video output from a plurality of cameras into a single video that is spatially connected and displaying in real time is required.

The present invention has been made to solve the above-mentioned problems of the prior art, the problem to be solved by the present invention, a plurality of network-connected surveillance cameras can be merged into a single spatially connected video video obtained separately Is to provide a real time video merging method.

In addition, another problem to be solved by the present invention is to provide a real-time video merging system that can merge the video obtained separately by a plurality of network-connected surveillance cameras into a single spatially connected video.

In addition, another problem to be solved by the present invention is that a plurality of networked cameras are integrated by real-time merging the video taken by each camera into a single video connected in space in a system for monitoring a specific area and facilities It is to provide a video surveillance system using real-time video merging that can be displayed on the control screen.

In addition, another problem to be solved by the present invention, it can show the situation of the path viewed from the point of view of the virtual experienced person or the observer moving the path from one point to another point in the surveillance area merged into a continuous video in space and time. Is to provide a virtual video tour system using a real-time video merger.

Real-time video merging method according to an embodiment of the present invention for solving the above problems,

(A) receiving a video from a plurality of cameras;

(B) calculating three-dimensional coordinates of the background based on the received video and storing the background and three-dimensional coordinates of the background;

(C) updating the texture of the background in real time based on the received video;

(D) detecting an external object introduced into the background based on the received video, the three-dimensional coordinates of the stored background, and the texture of the background updated in real time;

(E) calculating three-dimensional coordinates of the detected external object based on the detected external object and the stored background;

(F) mapping the texture to the object after modifying the three-dimensional coordinates of the external object to match the background;

(G) generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background; And

(H) inserting the texture-mapped object into the final background and merging with the background of an adjacent camera to generate a final image.

In the step (A), a video is received from a plurality of cameras via a network, and in the step (E), the detected external object is based on a coordinate of a specific point where the detected external object meets the stored background. It is desirable to calculate three-dimensional coordinates.

Real-time video merging system according to an embodiment of the present invention for solving the other problem,

A background three-dimensional coordinate calculation unit configured to calculate a three-dimensional coordinate of the background based on the video received from a plurality of cameras and to store the background and three-dimensional coordinates of the background;

A background texture real time updating unit which updates a background texture in real time based on the received video;

An inflow object detector configured to detect an external object introduced into the background based on the received video, the 3D coordinates of the stored background, and the texture of the background updated in real time;

An inflow object three-dimensional coordinate calculator configured to calculate three-dimensional coordinates of the detected external object based on the detected external object and the stored background;

An object deformer configured to map a texture to an object after deforming the 3D coordinates of the external object to match a background;

A background generator for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background;

A final image generator which inserts the texture-mapped object into the final background and merges with the background of an adjacent camera to generate a final image; And

Among the plurality of cameras, the background 3D coordinate calculator, the background texture real-time updater, the inflow object detector, the inflow object 3D coordinate calculator, the object deformer, the background generator, and the final image generator It characterized in that it comprises a control unit for controlling at least one.

The background 3D coordinate calculator, the background texture real-time updater, and the inflow object detector receive a video from the camera through a network, and the inflow object 3D coordinate calculator is configured to detect the detected external object and the stored background. It is preferable to calculate the three-dimensional coordinates of the detected external object based on the coordinates of the specific point where it meets.

The video surveillance system using a real-time video merge according to an embodiment of the present invention for solving the other problem,

A plurality of cameras for shooting a video;

A background three-dimensional coordinate calculator configured to calculate a three-dimensional coordinate of the background based on the videos received from the plurality of cameras, and store the background and three-dimensional coordinates of the background;

A background texture real time updating unit which updates a background texture in real time based on the received video;

An inflow object detector configured to detect an external object introduced into the background based on the received video, the 3D coordinates of the stored background, and the texture of the background updated in real time;

An inflow object three-dimensional coordinate calculator configured to calculate three-dimensional coordinates of the detected external object based on the detected external object and the stored background;

An object deformer configured to map a texture to an object after deforming the three-dimensional coordinates of the external object to match a background;

A background generator for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background;

A final image generator which inserts the texture-mapped object into the final background and merges with the background of an adjacent camera to generate a final image;

One or more user terminals that receive and display the merged image output from the final image generator; And

Among the plurality of cameras, the background 3D coordinate calculator, the background texture real-time updater, the inflow object detector, the inflow object 3D coordinate calculator, the object deformer, the background generator, and the final image generator And a control unit for controlling at least one and receiving a signal from the user terminal.

The virtual video tour system using a real-time video merge according to an embodiment of the present invention for solving the other problem,

A plurality of cameras for photographing a video, comprising: a plurality of cameras capable of adjusting a view angle, a zoom magnification, a panning angle, and a direction according to a control signal;

A background three-dimensional coordinate calculator configured to calculate a three-dimensional coordinate of the background based on the videos received from the plurality of cameras, and store the background and three-dimensional coordinates of the background;

Based on the user's location, the starting point, the arrival point, and the moving speed of the user, the cameras in the moving path from the starting point to the arrival point are activated according to the moving speed and the angle of view entered by the user, A controller for controlling the cameras according to the zoom magnification, the panning angle and the direction;

A background texture real-time updating unit that updates a background texture in real time based on a video received from the activated camera;

An inflow object detection unit that detects an external object introduced into the background based on a video received from the activated camera, a three-dimensional coordinate of a stored background of a location where the activated camera is located, and a texture of the background updated in real time. ;

An inflow object three-dimensional coordinate calculator configured to calculate three-dimensional coordinates of the detected external object based on the detected external object and the stored background;

An object deformer configured to map a texture to an object after deforming the three-dimensional coordinates of the external object to match a background;

A background generator for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background;

A final image generator which inserts the texture-mapped object into the final background and merges with the background of an adjacent camera to generate a final image; And

At least one user terminal for receiving and displaying the merged image output from the final image generating unit,

The controller may include the plurality of cameras, the background 3D coordinate calculator, the background texture real time updater, the inflow object detector, the inflow object 3D coordinate calculator, the object deformer, the background generator, and the final object. Control at least one of the image generating unit, and receives a signal from the user terminal.

According to the present invention, the real-time video obtained by rotating a plurality of network-connected surveillance cameras are merged into a single spatially connected video to reconstruct and display a plurality of screens taken by each camera into a single integrated screen. It is possible to easily grasp the whole situation and to operate the control system efficiently with a few control personnel. In addition, according to the present invention, when the virtual experiencer or the observer moves the path on the surveillance region, the surveillance region can be observed from a new perspective by constructing and displaying a video of the camera photographed independently according to the experiencer or the observer's time. In addition, a person who cannot visit the surveillance area directly can experience the surveillance area indirectly in real time.

1A is a side view of a camera for 3D image acquisition of a background in a surveillance zone.
FIG. 1B is a plan view of the camera shown in FIG. 1A.
1C is a diagram for explaining estimating three-dimensional information of a background.
2A is a diagram illustrating photographing a path while rotating a plurality of cameras.
FIG. 2B is a diagram exemplarily displaying moving images photographed with the camera fixed on each monitor. FIG.
FIG. 2C is a diagram illustrating an example of merging moving images taken by rotating a plurality of cameras into images connected in space and displaying them on the integrated monitor.
3A is a diagram illustrating two images obtained from an adjacent camera having an inflow object.
3B is a diagram illustrating an image obtained by simply merging two images without detecting, separating, and modifying an inflow object.
4 is a schematic block diagram of a video surveillance system using real-time video merging according to an embodiment of the present invention.
5 is a flowchart illustrating a real-time video merging method according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of reconstructing and displaying a video of a camera independently photographed according to a vision of an experienced person or a monitor when a virtual experienced person or a monitor moves a path on a surveillance area.

BRIEF DESCRIPTION OF THE DRAWINGS The objectives, specific advantages and novel features of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG. It should be noted that, in the present specification, the reference numerals are added to the constituent elements of the drawings, and the same constituent elements are assigned the same number as much as possible even if they are displayed on different drawings. Also, the terms "first "," second ", "one side "," other ", and the like are used to distinguish one element from another, no. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS In the following description of the present invention, detailed description of related arts which may unnecessarily obscure the gist of the present invention will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

4 is a schematic block diagram of a video surveillance system using real-time video merging according to an embodiment of the present invention, and FIG. 5 is a flowchart illustrating a real-time video merging method according to an embodiment of the present invention.

As shown in FIG. 4, the video surveillance system using the real-time video merge includes a plurality of cameras 42 through a network 48 and a plurality of cameras C1,... And a user terminal 44 connected to the real time video merging system 40 through a connected real time video merging system 40 and a network 46.

Meanwhile, the real-time video merging system 40 calculates three-dimensional coordinates of the background based on the videos received from the plurality of cameras 42 through the network 48 and stores the three-dimensional coordinates of the background and the background. A background three-dimensional coordinate calculation unit 410, a background texture real-time updating unit 400 which updates a texture of a background in real time based on the received video, the received video, three-dimensional coordinates of the stored background, and An inflow object detector 430 for detecting an external object introduced into the background based on the texture of the background updated in real time, and the three-dimensional coordinates of the detected external object based on the detected external object and the stored background. An inflow object three-dimensional coordinate calculation unit 440 that calculates an object deformation unit 450 that transforms a three-dimensional coordinate of the external object to a background and then maps a texture to the object; A background generator 420 for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background, and inserting the object to which the texture is mapped into the final background, The final image generating unit 460 and the plurality of cameras 42, the background three-dimensional coordinate calculation unit 410, the background texture real-time updating unit 400, and the inflow object detection unit generating a final image by merging with a background. 430, at least one of the inflow object three-dimensional coordinate calculator 440, the object deformer 450, the background generator 420, and the final image generator 460 to receive a control signal CS. Control unit 470 to control through, and receives a signal from the user terminal 44. Although the user terminal 44 illustrates one user terminal in this embodiment, the present invention is not limited thereto, and a plurality of user terminals may exist.

The operation of the video surveillance system using the real-time video merge configured as described above will be described with reference to FIGS. 4 and 5.

The real-time video merging system 40 receives signals from the plurality of cameras 42 and the user terminal 44 via the networks 48 and 46, respectively (S500). The camera 42 receives a video, and the user terminal 44 receives an angle of view, pan information, zoom information, direction, user position, and the like. When the angle of view, pan information, zoom information, direction, and user position of the camera are preset to a fixed value, receiving a signal from the user terminal 44 may be omitted.

As shown in FIGS. 1A and 1B, the background 3D coordinate calculation unit 410 receives a video output from the plurality of cameras 42 installed in the surveillance area through the network 48 and based on the 3D of the background. The coordinate value is calculated offline and the background and the three-dimensional coordinates of the background are stored (S520). Each of the plurality of cameras 42 as illustrated in FIGS. 1A and 1B may include a camera 10, a rotation shaft 12, and a driver 14 to perform a rotation function and a zoom function. In addition, when the camera 42 transmits a video to the real-time video merging system 40, in addition to the image information, each of the transmitted image frames includes pan information (up and down, left and right rotation angles) and zoom information (zoom ratio) of the camera. And meta information such as recording time.

In addition, it is assumed that the location information of each camera and the surveillance area for recording the situation of the partial area of the entire area are previously stored in the real-time video merging system 40. The camera 42 is arranged to monitor a large area while rotating and to superimpose the background with an adjacent camera.

The background 3D coordinate calculator 410 first calculates 3D coordinates of the background monitored by each camera as shown in FIG. 1C (S520).

The three-dimensional coordinates of the background are relative three-dimensional information based on the camera coordinates of the camera. Since the virtual coordinate systems of the two cameras are different, the three-dimensional information of the background captured by two adjacent cameras is displayed with different values in the coordinate systems of the two cameras. Since the three-dimensional information of the background represented by different values is the same, the virtual coordinate systems of adjacent cameras are unified into one through an affine transformation of one coordinate system to another. If the coordinate system is unified, three-dimensional information of the background in the surveillance area of the camera 42 linked to the network 48 is unified.

A process of calculating the three-dimensional coordinates of the background will be described with reference to FIG. 1C. In FIG. 1C, the coordinates X ₃ , Y ₃ , Z ₃ represent the three-dimensional coordinates of a point 100 on the background of interest, and the coordinates X ₁ , Y ₁ , Z ₁ represent an arbitrary rotation angle. The coordinates of the camera's focus 110, the coordinates (X ₂ , Y ₂ , Z ₂ ) represent the camera's focus 120 coordinates of any rotational angle, and the coordinates (x ₁ , y ₁ ) represent points on the background ( The coordinates (X ₃ , Y ₃ , Z ₃ ) of ₁₀₀ represent the coordinates projected onto the image plane of the camera having the focal coordinates (X ₁ , Y ₁ , Z ₁ ), and the coordinates (x ₂ , y ₂ ) represents the coordinates (X ₃ , Y ₃ , Z ₃ ) of the point 100 on the background projected onto the image plane of the camera having the focal coordinates (X ₂ , Y ₂ , Z ₂ ), and reference numeral 150. Denotes an image plane of the camera having the focal coordinates (X ₁ , Y ₁ , Z ₁ ), and reference numeral 160 denotes an image plane of the camera having the focal coordinates (X ₂ , Y ₂ , Z ₂ ).

First, reference coordinates of a rotating camera are set, and an internal variable of the camera is estimated through a widely used camera calibration method. The relative coordinate values of the point 110 and the point 120 may be known from the estimated internal variable and the rotation information of the camera. Using the widely used same point estimation method, the coordinates of the same points 110 and 120 projected on two image planes are estimated. When a point in the background is projected on a plurality of image planes of a rotating camera, a line connecting the point 110 and the point 130 and a line connecting the point 120 and the point 140 meet at the point 100. , The coordinates (X ₃ , Y ₃ , Z ₃ ) of the point 100 can be known. If one point of the object knows the two-dimensional coordinates of the same point projected on the two images and the coordinates of the focal point of the camera, the method of estimating the three-dimensional coordinates of the one point of the object is well known, and thus the description thereof will be omitted. do.

When the 3D coordinates of the background are calculated offline by the background 3D coordinate calculator 410, the background texture real-time updater 400 updates the textures (brightness and color) of each coordinate of the background in real time (S510).

The image frame obtained at any time in each camera has the camera's position, pan information and zoom information. Based on this information, the three-dimensional coordinates (X, Y, Z) of the background corresponding to each pixel (x, y) of the frame are determined. Based on this, the color values of the pixels (x, y) are three-dimensional of the background. This is the texture value of the coordinates (X, Y, Z).

Since transmitting images acquired from multiple cameras in real time is a burden on the network, it is desirable to update the texture only when the change in the background can be visually detected.

Next, a description will be given of detecting external objects (vehicles, people, etc.) introduced into the background in real time and separating them from the background.

The inflow object detector 430 detects an external object introduced into the background based on the received video, the three-dimensional coordinates of the stored background, and the texture of the background updated in real time (S530). The inflow object three-dimensional coordinate calculator 440 calculates three-dimensional coordinates of the detected external object based on the detected external object and the stored background (S540). The calculation of the three-dimensional coordinates of the detected external object is performed based on the coordinates of the specific point where the detected external object and the stored background meet. For example, when the detected external object is a person, three-dimensional coordinates of the person, which is the external object, are calculated based on the point where the sole and the background of the person meet. The object deforming unit 450 deforms the 3D coordinates of the external object to match the background and then maps a texture to the object (S550).

Each camera captures a surveillance area at an arbitrary angle at an arbitrary position. The image frames captured at each time point undergo an affine transformation to combine into an integrated image connected in space. In this case, there is no problem in the case where each pixel represents the background. However, in the case of the object into which the pixel is introduced, the three-dimensional coordinates of the pixel and the background are different. Similarly, distortion of the image occurs. 3A and 3B are diagrams for explaining that distortion occurs in the shape of an object having three-dimensional coordinates different from the background when applying a geometric transformation, that is, an affine transformation, when combining.

3A and 3B, reference numeral 30 denotes a camera connected by a network, reference numeral 32 denotes a camera adjacent to a networked camera 30, and reference numeral 34 denotes an external projected onto an image plane of the camera 30. Reference numeral 36 denotes an inflow object, and reference numeral 36 denotes an external inflow object projected on the image plane of the camera 32, and reference numeral 37 denotes that no geometric transformation is performed when the image of the camera 30 and the camera 32 is connected. The image of the camera 30 is shown, and since there is no geometric transformation in this image, there is no distortion in the shape of the inflow object. In addition, reference numeral 38 denotes an image of the camera 32 undergoing a geometric transformation when the images of the camera 30 and the camera 32 are connected, and it is understood that distortion has occurred in the shape of the inflow object due to the geometric transformation. Can be.

To correct this, the incoming object is detected, separated from the background, the coordinates of the incoming object are calculated, transformed, and then projected onto the integrated image. In this case, the introduced object is separated from the background, and the 3D coordinate information of the area and the object is managed separately.

The background generator 420 generates a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background (S560). The final image generator 460 inserts the deformed texture-mapped object into the final background and merges it with the background of the adjacent camera to generate a final image (S570). The final image generated in this way is transmitted to the user terminal 44 through the network 46, and as shown in FIG. 2C, the user terminal 44 is spatially connected to a video obtained by a plurality of cameras individually connected to the network. It is merged into one video and displayed.

Referring to FIG. 6, a virtual video tour system using real-time video merging according to an embodiment of the present invention will be described.

The virtual video tour system using the real-time video merging according to an embodiment of the present invention has the same configuration as the video surveillance system using the real-time video merging shown in FIG. 4, but the controller 470 controls the user's location and Based on the starting point, the arrival point, and the moving speed, the cameras in the user's moving path from the starting point to the arrival point are activated according to the moving speed, and the user's angle of view, zoom ratio, pan information, and direction are input. There is only a difference in receiving the necessary information from the user terminal 44 to control the cameras accordingly.

In a virtual video tour system using real-time video merging according to an embodiment of the present invention, when a virtual experience person (monitor) moves along a path on a surveillance area, a video of a camera photographed independently according to the time of the experience person (monitor) is displayed. Reconstruct and show to the experiencer.

In FIG. 6, (a) illustrates a plurality of cameras photographing a path while rotating. Each camera stores a photographed image while rotating and monitors a path by overlapping with an adjacent camera.

In FIG. 6, (b) shows an image corresponding to the viewpoint of the experience person by referring to the 3D information of the background by taking an image taken by an adjacent camera according to the time passing through each point of the path when the virtual experience person moves along the movement path. To illustrate the reconstruction.

The real-time video merging system 40 has three-dimensional coordinates of each camera located on the facility map, and stores image information and meta information captured by each camera.

When a user who accesses the real-time video merging system 40 through the user terminal 44 and the network 46 sees a map of the facility and designates a starting point, an intermediate route, and a destination point, the real-time video merging system 40 next asks the user. Providing services like

The path is selected as the optimal path on the map, and the video taken by the camera located in the path is selected for each time. If the user wants to experience the scene while moving along the path, each of the stored camera images are selected and reconstructed and displayed as one video so that the user feels the path moving. Since this operation is as described above, it will be briefly described.

If the user is located at a point P ₀ on the path at time t ₀ , select multiple cameras monitoring one point P ₀ and retrieve images stored at the time closest to time t ₀ in each camera.

The area of the background viewed from the user's point of view is selected and the surveillance image of the area of the selected background is mapped. This uses a well known method of mapping texture images to shapes.

In the stored image, an object introduced into the background exists. When the camera image corresponding to the area of the user is mapped as a whole, the image of the object is distorted. In order to solve the distortion of the image, a process of separating the image of the object from the background image is necessary.

The real-time video merging system 40 stores the texture information of the background while updating in real time, and separates the object from the background by comparing the color and texture of the latest stored background with the color and texture of the area of the imported object. After separating the object from the background, the background is mapped to the background image seen by the user. Finally, the final image is generated by relocating the separated object to a new background generated by mapping the background and provided to the user through the network 46 and the user terminal 44.

The methods discussed herein may be implemented using various means depending on the application. For example, these methods may be implemented in the form of hardware, firmware, software, or any combination thereof. In an implementation involving hardware, the processing unit may include one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, It may be implemented in microprocessors, electronic devices, other electronic units designed to perform the functions discussed herein, or a combination thereof.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, the same is by way of illustration and example only and is not to be construed as limiting the present invention. It is obvious that the modification or improvement is possible.

It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

The present invention can be used for indirect experiences such as those who do not participate in the field at large events such as the Olympics and Expo. In addition, the present invention is used to confirm whether there is no danger on the route in the parking lot of the late night, such as a smart phone in advance, to effectively monitor the places where people are difficult to access, such as nuclear power plants, or to visit the newly visited area in advance Can be.

40: Real Time Video Merger System
42: camera
44: user terminal
46: network
48: network
400: background texture real-time update unit
410: background 3D coordinate calculation unit
420: background generator
430: inflow object detector
440: inflow object three-dimensional coordinate calculation unit
450: object deformation
460: the final image generating unit
470: control unit

Claims (6)

(A) receiving a video from a plurality of cameras;
(B) calculating three-dimensional coordinates of the background based on the received video and storing the background and three-dimensional coordinates of the background;
(C) updating the texture of the background in real time based on the received video;
(D) detecting an external object introduced into the background based on the received video, the three-dimensional coordinates of the stored background, and the texture of the background updated in real time;
(E) calculating three-dimensional coordinates of the detected external object based on the detected external object and the stored background;
(F) mapping the texture to the object after modifying the three-dimensional coordinates of the external object to match the background;
(G) generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background; And
(H) real-time video merging method comprising the step of inserting the texture-mapped object to the final background and merging with the background of an adjacent camera to generate a final image. The method according to claim 1,
In the step (A), a video is received from a plurality of cameras via a network, and in the step (E), the detected external object is based on a coordinate of a specific point where the detected external object meets the stored background. Real-time video merging method, characterized in that for calculating the three-dimensional coordinates. A background three-dimensional coordinate calculation unit configured to calculate a three-dimensional coordinate of the background based on the video received from a plurality of cameras and to store the background and three-dimensional coordinates of the background;
A background texture real time updating unit which updates a background texture in real time based on the received video;
An inflow object detector configured to detect an external object introduced into the background based on the received video, the 3D coordinates of the stored background, and the texture of the background updated in real time;
An inflow object three-dimensional coordinate calculator configured to calculate three-dimensional coordinates of the detected external object based on the detected external object and the stored background;
An object deformer configured to map a texture to an object after deforming the three-dimensional coordinates of the external object to match a background;
A background generator for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background;
A final image generator which inserts the texture-mapped object into the final background and merges with the background of an adjacent camera to generate a final image; And
Among the plurality of cameras, the background 3D coordinate calculator, the background texture real-time updater, the inflow object detector, the inflow object 3D coordinate calculator, the object deformer, the background generator, and the final image generator Real-time video merging system comprising a control unit for controlling at least one. The apparatus of claim 3, wherein the background 3D coordinate calculator, the background texture real-time updater, and the inflow object detector receive a video from the camera through a network, and the inflow object three-dimensional coordinate calculator is configured to detect the detected external object and And a three-dimensional coordinate of the detected external object based on coordinates of a specific point where the stored background meets. A plurality of cameras for shooting a video;
A background three-dimensional coordinate calculator configured to calculate a three-dimensional coordinate of the background based on the videos received from the plurality of cameras, and store the background and three-dimensional coordinates of the background;
A background texture real time updating unit which updates a background texture in real time based on the received video;
An inflow object detector configured to detect an external object introduced into the background based on the received video, the 3D coordinates of the stored background, and the texture of the background updated in real time;
An inflow object three-dimensional coordinate calculator configured to calculate three-dimensional coordinates of the detected external object based on the detected external object and the stored background;
An object deformer configured to map a texture to an object after deforming the three-dimensional coordinates of the external object to match a background;
A background generator for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background;
A final image generator which inserts the texture-mapped object into the final background and merges with the background of an adjacent camera to generate a final image;
One or more user terminals that receive and display the merged image output from the final image generator; And
Among the plurality of cameras, the background 3D coordinate calculator, the background texture real-time updater, the inflow object detector, the inflow object 3D coordinate calculator, the object deformer, the background generator, and the final image generator And a control unit for controlling at least one and receiving a signal from the user terminal. A plurality of cameras for photographing a video, comprising: a plurality of cameras capable of adjusting a view angle, a zoom magnification, a panning angle, and a direction according to a control signal;
A background three-dimensional coordinate calculator configured to calculate a three-dimensional coordinate of the background based on the videos received from the plurality of cameras, and store the background and three-dimensional coordinates of the background;
Based on the user's location, the starting point, the arrival point, and the moving speed of the user, the cameras in the moving path from the starting point to the arrival point are activated according to the moving speed and the angle of view entered by the user, A controller for controlling the cameras according to the zoom magnification, the panning angle and the direction;
A background texture real-time updating unit that updates a background texture in real time based on a video received from the activated camera;
An inflow object detection unit that detects an external object introduced into the background based on a video received from the activated camera, a three-dimensional coordinate of a stored background of a location where the activated camera is located, and a texture of the background updated in real time. ;
An inflow object three-dimensional coordinate calculator configured to calculate three-dimensional coordinates of the detected external object based on the detected external object and the stored background;
An object deformer configured to map a texture to an object after deforming the three-dimensional coordinates of the external object to match a background;
A background generator for generating a final background based on the texture of the background updated in real time and the three-dimensional coordinates of the stored background;
A final image generator which inserts the texture-mapped object into the final background and merges with the background of an adjacent camera to generate a final image; And
At least one user terminal for receiving and displaying the merged image output from the final image generating unit,
The controller may include the plurality of cameras, the background 3D coordinate calculator, the background texture real time updater, the inflow object detector, the inflow object 3D coordinate calculator, the object deformer, the background generator, and the final object. Virtual video tour system using a real-time video merge, characterized in that for controlling at least one of the image generating unit, and receiving a signal from the user terminal.

Images