KR20110014864A - System for reconstructing 3d video data based on data-flow framework - Google Patents
System for reconstructing 3d video data based on data-flow framework Download PDFInfo
- Publication number
- KR20110014864A KR20110014864A KR1020090072437A KR20090072437A KR20110014864A KR 20110014864 A KR20110014864 A KR 20110014864A KR 1020090072437 A KR1020090072437 A KR 1020090072437A KR 20090072437 A KR20090072437 A KR 20090072437A KR 20110014864 A KR20110014864 A KR 20110014864A
- Authority
- KR
- South Korea
- Prior art keywords
- data
- node
- image
- video data
- frame
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/97—Determining parameters from multiple pictures
Abstract
The present invention relates to a three-dimensional reconstruction system of video data based on a data flow framework, comprising: at least one image capturing means for photographing an object from different locations; At least one image acquisition node for acquiring image data from each of the image capturing means; At least one silhouette data acquisition node for acquiring silhouette data of an object from each of the image capturing means; At least one image control node for acquiring parameter data including a position and an angle of the image capturing means; At least one frame forming node receiving data from each of the image obtaining node, the silhouette data obtaining node, and the image adjusting node to form image frame data; A super frame forming node configured to receive image frame data from each of the frame builder nodes to form composite image frame data; A 3D video data construction node for forming 3D video data based on the composite image frame data formed from the super frame forming node; And a rendering node for drawing three-dimensional video data based on the three-dimensional video data formed by the three-dimensional video data construction node, each of which is connected by a pipeline to sequentially transmit data. Provided is a three-dimensional reconstruction system of video data based on a characteristic data flow.
Description
The present invention relates to a three-dimensional reconstruction system of video data based on a data flow framework, and more particularly, to a system for quickly and efficiently reconstructing three-dimensional video data based on a data flow framework from a plurality of still image data. will be.
Data-flow based or oriented processing has been used in a variety of applications. In particular, it can be very useful in the field of processing a large data stream. For example, it may be used in application fields such as video and audio processing or visualization of engineering calculation results. Systems for reconstructing video data in three dimensions require that multiple synchronized image streams be processed. When configuring a three-dimensional reconstruction system using data flow methods. It is easy to develop, easy to integrate, and flexible to distribute new algorithms and data format tests and results to multiple computer systems for processing. The framework also has the advantage of being flexible, scalable, and easy to add to the system for rapid prototyping, three-dimensional reconstruction algorithms, and application implementation. The data flow method is divided into a series of consecutive processing steps in which each processing step depends only on the output of the previous processing step, and each processing step is called a filter. Users can easily create new applications by reassembling already created filters.
The data-flow method has been used in various systems in various fields such as digital image processing, hardware, music composition, multimedia, operating system, parallel computer, and engineering visualization. The most widely known example is the UNIX system. Pipe concepts, the OpenGL rendering pipeline, and the Visualization Toolkit (VTK), a visualization tool kit. A recent example is Network-Integrated Multimedia Middleware (NMM). NMM is middleware that configures distributed system and connects multimedia system that exists in remote place to network. Users using NMM can set up data-flow graphs with simple text representations for a variety of networked multimedia devices, including MP3 players and TVs as well as software components. The NMM is responsible for distributing and synchronizing multimedia streams between various devices and components. The NMM system can only process multimedia streams such as audio and video, but it cannot process three-dimensional (3D) data. Another recent study is FlowVR, a high-performance interactive application system that works in PC clusters and grid environments. The goal of this system is to apply for the visualization of virtual reality and engineering calculations. The system consists of a data-flow oriented programming approach and core middleware using a parallel rendering library.
An object of the present invention is to provide a flexible, scalable, and easy system expansion structure for fast prototyping, three-dimensional reconstruction algorithm, and application in reconstruction of three-dimensional video data.
It is another object of the present invention to provide a system that is easy to develop, easy to integrate, and that can flexibly distribute and process new algorithms and data types for testing and results. .
According to an aspect of the present invention, there is provided a three-dimensional reconstruction system for video data based on a data flow, the system comprising: at least one image capturing means for photographing an object from different locations; At least one image acquisition node for acquiring image data from each of the image capturing means; At least one silhouette data acquisition node for acquiring silhouette data of an object from each of the image capturing means; At least one image control node for acquiring parameter data including a position and an angle of the image capturing means; At least one frame forming node receiving data from each of the image obtaining node, the silhouette data obtaining node, and the image adjusting node to form image frame data; A super frame forming node configured to receive image frame data from each of the frame builder nodes to form composite image frame data; A 3D video data construction node for forming 3D video data based on the composite image frame data formed from the super frame forming node; And a rendering node for drawing three-dimensional video data based on the three-dimensional video data formed by the three-dimensional video data construction node, each of which is connected by a pipeline to sequentially transmit data. Provided is a three-dimensional reconstruction system of video data based on a characteristic data flow.
According to the present invention, in the reconstruction of three-dimensional video data, it is possible to provide a flexible, scalable and easy system expansion structure for the rapid prototyping, three-dimensional reconstruction algorithm and application implementation.
In addition, according to the present invention, it is easy to develop, easy to integrate, and it is possible to flexibly distribute and process new algorithms and data types of tests and results to multiple computer systems.
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
The present invention describes the basic concepts and definitions of the data flow method and describes the case where such a data flow method is applied to a three-dimensional reconstruction system having a plurality of image streams.
1. Overview
The configuration of the "data flow structure" or "pipe and filter" scheme is very useful when the data stream has to be processed. Basically, this data flow system is divided into a series of successive processing steps where each processing step depends only on the output of the previous processing step. The data flow structure goes through the "pipeline" where "data" delivered from "source" is processed, followed by "filter" and "sink". Steps are connected by data and pipe. A set of piped filters becomes a data processing pipeline. A pipeline consists of a "Data-Flow graph" with sources, filters and sinks as "nodes" and pipes with specific data as "edges." Complex processing is accomplished by processing the data through several simple processing with appropriate continuity. Thus, the data-flow approach can be best used to develop application systems that focus on the "flow" of data. The easiest examples of data-flow-oriented applications are those that process video and audio or visualize engineering calculations.
1 illustrates an example of a simple data-flow pipeline. Referring to FIG. 1, data flows from the source node to the filter and sink nodes. Different colors mean that the data processed or used by each node can be of different kinds.
2. Requirements
Next, the basic requirements for the present invention will be described. Here we will explain what should be possible and what characteristics should have.
First, the present invention assumes the following environment. That is, there is currently a studio for extracting dynamic scenes from the synchronized video stream, and the synchronized images are obtained from a plurality of image capturing means (cameras) installed at various locations. The computer system used in the present invention is composed of a plurality of standard PCs connected by Ethernet, and through a special configuration process, some PCs can be connected to one or more video input cameras having an IEEE1394 interface.
The software portion of the present invention must perform various tasks such as image capture, image processing, three-dimensional reconstruction and representation of an image or reconstructed coordinates. Thus, software components that perform these tasks must be deployed on multiple hosts in the form of distributed applications. In addition, these distributed software components transmit data streams to each other in order to process image-based three-dimensional reconstruction work, so that communication between each component must be properly performed. Developers need to provide a basic foundation for easily adding and extending the functionality of the framework by adding and testing new software components. In addition, a user of the system according to the present invention should provide a method for easily setting up and executing a distributed application program. The individual characteristics of the system according to the invention are as follows.
1) Flexibility and Extensibility
The system according to the present invention should provide a basic platform for easy prototyping, testing and implementation of functions related to image based three dimensional reconstruction and rendering. New modules developed through these tools should be easily fused and structurally appropriately rearranged with the relevant applications in existing systems.
2) Scalability
Scalability is a feature applied when trying to increase the amount of computing resources to process large data sets more effectively. In the system according to the present invention, in order to support a larger number of simultaneous camera input streams, it is necessary to make it easy to distribute and relocate software modules to each host computer or to add a new host computer. The system according to the invention must support the concept of parallel processing of data, tasks and pipelines. That is, the number of hosts and software modules that make up the system according to the present invention is limited only by the available hardware resources.
3) Understandability and Maintainability
The design contents of the system according to the present invention are communicated to developers and users by using specific expressions and patterns. Developers or users who are fluent in patterns will be able to understand the structure and characteristics of basic systems more easily. The use of design patterns is the recommended method for designing organizational and structural systems.
4) Configurability and Usability
The framework and related components used in the system according to the present invention should be easily set up or controllable via, for example, a graphical front end user interface (UI) provided for the user to manipulate all the necessary information in the system.
3. Configuration
Next, a configuration of a data flow based 3D video data reconstruction system according to the present invention that satisfies the requirements described above will be described.
3.1. Nodes
In the system according to the invention, all sources, sinks and processing modules are denoted as "Nodes". In other words, the node means the smallest unit of processing. Each node can have multiple input and output ports. Each port has only one data format, depending on the data extracted from the node or coming through that port. Depending on the number of ports, nodes are classified into six general classes derived from the base node class:
DataSourceNode
; Create a data object and provide it to the processing pipeline.
; It has no input port and has one output port.
DataSinkNode
; Nodes that use data objects passed from the processing pipeline
; It has one input port and no output port.
-ProcessorNode
; Basic filter nodes in the pipeline for data processing
; Each has one input port and one output port.
MultiplexerNode
; Data filter node with multiple input ports. Used to merge data from different nodes
; Can have multiple input ports and one output port.
DemultiplexerNode
; Data filter node with multiple output ports. Used to separate and forward data to different nodes.
; Can have multiple output ports and one input port.
MuxDemuxNode
; Data Filter Nodes with Multiple Input and Output Ports, Each
; Can have multiple input and output ports.
2 illustrates a hierarchical structure of each node as described above. Real nodes are created as subclasses of the above general node type. Thus, each real node typically operates in the form of an active component within its own thread. Each node can also be grouped in the form of a "CompositeNode" class that acts as a container for multiple nodes with a mixed pattern. Nodes in a group do not operate as active components, but are controlled by a mixed thread that reduces the number of concurrent threads.
3.2 Tokens
Each node as described above exchanges information while transmitting / receiving a message. These messages, called "tokens", take the form of "payloads" and are structured to hold all kinds of information. Tokens are "self-identifying objects" that support an abstract model, such as the kind of information they have, so that they can be easily distinguished by each node. Tokens consist of two elements: "descriptor" and "data". The "descriptor" contains general information such as the type and format of the token. "Data" has a special kind of data, and in some cases, a token may not have any "data". For example, there may be two kinds of basic tokens, "Event" and "Data" as follows.
Event
Token representing control information passed along the pipeline. Node classes either use the event token associated with them or pass an unused event token to another node.
Data
Token that holds the data processed by the node. The data token may include various general types of data such as an image, a volume, and a mesh. Subclasses of this class explicitly specify the actual "Data Format" provided by the node's input and output ports. It provides a way to serialize (or vice versa) from byte streams for transmission within the network of tokens.
3.3. Pipe
The pipe is connected to one output port of one node and then to one input port of the other node connected to it. The output port of the node that is connected to the pipe must be a port that is compatible with the input port of another node that is subsequently connected (which carries the same kind of data). This means that the data format (or Plug-Format) supported by the port of each node connected to the pipe must be the same. Pipes only support one form of node-to-node connection. A pipe is implemented as a FIFO queue of token objects. Tokens that are later allowed to be added in the form of new tokens are not interpreted by the pipe.
3 shows a queue connection between two nodes. The producer node continuously fills the pipe while the consumer node reads data from the pipe. Thus, the pipe will perform synchronization between two active components. Tokens are only passed to the "reference" object to avoid the overhead of "copy".
3.4. Pipeline Segment
A pipeline segment is a portion of a distributed pipeline that is placed on a particular host. It is seamlessly connected to the data-flow pipeline from the source to the sink. Therefore, the minimum pipeline segment is a form in which one source node is connected to one sink node. Multiple nodes must be added to the pipeline segment that is part of the distributed pipeline. In order to operate as an intact pipeline segment, each node in the segment must be properly connected, and each node's ports must be connected to a particular pipe port.
3.5. Distributed Pipeline
The distributed pipeline consists of at least one pipeline master and a plurality of pipeline slaves. Each slave manages one pipeline segment. Segments in a distributed pipeline are connected through special sources and sink nodes in the form of a network sink node of a particular segment connected to the network source node of the segment to which it is connected. The pipeline master is responsible for managing the entire distributed pipeline. The master reads the pipeline configuration file and checks the consistency of all nodes for proper connection. The master uses the configuration information to initialize the slaves and to control them during runtime. The master can also manage one pipeline segment.
4 shows a simple example of a distributed pipeline. The master reads the configuration information of the entire pipeline, inspects it, evaluates it, and distributes it to the connected slaves. Each slave configures its own pipeline segment using the configuration information, and connects the network sink node to the network source node of the segment connected later. The user of the system according to the invention controls the master through a graphical user interface to start the pipeline and view status information between the master and the slave.
3.6. Pipeline Configuration
To create a new distributed pipeline, we define a data-flow graph description format (.pipe), for example in XML. This format consists of two main parts. In the first part, pipeline connection information related to how each node and port are connected to each other is set. In the other part, the characteristics of the node configured in each node are included. For the convenience of the user, the use of visual programming tools will facilitate the creation of data-flow graphs. This visual programming tool can be used to simply connect individual modules and set their parameters without any programming or editing the XML file directly.
3.7. Node Development
To simplify node development, we defined a node descriptor format (.node) in XML format. The developer writes the supported data formats and specific attributes of the new node in the description file (.node). The node compiler will generate code with a basic skeleton that can be integrated into the framework, for example in the form of a C ++ stub class. To implement the functional part of the node, you must implement a subclass of the generated stub class. The generated node class is then compiled into a shared library and stored in a specific plug-in directory so that it can be loaded at runtime.
FIG. 5 illustrates an example of processing actual data by a pipeline based on a plurality of nodes and a data flow between each node in the data flow framework.
4. Specific Configuration Example
Based on the data flow framework as described above, a three-dimensional reconstruction system of volume-based video data is implemented. The system used images extracted from eight synchronized video streams to reconstruct the Voxel model of a dancing female actor. The system according to the present invention consists of a pipeline which connects dozens of nodes and nodes which perform various tasks such as reading and processing an image, performing a lot of processing such as background separation, JPEG encoding and decoding. 6 is a block diagram showing an example of such a system according to the present invention.
Referring to FIG. 6, the a_segment and the b_segment each show the configuration of respective nodes that acquire / set the image data, the silhouette data, and the parameter data by four image capturing means. That is, for each of the image capturing means (cameras), at least one or more image capturing nodes ImageReaderCamera1 ... 8 for obtaining image data from each of the image capturing means, and the silhouette data of the object from the image capturing means, respectively. At least one silhouette data acquisition node (SilhouetteReaderCamera1 ... 8) and at least one image adjustment node (CalibReaderCamera1 ... 8) for acquiring parameter data including the position and angle of the image capturing means. Each of the image acquisition nodes includes an image encoder node (JPEGEncoderCamera1 ... 8) and an image transmission node (ImagesenderCamera1 ... 8) for transmitting the image data processed by the image encoder node to a frame forming node described later. It may include.
The silhouette data acquisition node may also include a node (BRLEEncoderCamera1 ... 8) and a silhouette data transmission node (SilhouetteSenderCamera1 ... 8) for data compression, and the image control node is a transmission node for transmitting image parameter data. (CalibsenderCamera1 ... 8).
In addition, at least one frame forming node (FrameBuilderCamera1... 8) that receives data from each of the image acquisition node, the silhouette data acquisition node, and the image control node to form image frame data is included in the c_segment. As illustrated, the frame forming node may include a node ImageReceiverCamera1 ... 8 for receiving image data and a decoding node JPEGDecoderCamera1 ... 8 for decoding.
In addition, a super frame building node (SuperFrameBuilder) for receiving the image frame data from each of the frame forming nodes to form the composite image frame data is provided, and based on the composite image frame data formed from the super frame forming node 3D video data The 3D video data construction node VoxelReconstructor may be connected. The 3D video data formed by the 3D video data composition node may be transmitted to the rendering node by a transmission node VoxelVolumeSender, and the rendering node renders the 3D video data based on the received 3D video data. . The rendering node may include a receiving node VoxelVolumeReceiver and a forwarding node VoxelForwarder.
In FIG. 6, the segments may be physically separate computer systems, but are not limited thereto.
Meanwhile, FIG. 7 illustrates a voxel model of a dancing female actress who is processed and finally reconstructed by the system according to the present invention as shown in FIG. 6, and each node and pipe are sequentially formed by the data flow based framework as shown in FIG. The data processed along the lines shows the final state rendered in three dimensions.
1 illustrates an example of a simple data-flow pipeline.
2 shows a hierarchical structure of each node.
3 shows a queue connection between two nodes.
4 shows a simple example of a distributed pipeline.
5 illustrates an example of processing actual data by a pipeline based on a plurality of nodes and a data flow between each node in a data flow framework.
6 is a block diagram showing an example of a system according to the present invention.
FIG. 7 illustrates a final state rendered in three dimensions by data processed along each node and pipeline sequentially by the data flow based framework according to the present invention.
Claims (1)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020090072437A KR20110014864A (en) | 2009-08-06 | 2009-08-06 | System for reconstructing 3d video data based on data-flow framework |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020090072437A KR20110014864A (en) | 2009-08-06 | 2009-08-06 | System for reconstructing 3d video data based on data-flow framework |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| KR20110014864A true KR20110014864A (en) | 2011-02-14 |
Family
ID=43773857
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| KR1020090072437A KR20110014864A (en) | 2009-08-06 | 2009-08-06 | System for reconstructing 3d video data based on data-flow framework |
Country Status (1)
| Country | Link |
|---|---|
| KR (1) | KR20110014864A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2021532497A (en) * | 2019-04-11 | 2021-11-25 | 平安科技(深▲せん▼)有限公司Ping An Technology (Shenzhen) Co., Ltd. | Visualization method, equipment and recording medium for multi-source earth observation image processing |
-
2009
- 2009-08-06 KR KR1020090072437A patent/KR20110014864A/en not_active Application Discontinuation
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2021532497A (en) * | 2019-04-11 | 2021-11-25 | 平安科技(深▲せん▼)有限公司Ping An Technology (Shenzhen) Co., Ltd. | Visualization method, equipment and recording medium for multi-source earth observation image processing |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113255170B (en) | Cloud-edge cooperative factory digital twin monitoring modeling system and modeling method | |
| KR101126255B1 (en) | Signal processing apparatus | |
| KR20060103131A (en) | Signal processing apparatus | |
| CN103514620B (en) | 3D 3-D cartoon whole process makes cloud computing platform | |
| JP7392136B2 (en) | Methods, computer systems, and computer programs for displaying video content | |
| EP2126851A1 (en) | Graphics rendering system | |
| Ohlenburg et al. | The MORGAN framework: enabling dynamic multi-user AR and VR projects | |
| CN103927779B (en) | Method for generating two-dimensional animation on basis of configuration | |
| De Mey et al. | A multimedia component kit: experiences with visual composition of applications | |
| Friston et al. | 3DRepo4Unity: Dynamic loading of version controlled 3D assets into the Unity game engine | |
| CN115964185A (en) | Micro-service management system for technical resource sharing | |
| KR20110014864A (en) | System for reconstructing 3d video data based on data-flow framework | |
| CN102929592A (en) | Three-dimensional interaction method and device for equipment based on monitoring system | |
| Schiefer et al. | Service-oriented scene graph manipulation | |
| KR20130035485A (en) | System for publishing 3d virtual reality moving picture and method for publishing the same | |
| Repplinger et al. | DRONE: A flexible framework for distributed rendering and display | |
| Aspin | Supporting collaboration, in colocated 3D visualization, through the use of remote personal interfaces | |
| Kessler et al. | RAVEL, a support system for the development of distributed, multi-user VE applications | |
| Repplinger et al. | URay: A flexible framework for distributed rendering and display | |
| Lesage et al. | High performance interactive computing with FlowVR | |
| Lei et al. | Design and implementation of a cluster based smart camera array application framework | |
| US20230146230A1 (en) | Immersive media analyzer for reuse of scene assets | |
| Valkov et al. | Vins: Shared memory space for definition of interactive techniques | |
| Hoffmann et al. | Integration of realtime ray tracing into interactive Virtual Reality systems | |
| Green | An Interface for Real-Time Networked Streaming of 3D Scenes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| WITN | Withdrawal due to no request for examination |