KR20110110895A - System for fusing realtime image and context data by using position and time information - Google Patents

Abstract

The present invention relates to a real-time video and contextual text data fusion system that fuses sensing data and images collected from a sensor network using an index. Real-time video and contextual text data fusion system according to the present invention, the sensing data collection unit for extracting the position and sensor index from the sensing data provided from the sensor network; An image collecting unit extracting a position and an image index from the image information provided from the imaging apparatus; An index processing unit for generating an integrated index by integrating the sensor index extracted by the sensing data collector, the image index extracted by the image collector, position and time information; A unitary index storage unit for storing a unitary index generated by the index processor; A sensing data storage unit for storing sensing data provided from the sensor network via the sensing data collector; An image data storage unit for storing image information provided from the imaging apparatus via the image collecting unit as image data; And import sensing indexes from the unified index storage unit, and connect sensing data and image data stored in the sensing data storage unit and the image data storage unit with the retrieved unified index, respectively. And an index connection unit for storing the sensing data storage unit and the image data storage unit.

Description

Real-time video and contextual text data fusion system using location and time information {SYSTEM FOR FUSING REALTIME IMAGE AND CONTEXT DATA BY USING POSITION AND TIME INFORMATION}

The present invention relates to a sensor web technology for controlling and displaying data collected from a sensor network on a web. More specifically, the present invention provides a method of integrating sensing data collected from a sensor network and an image collected from an imaging device based on location and time information. The present invention relates to a real-time video and contextual text data fusion system that fuses using an index and enables searching of images and sensing data.

The concept of ubiquitous computing was first proposed in 1988 by Mark Weiser of the Palo Alto Research Center (PARC) of XEROX. According to this ubiquitous computing concept, a human-centered computing environment that emphasizes the use and connection of computers by embodying computers that exist everywhere and do not seep into the environment or objects is emphasized.

In order to realize a ubiquitous computing environment, sensor networks have been established in various fields and real-time situation information is collected. Recently, researches for transmitting sensing data collected from the sensor network to the general user through the Internet have been conducted. In addition, research has been attempted to display sensing data on a map in connection with a geographic information system (GIS) as well as a text or graph form.

As part of this research, OGC (open geospatial consortium) sensor web technology is representative. The sensor web is a new paradigm that connects the physical world and cyber space, and displays stored sensing data using various sensors, facilities, and imaging devices. Web application technology that can be controlled.

Currently, sensor web technologies that display data collected from unit sensor networks on the web include service oriented sensor webs, standard representations or protocols of sensing data, sensing data, and the like at Microsoft, OGC, and Melbourne University, Australia. The researches on the structure of searching and managing nodes have been conducted. In order to display the sensing data and the image collected from the imaging apparatus in such a sensor web, a fusion between the sensing data and the image data is required.

According to the related art, when the client receives the image information and the sensing data, the information collected from the sensor network and the image system are displayed separately, so that it is difficult to decipher the relation between the provided image information and the sensing data. There was this. As a result, it is difficult to effectively provide both the situation sensing data and the image information generated at the same time or place to the user.

The present invention is to achieve the technical problem as described above, real-time video that integrates the sensing data collected from the sensor network and the image information collected from the imaging device to the user using location and time (timestamp) information and It is an object to provide a contextual text data fusion system.

Real-time video and contextual text data fusion system according to an aspect of the present invention, the sensing data collection unit for extracting the position and sensor index from the sensing data provided from the sensor network; An image collecting unit extracting a position and an image index from the image information provided from the imaging apparatus; An index processor for generating a unified index by integrating a sensor index extracted by the sensing data collector, an image index extracted by the image collector, and position and time information; A unitary index storage unit which stores a unitary index generated by the index processor; A sensing data storage configured to store sensing data provided from the sensor network via the sensing data collector; An image data storage unit for storing image information provided from the image apparatus via the image collecting unit as image data; And import sensing indexes from the unified index storage unit, and connect sensing data and image data stored in the sensing data storage unit and the image data storage unit with the retrieved unified index, respectively. And an index connection unit for storing the sensing data storage unit and the image data storage unit.

According to the real-time video and contextual text data fusion system using the location and time information of the present invention, the sensor index extracted from the sensing data collected from the sensor network, the image index extracted from the image information collected from the imaging device, the position and time information By generating a unified index using, it is possible to effectively fuse the real-time image information and the sensing data for the node in the sensor network to provide to the user, the user can easily decipher the relationship between the image information and the sensing data Can be achieved.

1 is a block diagram illustrating a real-time video and contextual text data fusion system according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a process of collecting and storing images and sensing data in the fusion system of FIG. 1.
FIG. 3 is a diagram illustrating an image and sensing data query process performed by a client in the fusion system of FIG. 1.
FIG. 4 is a diagram illustrating a schema structure of a unified index table used in the unified index storage 40 of the fusion system shown in FIG. 1.
FIG. 5 is a diagram illustrating a process of displaying sensing data and image data in the display module 82 of the fusion system illustrated in FIG. 1.
6 is a diagram illustrating a process of moving and updating a node component in a fusion system according to an embodiment of the present invention.
FIG. 7 is a diagram illustrating the imaging device 12 illustrated in FIG. 1.
FIG. 8 is a diagram illustrating an administrator application screen by executing the sensor web application 81 shown in FIG. 1.
FIG. 9 is a diagram illustrating a user application screen by executing the sensor web application 81 shown in FIG. 1.
FIG. 10 is a diagram illustrating an image and sensing data display screen of the first sensor node by executing the sensor web application 81 shown in FIG. 1.
FIG. 11 is a diagram illustrating an image and sensing data display screen of a second sensor node by executing the sensor web application 81 shown in FIG. 1.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be embodied in various different forms, and only the embodiments are intended to complete the disclosure of the present invention and to those skilled in the art. It is provided for complete information.

In the following description of the embodiment, "real-time video and contextual text data fusion system" is briefly abbreviated as "fusion system", but it will be apparent to those skilled in the art that these terms refer to the same object.

First, the configuration and operation of a real-time video and contextual text data fusion system according to an embodiment of the present invention will be described with reference to FIG. 1.

As shown in FIG. 1, the real-time video and contextual text data fusion system according to an embodiment of the present invention receives an output signal from the sensing data collector 21 and the imaging device 12 connected to the sensor network 11. Index processing unit 30 that incorporates an image collector 22, a sensor index table 31, and an image index table 32, an integrated index storage unit 40, an index connection unit 50, and a sensing data storage unit 61. And an image data storage 62, a node manager 70, a sensor web application 81, a display module 82, and a map manager 83.

The sensor network 11 is a collection of sensors for measuring a specific physical quantity, for example, humidity, temperature, light intensity or power capacity, etc. at a plurality of nodes in the observation target space. Each sensor of the sensor network 11 outputs the position information of the sensors together with the data about the measured physical quantity. The imaging device 12 is installed at each of the plurality of nodes to provide a real time image. The imaging device 12 outputs the location information of the imaging device 12 together with the captured real-time image data.

The sensing data detected from each sensor of the sensor network 11 is provided to the sensing data collector 21, and the sensing data collector 21 extracts the sensor index and the position information from the sensing data to generate a sensor index table ( 31 and at the same time, the sensing data is output to the sensing data storage 61.

Similarly, the image information photographed by the image device 12 is provided to the image collecting unit 22, and the image collecting unit 22 extracts the position and the image index from the image information to the image index table 32. The video information is outputted to the video data storage 62.

The index processing unit 30 generates an integrated index by integrating the position, time, sensor index, and image index stored in the sensor index table 31 and the image index table 32, respectively. The index processor 30 outputs the generated unified index to the unified index storage 40.

The index connecting unit 50 retrieves the integrated index from the integrated index storage 40 and connects the sensing data and the image data stored in the sensing data storage 61 and the image data storage 62 with the imported integrated index. To do the job. The index connection unit 50 outputs the sensing data and the image data connected to the integrated index to the sensing data storage 61 and the image data storage 62, respectively, and stores them.

The sensor web application 81 is executed by an operation of a user or a client, and interfaces a user's or client's access to sensing data and image data provided from the sensor network 11 and the image device 12.

When a user or client inputs a data request command for a node to be observed through the sensor web application 81, the node manager 70 transmits a data request command to the index connector 50.

The index connection unit 50 transmits the integrated index information to the node manager 70 in response to the data request command, and the node manager 70 transmits the sensing data and the image data based on the responded integrated index information. 61 and the image data storage 62, and the sensing data and the image data are fused to a map provided from the map manager 83 to the display module 82.

The display module 82 visually displays the sensing data and the image data fused on the map provided by the node manager 70 to the user or the client to enable complex observation of the sensing data and the image data.

Next, a process of collecting and storing the image and the sensing data performed in the fusion system of FIG. 1 will be described with reference to FIG. 2.

FIG. 2 is a diagram illustrating a process of collecting and storing images and sensing data in the fusion system of FIG. 1.

The sensing data and position information detected by each sensor are provided from the sensor network 11 to the sensing data collector 21, and the sensing data collector 21 extracts sensor node indexes and position information from the sensing data. It outputs to the sensor index table 31 of the index processing part 30. FIG. At the same time, the sensing data collector 21 outputs the sensing data provided from the sensor network 11 to the sensing data storage 61.

Similarly, the image information and the position information photographed by the image device 12 are provided to the image collecting unit 22, and the image collecting unit 22 extracts the image index and the position information from the image information and index processing unit. The video is output to the video index table 32 at 30. At the same time, the image collector 22 outputs the image information provided from the image device 12 to the image data storage 62.

Next, a process of querying the image and the sensing data by the user or the client using the sensor web application will be described with reference to FIG. 3.

FIG. 3 is a diagram illustrating an image and sensing data query process performed by a client in the fusion system of FIG. 1.

First, the user or client executes the sensor web application 81 to request the image and sensing data from the node manager 70. Correspondingly, the node manager 70 transmits a data request command to the index connector 50 to obtain the combined index information, and the index connector 50 sends the integrated index information corresponding to the data request to the node manager 70. Answer.

Next, the node manager 70 requests the sensing data storage unit 61 for sensing data corresponding to the obtained integrated index information, and obtains the sensing data from the sensing data storage unit 61.

As described above, since the sensing data storage 61 stores sensing data to which the unified index is connected, the sensing data storage 61 may easily output the sensed data corresponding to the input unified index information. The node manager 70 provides the sensing data obtained as described above to a user or a client.

Meanwhile, the node manager 70 requests the image data storage 62 for image data corresponding to the integrated index information obtained from the index connector 50, and obtains the image data from the image data storage 62. The node manager 70 provides the image data obtained as described above to a user or a client.

As described above, the real-time video and contextual text data fusion system according to the embodiment of the present invention generates the integrated index information from the sensor index extracted from the sensing data, the image index extracted from the image data, and the position and time information, Based on this integrated index information, it is possible to search and output image information and sensing data by a user or a client.

Next, the integrated index employed in the present invention will be described in more detail with reference to the diagram of FIG. 4.

Consolidated indexes are designed in the form of tables in a database. 4 illustrates a schema structure of the node information table which is the core of the unified index.

In FIG. 4, each record expressed as "column name" is information about each node to be displayed on the screen of the sensor web application, and its data format, length, nullability, options, and descriptions are described.

In the real-time video and contextual text data fusion system according to an embodiment of the present invention, the index key information NI_SC_Code in the unified index table of FIG. 4 and the video device 12 installed in the node, that is, the IP and port of the webcam. In addition to the NI-WebCamURL information, location information and time information are integrated and used as one index key information (NI_II_Code). In the sensor web application, node information can be searched using this index key information.

Next, a process of displaying sensing data and image data by the display module 82 in the fusion system according to an exemplary embodiment of the present invention will be described with reference to FIG. 5.

In the execution of the sensor web application 81 of FIG. 1, when a user or a client selects a specific node component and requests information of the corresponding node, the sensor web application 81 assigns the node ID of the integrated index to the node manager 70. To pass on. In FIG. 5, a is a node ID, b is an index, and Request_Event means a request for information of a corresponding node.

The node manager 70 transfers the node ID transmitted from the sensor web application 81 to the index connector 50, and the index connector 50 retrieves the integrated index information corresponding to the node ID, and retrieves the index information. )

The node manager 70 accesses the sensing data storage 61 and the image data storage 62 by using the integrated index information provided from the index connector 50, and the image information and the sensing data corresponding to the integrated index information. Bring up The node manager 70 transmits the image information and the sensing data thus obtained to the display module 82 so that the node manager 70 displays the image information and the sensing data so as to display the image information and the sensing data.

Next, a process of moving and updating a node component in a fusion system according to an embodiment of the present invention will be described with reference to FIG. 6.

The movement and update process of the node component shown in FIG. 6 is executed by the sensor web application 81. According to the procedure of moving and updating a node component of the sensor web application 81, the node manager 70 retrieves node information from the integrated index storage 40 and sets information of the corresponding node component according to the retrieved node information. do.

When a node movement event occurs by the user or client selection for the node component, the node manager 70 searches for the modified information according to the node ID, and transmits the modified node information to the node component. In FIG. 6, a is node information and b is modified node information.

When a node information storage event occurs by a user or a client selection for the node component having the modified node information, the node manager 70 refers to the node ID of the node information table and stores the modified node information in the node information table. Update.

FIG. 7 illustrates an embodiment of the imaging device 12 shown in FIG. 1.

In the implementation environment of the imaging device 12 illustrated in FIG. 7, a .net based C # language is used in the Windows XP operating system, and Logitech Corporation's QuickCam is used as the imaging device 12. The image device 12 illustrated in FIG. 7 is connected to a computer using a USB interface and supports a maximum 1.3 million pixel image and a frame rate of 30 frames per second. Since the implementation of the imaging device 12 is only one example, the imaging device 12 may be replaced with an imaging device having different specifications, and the technical scope of the present invention is not limited thereto.

FIG. 8 is a diagram illustrating an administrator application screen by executing the sensor web application 81 shown in FIG. 1.

As shown in FIG. 8, in the manager application screen of the sensor web application 81, an administrator may create a node and correct a location. In order to perform the node creation role, after passing the administrator authentication on the sensor web application 81, it is possible to generate a node to collect the sensing data and image information. By executing the sensor web application 81, the administrator generates a node and sets the position of the node to a location where the sensing data and the image information are collected, and the index key information of the sensor information table is set according to the node ID of the node information table. In addition to NI_SC_Code and NI_WebCamURL, which is IP and Port information, location information and time information can be used as one index information.

FIG. 9 is a diagram illustrating a user application screen by executing the sensor web application 81 shown in FIG. 1.

As shown in FIG. 9, a user may observe a node in which an administrator is created in the user mode of the sensor web application 81. In the provided node, sensing data and image information are integrated into index key information according to each node ID.

FIG. 10 is a diagram illustrating an image and sensing data display screen of the first sensor node by executing the sensor web application 81 shown in FIG. 1.

Referring to FIG. 10, information about a real time image and sensing data of a corresponding node is displayed on a screen provided by the display module 82 by the execution of the sensor web application 81. In the window located in the middle of the screen of FIG. 10, sensing data about humidity, temperature, intensity, and power capacity are displayed along with a real-time image of the selected node.

For example, a user or a client selects a node component that he / she wants to know from among multiple nodes created by an administrator, generates an information request event for the node component, and the node of the node through the process described with reference to FIG. 5. The image information and the sensing data are displayed on the display module 82, whereby the user can compositely observe the real-time image and the sensing data of the corresponding node.

FIG. 11 is a diagram illustrating an image and sensing data display screen of a second sensor node by executing the sensor web application 81 shown in FIG. 1.

Referring to FIG. 11, when a user selects another node component to generate an information request event for the corresponding node component when the sensor web application 81 is executed, an integrated index key corresponding to the node ID of the corresponding node information table. It can be seen that the image information and the sensing data retrieved using the information are displayed on the display module 82.

The real-time video and contextual text data fusion system of the present invention described above, by combining the sensing data collected in the sensor network and the image information collected from the imaging device using an integrated index, the real-time video for the node in the sensor network The image information and the sensing data can be effectively fused and provided to the user, and the user can achieve the effect of easily decoding the relation between the image information and the sensing data.

So far, the configuration and operation of the embodiment of the present invention have been described. The present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the claims but also by the equivalents of the claims.

11 sensor network 12 image device
21: sensing data collector 22: image collector
30: index processing unit 31: sensor index table
32: video index table 40: integrated index storage unit
50: index connection 61: sensing data storage
62: image data storage unit 70: node management unit
81: sensor web application 82: display module
83: Map Management Department

Claims (11)

A sensing data collector extracting a position and a sensor index from sensing data provided from a sensor network;
An image collecting unit extracting a position and an image index from the image information provided from the imaging apparatus;
An index processing unit for generating an integrated index by integrating the sensor index extracted by the sensing data collector, the image index extracted by the image collector, position and time information;
A unitary index storage unit for storing a unitary index generated by the index processor;
A sensing data storage unit for storing sensing data provided from the sensor network via the sensing data collector;
An image data storage unit for storing image information provided from the imaging apparatus via the image collecting unit as image data; And
Import the unified index from the unified index storage unit and connect the sensing data and the image data stored in the sensing data storage unit and the image data storage unit with the retrieved unified index, and connect the sensing data and the image data connected with the unified index. And an index connector configured to store the sensing data storage unit and the image data storage unit.
Real-time video and contextual text data fusion system.
The method of claim 1,
The sensor network includes a set of sensors for measuring a specific physical quantity at a plurality of nodes in the observation target space including location information.
Real-time video and contextual text data fusion system.
The method of claim 1,
The index processing unit
A sensor index table for storing a position and a sensor index provided from the sensing data collector; And
And an image index table for storing the position and the image index provided from the image collecting unit.
Real-time video and contextual text data fusion system.
The method of claim 1,
The index connection unit searches the unified index storage unit according to a data request command from a user and provides corresponding unified index information.
Real-time video and contextual text data fusion system.
The method of claim 1,
The unified index table is configured by integrally using the location and time of the index key information (N_II_Code), the NI_SC_Code, and the NI_WebCamURL, which is the IP and port information of the video device installed in the node, as one index key information (NI_II_Code).
Real-time video and contextual text data fusion system.
A sensing data collector extracting a position and a sensor index from sensing data provided from a sensor network;
An image collecting unit extracting a position and an image index from the image information provided from the imaging apparatus;
An index processing unit for generating an integrated index by integrating the sensor index extracted by the sensing data collector and the image index extracted by the image collector;
A unitary index storage unit for storing a unitary index generated by the index processor;
A sensing data storage unit for storing sensing data provided from the sensor network via the sensing data collector;
An image data storage unit for storing image information provided from the imaging apparatus via the image collecting unit as image data;
The integrated index is retrieved from the integrated index storage unit, and the position and time information, the sensing data, and the image data stored in the sensing data storage unit and the image data storage unit are respectively connected to the imported integrated index, and the integrated index is connected. An index connector configured to store data and image data in the sensing data storage unit and the image data storage unit;
A sensor web application executed by a user's or client's operation to interface a user's or client's access to sensing data and image data; And
When a user or a client selects a node to be observed through the sensor web application, a data request command for the node is transmitted to the index connection unit to receive corresponding index information, and based on the received integrated index information, And a node manager configured to retrieve corresponding sensing data and image data from the sensing data storage unit and the image data storage unit.
Real-time video and contextual text data fusion system.
The method according to claim 6,
The sensor network includes a set of sensors for measuring a specific physical quantity at a plurality of nodes in the observation target space.
Real-time video and contextual text data fusion system.
The method according to claim 6,
The index processing unit
A sensor index table for storing a sensor index provided from the sensing data collector; And
An image index table for storing an image index provided from the image collecting unit;
Real-time video and contextual text data fusion system.
The method according to claim 6,
The index connection unit searches for the unified index storage unit and provides corresponding unified index information to the node manager according to a data request command input from the node manager by a user's manipulation of the sensor web application.
Real-time video and contextual text data fusion system.
The method according to claim 6,
When a node component movement and update procedure is executed by a user's manipulation of the sensor web application, the node manager retrieves node information from the unified index storage unit, and retrieves node information according to the retrieved node information. To set
Real-time video and contextual text data fusion system.
The method according to claim 6,
A map manager providing map information for displaying nodes and sensing data to the node manager; And
When the node manager outputs map information, sensing data and image data for user display, the node manager further includes a display module that visually displays the user by fusing the sensing data and image data on the map.
Real-time video and contextual text data fusion system.

Images