KR20080078217A - Method for indexing object in video, method for annexed service using index of object and apparatus for processing video - Google Patents

Abstract

A method for indexing an object in a video, an annexed service method using information of the object, and an apparatus for processing the video are provided to analyze a screen size of the video and frame information, to re-encode the video for a streaming service, and to set the uppermost cell split number and the maximum cell split number for judging the position of the object which is set by a user regardless of a resolution and a screen size. A method for seeking video information comprises the steps of: checking whether video information of an analysis object exists in a video information storage(S110); discriminating whether the video needs a format and a codec when the video exists, selecting a proper codec, and analyzing the video(S111,S120); checking a video header and key frame information(S121); analyzing a screen size of the video and the frame information(S122); judging whether an original video needs the encoding for the service codec(S130); converting or re-encoding the video for a streaming service(S131); setting the uppermost cell split number and the maximum cell split number(S140); and storing the uppermost cell split number and the maximum cell split number(S150).

Description

OBJECT AND APPARATUS FOR PROCESSING VIDEO} METHOD FOR INDEXING OBJECT IN VIDEO, METHOD FOR ANNEXED SERVICE USING INDEX OF OBJECT AND APPARATUS FOR PROCESSING VIDEO}

1 is a block diagram of an image server and a user terminal.

2 is a frame and object analysis data table.

3 is an object service metadata table and a feature table.

4 is a diagram illustrating a relationship diagram between image object data and service metadata.

5 is a processing procedure diagram of the present invention.

6 is a flowchart of image information retrieval and processing.

7 is a flowchart of a process for analyzing an image image.

8 is a flowchart illustrating a process of storing an object identification result.

9 is an example of object analysis.

10 is an exemplary diagram of a sub-analysis cell processing of an object.

11 is an exemplary diagram of cell division of a server.

12 is an object identification and polygon recognition procedure.

13 is a scene division processing procedure diagram.

14 is an object qualification procedure procedure diagram.

15 is an object logic association diagram.

16 is a procedure for creating a service profile.

17 is a service search process and procedure diagram.

18 is an example of processing for a binary query.

19 illustrates an example of generating and processing a terminal control command.

20 is an exemplary view of a service editor for metadata, feature information, and service management.

21 is a view illustrating a video service interface of a user terminal.

22 illustrates a service embodiment of an interactive image retrieval terminal using a mobile terminal.

The present invention relates to an image processing apparatus and a method thereof, and more particularly, to a method of indexing an object included in an image, and a method and apparatus for providing an additional service using the object information.

Due to the development and popularization of communication technology, communication lines have been built up to the home, and users can access the desired internet site anytime and get the necessary information when they need it. By using it, the Internet is used as a marketing tool.

Moreover, a video device such as a television also has a communication function on its own or a communication function through an external device (eg, a set-top box, etc.), so that both a computer and a video device can communicate with each other. This enabled operators to use video equipment as a marketing tool. That is, operators have proposed a method of adding a product information to a broadcast signal received by a video device and selecting the product information to use for marketing a desired product.

In the conventional method for providing product information using a broadcast signal, a user who recognizes an object to be a product target from the image information to be broadcast separately inputs the product information, and then transmits the product information and the image information together, thereby including the product included in the image information. The method of providing information to the viewer was used. That is, in the related art, a person has to intervene to recognize an object requiring product information among objects included in a specific image.

As a result, in order to recognize an object included in a specific image, the user has to repeatedly listen to the image. In addition, it is difficult to objectively analyze the image because the product target is determined according to the subjective judgment of the person who wants to input the product information.

The present invention provides an object indexing method and apparatus for automatically recognizing an object included in a multimedia.

In addition, the present invention manages the virtual frames and cells for managing and storing the relative position of the objects included in the image to manage the position of the position specified by the viewer on the display device regardless of the resolution or screen size of the device on which the image is displayed A method and an apparatus for accurately determining an object are provided.

Another object of the present invention is to provide a method and apparatus for retrieving various and intelligent objects and information through assigning various property attributes of an object to indexed object information and calculating the attribute values.

In order to achieve the above object, the object indexing method provided by the present invention includes: detecting a basic screen size for an image; Dividing the screen into a plurality of virtual cells based on preset split information, and setting the split information into cells; Analyzing image information of each of the divided cells and storing cell mapping information and image analysis information; Identifying a set of adjacent cells among cells having the same image analysis information as one object; Analyzing the object identification information and indexing the object as an object; Classifying a scene using analysis information such as an object and a background identified using image identification information of an image frame; and adding object property information of the object to stored object information and assigning an association information search rule to create an object profile. And generating a service profile for providing various dynamic service methods in a method of providing a service.

In the present invention, in the multimedia to provide a variety of additional services and search methods for generating a search rule through the operation of the attribution information and feature information; providing the index information through a network; and inputs the user's search information And; interpreting the input search information; and searching and transmitting the corresponding corresponding service; and comprising a multimedia server and a terminal for providing the service; and a wireless and wired network. It characterized in that it comprises a control command interface method for providing a dynamic interface on a variety of networks.

In the detecting step, the pixel size of the image screen extracted from the image is checked to detect the size of the screen, and the pixel unit suitable for the split information is determined based on the screen width and the height of the screen based on the preset split information. Characterized in that it comprises the extraction.

If the pixel information according to the size of the screen is obtained in the step of setting the split information as a cell, the pixel coordinates in the frame allocated to each cell area are mapped in the process of setting the relative split information as the cell. The coordinate information of the cell and the mapping information of the cell is characterized by setting to a relative position value.

In the dividing process, the frames are hierarchically divided and the number of divided virtual cells is a multiple of two.

In addition, the dividing process divides the image frame into virtual cells by a specific number, and each cell divides the subcells by a multiple of 2 and repeats the cell dividing step, which is repeatedly designated.

Each subcell divided into the cells preferably has a rule for mapping pixel unit coordinate information in a target image to a virtual cell.

In this case, it is preferable that the frame analyzing the image in the frame analyzes frames at a predetermined interval based on time or frame identifier.

The dividing process in the step of setting the segmentation information to a cell is performed simultaneously with the process of analyzing the video image of the frame. In this case, the pixel information image analysis information of the highest cell has a single analysis information or a plurality of analysis information. If a plurality of pieces of analysis information exist in the cell is determined, the corresponding cell is divided into multiples of two.

In the method of analyzing a pixel of a coordinate region corresponding to the cell, if the analysis information of the cell is plural, the current cell is divided.

The cell image information analysis process in the step of storing the analysis information is analyzed using analysis information such as color, texture, and outline detection. If a plurality of analysis information exist in one selected cell, the cell is analyzed. The subcell is divided into multiples of two, for example, divided into four or eight, and the divided images are analyzed in the same manner.

In the above-described image analysis process, it is preferable to divide and analyze up to a specified segmentation depth until a single image analysis property is detected.

Preferably, the analysis information of the video image is stored together with the divided subcell and the image analysis information.

In the object identification step, the adjacent cells corresponding to the cells are sequentially selected using the divided cell mapping information by using the image information of each divided cell, and as a result of analyzing the image information such as the color and texture information of the selected cell, If the analysis information is single, it is preferable to determine that the cell is included in a single object.

In addition, the cell analysis process may be performed when the one or more pieces of analysis information such as color, texture, and boundary information of the selected cell exist. If the analysis information is within the error range, it is preferable to interpret the same information and store one analysis information as representative image information of the corresponding cell.

The cell analyzing process divides the cell into a predetermined number of subcells after analyzing color and texture information of the selected cell, and analyzes image information of each subcell, and then analyzes a single analysis. For a subcell having information, it is preferable to store the single analysis information as image information of the corresponding subcell.

In addition, when the analysis result of the image information of the sub-cells, if the one or more analysis information is present, the sub-cell is subdivided into a predetermined number of lower-level sub-cells and the image information of each of the lower-level sub-cells is analyzed. Thereafter, it is preferable to repeat the process of storing the single analysis information as image information of the corresponding sub cell with respect to the lower level sub cell having the single analysis information.

In the cell analysis process, segmentation and image analysis of the subcells may be repeatedly performed up to a preset maximum level.

In the cell analysis process, when one or more pieces of analysis information exist as a result of image analysis of the maximum level subcell, it is preferable to store any one of them as image information of the corresponding cell.

In addition, the simple object information storing process may store cell information including each position and image information represented by cell information constituting the objects.

In order to identify and extract the object by using the image information of the analyzed cells in the object identification step, it is determined whether the image analysis information of adjacent and consecutive cells is the same, and if the same is displayed, the cells of the same information are displayed as the same object. .

In determining the analysis cell, the analysis target cell has a plurality of pixels.

In addition, in identifying the object from the analysis information of the image, the upper cell has an upper cell composed of multiples of two of the lowest cells, and in the same analysis information, it is preferable that the upper cell is not classified as the lowest cell but processed as a bundle. Do.

In the object indexing step, an object is identified as a set of cells having the same color, texture, or continuous outer division boundary while the adjacent lowermost cells or upper cells have adjacent surfaces consecutive to at least one cell in the analysis information. Characterized in that.

At this time, the object indexing process is characterized by representing and managing and storing as cell information about the relative position included in the image.

In the method of storing the identification information of the object, the designated cell is configured in the same frame.

In addition, in managing object identification information in units of frames, it is preferable that one frame is managed as binary summary information including whether an object is included in a cell position.

If this is expressed in a table, it can be expressed as shown in [Table 1].

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 (5,3) One 0 0 0 0 0 0 One One 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

The region of the table shown in [Table 1] represents a virtual cell region corresponding to the frame region in the image. In Table 1, the cell regions of '0' and '1' are expressed to distinguish the cells recognized as objects.

One virtual cell shown in Table 1 is a region obtained by dividing a frame in an image in a frame by a multiple of 2. This is a set of pixels corresponding to absolute pixel coordinates, and shows a virtual split cell of a maximum level.

 The cell region identified as the object is expressed, and the identification object is represented and managed as described above.

In addition, when the cell is expressed in hexadecimal, the relative position information of an object included in one frame may be expressed as 0x00000C0C0000 as object position identification information for each frame.

In storing and managing the identification information of the object position in the virtual cell, the number and representation of virtual cells of the maximum level may be variously configured as horizontal lines, vertical lines, or polygonal cell sets.

0 0 0 One 0 0 One One 0 One One One One One One One

Table 2 shows the object identification region in the divided subcell with respect to one of the highest cells shown in Table 1.

As shown in Table 1, each 8x6 top-level cell can have a subcell. If the last cell in the table is taken as the reference cell, it is located at the cell coordinate (5,3) starting from the top left indicated by '1'. The object identification and index information of the subcell may be represented as a virtual subcell table as shown in [Table 2], and this may be represented in binary again.

 In addition, to identify an object, a polygon for identifying an object is extracted by calculating a relative distance between a connection angle between consecutive adjacent cells of the lowest subcell and a vertex at which an angle variation occurs.

The polygon extracted for object identification compares the similarity of the sample database of the polygon database to yield an approximate object property.

In addition, it is preferable that image attribute information and object pattern information of an object represented by a binary number are managed by index object information such as searching for a similar frame and copyright.

It is preferable to segment the scene of the frame by using the image information analyzed by the frame unit and the variation information of the object through the object identification process and procedure.

In the frame scene segmentation and object identification, it is preferable to process the frame sampling by using the variation of the frame analysis information before and after having a predetermined frame selection period rather than performing every frame.

This is 29.97 frames / second for a typical video screen, but it is desirable to increase the frame count by a certain number or to process the object and image analysis at regular time intervals.

The scene classification step includes analyzing identification information of front and rear frames to determine whether variation information of the frame identification information is within an allowable range.

Looking at this in more detail, it is preferable to divide the scene by comparing the change of the background screen, the number of detected objects in the video image analyzed in the scene segmentation.

In the scene segmentation, the scene segmentation may be assigned a weight. In this case, the weight may be weighted to the background and weighted to the variation information of the object.

In the case of object identification in the scene division, the center cell of the cell is selected from the cell set information of the object, and the phase shift information for each frame of the center cell is checked. It is characterized by processing by analyzing.

In more detail, the same image information and the object analysis information may exist in coordinates of various cells, and the appearance in the frame may appear repeatedly. In this case, when an object in the start range frame of the reference frame does not exist within the allowable frame range, it is determined that the object is not related in the scene, and it is preferable to determine whether the change in the number of occurrences of the object falls within the specified range. .

In the object profile generation step, the attribute and feature set database of the object is added to add attribute information of the stored object.

In this case, the feature set may be expressed in a markup language such as XML (eXtensible Markup Language).

It is preferable that the feature of the object has a representative object feature, a general property feature, a relationship feature, a component property feature, and a special feature.

Representative object qualities assigned to the identified object; preferably has a representativeness of the object, such as people, buildings, mountains, cars.

The general attribute qualities include, in general, attributes including movement, spontaneous generation, artificial things, and living things.

The component attribute property; preferably includes an accessory attribute, a condition attribute, an action attribute, an event attribute, a time-season attribute, a place attribute, and the like.

The special feature is used for a special purpose feature used only for a specific limited object of a specific video, and has a feature property for extensibility with respect to additional property features other than the feature.

In addition, it is desirable that the relationship attribute qualities have qualities such as upper and lower relations, inclusion relations, parallel or association relations, and ownership or belonging relations.

The object attribute feature may be managed in a single or feature set pattern. At this time, the attribute value of the feature is a binary value. If the attribute property has a corresponding attribute, it is preferable to express it as "1".

If the object is managed with the detected attribute quality, it analyzes the relationship between the quality of the object and the property quality of the feature in the scene consisting of the frame and the set of frames consisting of the object and the background to recognize the situation and process various additional services. Characterized in that.

In this case, it is preferable that the object includes a relation between a sub object, that is, a main object and a sub object, which is a subordinate relationship of the object.

In addition, objects and objects can form a variety of relationships, it is desirable to configure such relationships, including procedures, parallelism, dependencies.

Attribute and feature information of the object and sub-object is preferably represented and managed in a format such as a database or XML.

In addition, in order to recognize various situation recognition and event occurrence through the operation of the object feature, it is preferable to include a rule base including conditions and processing algorithms for object situation and event recognition.

The rule database for object situation and event recognition may recognize a situation or event by deriving a result by calculating a condition according to the presence or absence of a plurality of objects.

The service profile generation process is to call the object information detected in the indexing process and the operation information or the specific context associated with the object, and the service profile includes the operation information for each condition.

The frame identifier of the frame in the image including the object for the service or the time information of the frame section and the situation information in the scene, the cell information including the location information of the object, the search information related to the object, the product purchase information , Video play information and the like, and advertisement information related to the object and frame scene situation.

For this purpose, a metadata addition process including quality information; Generating metadata based on context recognition and service context by applying a rule to each of the objects using the metadata when the metadata is input in response to the request; Generating a hierarchical structure between objects by using information such as attribute information and relationship information in the feature information of the objects; And storing hierarchical information including a logical object created by the hierarchical structure; and creating a connection between an object area and a service required for the information.

Tables 3a, 3b, 3c, and 3d below describe the sub-tasks required for object analysis, detection, and service analysis from video or multimedia.

[Table 3a] is a table showing the objects and sub information extracted from the frame of the image.

Media title A Frame index B Item Object 1 Object 2 Object 3 Object 4 Object 5 Object 6 Cell vector C D E F G H Additional Documents / Subtitles I Linguistic analysis index J

Processes and procedures for analyzing and processing moving images using the table have been described above, and thus, a preferred processing procedure will be shown based on this table. When analyzing a single video, the analyzed initial data is loaded into the table of [Table 3a]. The media title means the title of the video when the video is a video.

The frame index refers to an identifier capable of expressing a position of a current target frame, such as an identifier of a frame currently being analyzed in a video or a video showing time. Objects such as object 1 and object 2 mean a set of cell coordinates of objects detected in one frame.

In this case, as described in Tables 1 and 2, the cell coordinates define (X, Y) coordinates in the corresponding reference coordinates, and each cell is represented by a binary number to represent an area cell of the object. In this case, it is preferable to detect one frame by dividing the detected object into a background or environment object instead of the object.

If the analysis video includes an additional description document or subtitle I for the media, it is preferable to include a separate language analysis process by synchronizing with the frame region or the frame region of the document.

Index information through the language analysis process preferably includes a language analysis index (J).

Table 3b is an object feature analysis rule data table.

 Item Qualities vector  Situations and event types Qualities pattern Qualities pattern Qualities pattern Rule 1 Rule 2

[Table 3b] shows the rules database for identifying the core objects and the main objects by analyzing the objects that specify the analysis information in [Table 3a].

The rule is for the purpose of extracting the desired result by analyzing the information or association required by the object to analyze the feature value.

In detail, it is characterized by extracting the situation or event of the object from the database that matches the object bit pattern by determining whether the object and the object have a certain feature in common or not.

Object 1 Object 2 Object 3 Object 4 Representative Object Qualities General property qualities Ingredient Property Qualities Relationship attribute qualities Type / color attribute

By analyzing the metadata including the feature information of the frame and the object using the rules of [Table 3b] for the object detected in the [Table 3a], the association between the objects, the context extraction of the frame is performed.

Metadata added to and analyzed in Table 3c is preferably stored in a feature table described below for user service.

Item Frame section  Meaning qualities Object identifier Index word 1 Index word 2

[Table 3d] shows semantic feature information and lexical feature information extracted from a scene or a frame when it includes voice recognition, text information or subtitle information in a video or multimedia according to the present invention. It is desirable to provide a variety of intelligent services by mutual operation.

This is to analyze the semantic qualities through morphological analysis and segmentation analysis of a plurality of index words extracted in a specific frame section A, and if there is an object that matches the situation of the frame, that is, the context, the index word is associated with the object.

In addition, in the determination process, the input method may include: selecting and pointing an object; It is preferable to use any one of natural language input methods including voice and keywords.

The object selection information in the frame may include a frame identifier and an identifier of a cell.

In this case, a function or procedure for mapping the pixel coordinates of the screen and the relative coordinates of the object is provided.

In this case, the input information for the search is preferably performed to search the related service before and after the time frame at which the input object selection pointing, voice, or keyword is input.

In the service providing process, it is preferable to display corresponding object information based on the service profile information generated in the service profile generation process.

In addition, the service providing process is preferably moved to the screen linked to the corresponding object based on the service profile information generated in the service profile generation process.

In order to achieve the above object, it is preferable that the image processing apparatus provided by the present invention is composed of an image server and an image display terminal and connected by one or more methods of mobile communication, wireless or wired communication network.

The image server comprises: an image information storage unit which stores image information including the object information and cell information which is screen division information of each image; An input unit for receiving image information; An encoder for encoding the input image information and storing the image information in the image information storage unit; An index unit which detects object information from the stored image information and matches the image information with the image information storage unit and stores the object information; An object information manager configured to manage the meta data of each of the object information and the hierarchical relationship between the object information; A search unit for searching the image information storage unit based on the request information; A user interface for providing interfacing with a user; A communication interface unit for providing an interface with a communication network; And a control unit for controlling operations of the storage unit, the input unit, the encoding unit, the index unit, the object information management unit, the user interface unit, and the communication interface unit.

An image server configured to store image information including object information and cell information, which is screen division information of each image, and to provide corresponding image information and cell information, which is screen division information of the image, in response to the image information request; And a display terminal receiving display image information and corresponding cell information from the image server, dividing the display screen into cell units based on the cell information, and outputting the display image information on the display screen. Do.

In more detail, the image server may include image information including the object information, cell information which is screen division information of each image, and feature attributes and service link information included in the object; An input unit for receiving image information; A video converting unit for converting the input image information into an encoding or an appropriate format and storing the image information in the image information storage unit; An information indexing unit for detecting object information from the stored image information and matching and storing a frame of the image information of the image information storage unit and an object area and connection information in the frame; An index information manager configured to manage the meta data of each of the object information and the hierarchical relationship between the object information; An object feature register for managing and assigning features and property values of object information; Object feature information management database unit; Index and service information database manager; A service register for connecting various services to an image and managing mapping; A search provider for searching the video information storage based on various request information; A service request interpreter for interpreting and processing service requests from a user and a terminal; A result output unit for extracting and processing a result to transmit a search result to a terminal; and a network connection unit providing an interface with a communication network; and including the storage unit, an input unit, a video conversion unit, an information index unit, and an object index information management unit. , A service manager, a search provider, a service request interpreter, a result outputter, and a controller for controlling operations of the communication network connection unit.

In addition, the index unit detects the relative position of the objects included in the image, and analyzes the image information of the object to analyze the simple object information including the position and image information of each of the objects represented by the cell information constituting the object It is preferable to store in the image information storage unit.

In addition, the index unit detects a basic screen size for the image, and divides the screen into a plurality of virtual cells based on preset split information, and analyzes image information of each of the divided cells, and among the cells having the same analysis information. It is preferable to recognize a set of adjacent cells as one object and to store simple object information of each of the objects.

The image information storage unit may further include object identification information, image identification information including the object, configuration cell information including identification information of each of the divided cells constituting the object, area of the object, center point coordinates of the object, and the object. It is preferable to store the simple object information including the phase transition information and the image property of.

In this case, the image information storage unit preferably includes feature and property information of the object and service connection information connected to the object.

In addition, the index information management unit receives metadata about each of the simple objects detected by the index unit, generates a service profile for each of the simple objects using the metadata, and receives the related information between the simple objects. After generating a hierarchical structure between simple objects, it is preferable to store hierarchical information including a virtual logical object generated by the hierarchical structure.

The hierarchical information of the virtual logical object or sub-object is defined in the feature information of the object and processed.

The metadata may include feature attributes necessary for extracting association information between the object, including pixel position mapping information of the screen with respect to the virtual cell of the object, and attribute and feature information of the object.

The image information storage unit may store simple object-specific metadata information including object identification information, image identification information including the object, classification information of the object, link information, object detail information, and operation information according to an event. desirable. At this time, the motion information according to the event is motion information according to selection of an object or voice human input or a keyword, and includes at least one of moving to a corresponding link location, searching for related information or purchasing a product, and waiting for a subsequent operation. It is preferable to include one.

The image information storage unit may include logical object identification information, image identification information including the logical object, classification information of the logical object, identification information about simple objects included in the logical object, and motion information according to an event. It is desirable to store logical object information. In this case, the logical object information may further include lower logical object information identification information. In addition, the motion information according to the event is motion information according to the selection of the logical object, and includes at least one of a movement to a corresponding link, a purchase decision, a waiting operation for a subsequent operation, and a display of a list of simple objects included in the logical object. It is desirable to.

The controller may be configured to display corresponding object information or move to a screen linked to the corresponding object based on the service profile information generated by the object information manager.

In addition, the image information storage unit preferably stores the image identification information, the identification information of the cell as the screen division unit for the image, the cell division information including the start coordinates and the end coordinates of the corresponding cell with the corresponding image information. .

The image information storage unit may include image analysis information including image identification information, identification information of a cell which is a screen division unit for the image, division level information of a corresponding cell, identification information of the highest cell of the cell, and image analysis information. It is preferable to store the corresponding image information together.

In addition, the video server manages a control command set in a storage unit to provide a control command when providing a service through a communication network to a user terminal, and transmits the streaming information together at the start of every scene. At this time, the control command is for requesting additional input to the user when performing a search or requesting user intention, and more specifically, to provide various search options on the output screen. This is to provide various user input interfaces by integrating the user input means into the image information display area and displaying it on the image display screen.

To this end, the video server manages a common input control command set and a control command set suitable for each frame area in association with an object and index information and manages it in a database.

The display terminal receives the object selection information included in the image being displayed on the screen and transfers the object selection information to the image server, and the image server preset operation for the corresponding object based on the object selection information. It is characterized by providing an additional service according to the object selection based on the information.

The display terminal may further include a display unit including a display screen divided in units of cells and outputting the display image information; A search information input unit for providing a user input method of various means such as object selection, keyword input, or voice input; An input information analysis unit to determine whether the additional information is required by determining the input information; An input information generation unit for providing an additional input interface for supplementing the user's intention of whether the input query is sufficient to perform a search, or a network connection unit for providing an interface with a communication network; And a result output unit for outputting a result transmitted from the video server.

In this case, for the purpose of generating input information, for example, when a user selects an object, it provides a means for additionally selecting a search condition for searching for information on the selected object, searching for related videos, or linking link information. It features.

The input information may include an image identifier; Frame identifier or time information; Cell information of the object location; Control command selection information, and the like.

In order to generate the input information, the video server checks a frame or scene identifier and distinguishes command control information to be provided in common from the corresponding scene and command control information to be provided in a specific frame and transmits them in synchronization with the video information of the terminal. Do.

The command control information is preferably managed in the form of a database or a table in the video server.

At this time, the synchronization information managed by the server is preferably transmitted at an appropriate time interval in consideration of the scene or frame identifier information and the transmission status of the network.

In generating various input information, it is preferable that the terminal identifies the area of the object selection cell and allocates it to the most identifiable position corresponding to the cell location information where the object selection information is located.

Preferably, the display apparatus includes vector mapping information that stores mapping information of the coordinate information of the display screen and the cell division information, and controls object recognition information to be transmitted to the image server based on user input information.

In addition, the video server and the display terminal is preferably configured as an integrated device.

In addition, the video server and the display terminal preferably transmits / receives data through a communication network.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. At this time, it should be noted that the same elements in the accompanying drawings are represented by the same reference numerals wherever possible. In addition, detailed descriptions of well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention will be omitted.

1 is a block diagram of an image server and a user terminal according to an exemplary embodiment. The present invention is an overall block diagram of an image processing apparatus and processing method including a video server and a display terminal and connected to a network to provide various additional services.

In FIG. 1, the video service server 100 searches for a service target video, requests an index registration for the service, and converts or encodes the video according to a format for the service on the network; and a basic screen of the video. Detects size, maps divided cell information, maps pixel coordinate area of base screen per cell, analyzes screen information of frame corresponding to cell area, detects cell area of object, and extracts polygons from detected cell object area. There is an information index unit 120 for generating object identification information through.

In this case, the information index unit performs a language processing procedure for a frame through a language analysis process when a caption or related document information exists in the analysis target image. The linguistic analysis procedure preferably includes a procedure including semantic analysis including morphological analysis and syntax analysis. In the case of an index keyword, an object feature information and semantic information including a cell are included together with a video frame and object information. It is characterized by managing. An index information management unit 130 for inputting and confirming object feature information on an identified object and undergoing a metadata input procedure including object index information and feature assignment; and using the feature information in the metadata managed therefrom. It is preferably processed as described in the above procedure, including context information, object relationship information, behavior information, etc. of the corresponding frame.

The service register 140 calculates the detected object information, the polygon information of the object, the feature information of the corresponding object, and the like by using the analyzed image object and the relative coordinate information which is the virtual cell information where the object is located. The metadata generated by the processing and the extracted context of objects, frames, and scenes are stored and managed in the image information storage unit 190.

In this case, in order to store and manage metadata and feature information in the image information storage unit, an object feature database, a processing rule database, and the like are pre-built and processed as shown in [Table 3].

The service registration unit 140 may provide services of various methods for objects, frames, and situations by using metadata generated through the methods and procedures of the information index unit 120 and the index information management unit 130. To do this, it is responsible for registering a service. Metadata is stored in the index and service information management DB (192) and stored and processed in the feature information management DB (191), including various calculation rule DBs, feature analysis DB, and language analysis DB for generating and managing such metadata. do.

In addition, the service register registers a rule for processing terminal information and interactive command information and a control command code for displaying on the terminal to the terminal to effectively manage input control information and object cell mapping information required by the user terminal. At the same time handle the service.

When the service request comes from the terminal, the service request interpreter 160 interprets the request query. First, it is preferable to analyze the service request type, and to analyze the analyzed result in detail so that an appropriate search can be made. Do.

The above-described analysis of the query request service type analyzes whether an object is selected, a query input or a voice input, and performs a procedure of pointing, query and voice recognition according to the result.

The query interpreted by the service request interpreter 160 is searched in the index database and the service information management database through the search provider 150, and is formatted according to the terminal output format through the result output unit 170 and then connected to the network connection unit ( 180 is serviced to the user terminal through one or more networks of mobile communication or wired and wireless networks.

The user terminal 200 includes an image display unit 210, a search information input unit 220, an input information analyzer 230, an input information generator 240, a vector mapping information table 270, and a control command information code 280. And a network connection unit 250 and a result output unit 260.

The image information display unit 210 displays the image received from the image server connected to the network.

The search information input unit 220 is provided using an input device such as a coordinate input input using a mouse or other pointing device, a keyboard for inputting a natural language or a keyword, a keyboard for inputting a natural language or a keyword, and a microphone for voice input. Can be.

The input information analyzer 230 interprets various input devices and methods input by the search information input unit 220.

At this time, the identifier of the cell corresponding to the screen is extracted according to the method input by the user, and the vector mapping information 270 table is referred to in order to provide various interfaces linked to the frame and the object at the point of query input.

The query interpreted by the input information interpreter 230 undergoes a process of determining to query the input information to the image server 100. At this time, it is determined whether additional user query information is required and the control command information 280 is displayed on the user screen. You will be asked for various additional inputs.

Through the above process, the terminal generates input query information in the input information generator 240 to send to the image server 100. At this time, the format of the generated query information may be as shown in [Table 4] below.

Protocol identifier Session id Message type Reserved Video ID Frame ID Cell ID Payload Length Payload

The data format generated as shown in [Table 4] is referred to as a packet according to the packet rule in the communication network. It has a protocol identifier, a session ID, a message type, a reserved field, a video ID, a frame ID, a cell ID, and a payload length field which are common header parts of the packet. The payload may include a user identifier (ID), natural language query text or voice query data, and a code for authentication.

The message type defines various types of input messages for cell pointing in a video, control command processing, query input or voice input.

When the terminal query packet is sent to the image server 100 through the terminal network connection unit 250, the result processed by the image server is output through the result output unit 260 of the terminal.

2 is a data table for an image index. The main table for the image index is a database table for generating and managing various data obtained by analyzing and processing image information. In detail, the image main table 10 includes a video main table 10, a scene division table 20, a frame table 30, an object table 40, a sub object table 50, and an inverse frame object set table 60.

The video main table 10 is a general video information table for a target video. Video_ID 11 is an identifier for identifying a target video in the video server. Disp_Size (12) is the screen size of the video, which means the screen size when encoding the screen. Run_Time 13 is a play time of a video. Cell_No_Depth 14 is the number of repeated divisions of the subcell for dividing the video. Video_Title 15 is the title (title) of the video. Idx_Term (16) refers to an interval for extracting and indexing frames from a video, which can be dynamically indexed and processed in a specific cycle, that is, a time interval or a frame interval. No_Scenes (17) represents the number of split scenes in the video. No_Frames 18 is the total number of frames in the image. Cate_Class_Id 19 indicates a category classification system of the corresponding video.

The scene division table 20 is an information management table for managing a scene division area in an image. Scene_Id 21 is a scene identification identifier and has a scene start frame Start Frame 22 and a scene end frame End_Frame 23. The scene segmented time has a scene time 24, a key object key object 71 in the scene, an object set 71, a control command 72, and a scene context ID 28, which is a scene context recognition identifier.

At this time, the core object 25 and the object set 71 refer to the objects in the frame section divided into scenes, and are for managing which objects are composed in a particular scene.

The frame table 30 includes a frame identifier Frame ID 31, a frame index Frame Index 32, a frame time Frame Time 33, a frame object set Object Set 71, a control command Control Command 72, and a frame status identification. Code Context ID 34 and Service Anchor 73 to process the service.

The frame identifier 31 is for identifying a specific frame region in the image, and the frame index 32 manages the relative coordinates of the object in the region divided into cells in the frame, and corresponds to the object cell region corresponding to the pointing coordinates transmitted from the terminal. Is an index managed for object detection and object search for the corresponding pointing coordinate area cell.

In more detail, as shown in [Table 1], the in-frame object index values for the objects masked with 'ga', 'me', 'da', and 'la' are divided into cells. When each is expressed in bits and converted to hexadecimal, it is as follows. {0xC000, 0x8000, 0x0010, 0x2310, 0x7390, 0x21B8, 0x0038, 0x0078, 0x007C} are managed by the same frame index.

The frame time 33 represents the time position at which the corresponding index frame is located. The frame object set 71 is four cell set information indicated by '1' in the case of [Table 1].

Control Command Control Command 72 is a function provided to the user for additional search options in the terminal. The server can provide various search options and functions for each image, scene, frame, and object, and integrates control commands into the video screen. The biggest advantage of this is that it has great flexibility in the limited screen and limited functionality of the video screening player.

To provide various search options or functions to the terminal video player The function of the player is complex and it is difficult to apply and process it to all terminals, but the desired control command is superimposed on the cell of the screen area of the player and the corresponding control command is selected. This is to support the function to transmit the region value of the overlaid cell to the server and interpret it to interpret the request as a specific function.

The frame context ID 34 is a key for managing context identification information of a corresponding frame.

The service anchor 73 is a service reference key for referencing and processing service information provided to the object of the frame and the frame area.

The object table 40 includes an object identifier 41, an object description name 42, a frame identifier 31, an object index 43, an object pattern 44, a polygon extraction type 45, a control command 72, The object context 45 has a feature set 75 and a service anchor 73.

The object identifier 41 is a unique identifier given to all objects extracted and identified in the frame.

The object description name 42 is an object name, and the object index 43 represents an index of polygons and image color attributes including the coordinates of the object subcells.

In addition, the object pattern (44) is extracted by representing the pattern of the object detection subcell in binary bits.

The polygon extraction type Polygon Ext. The type 45 may extract a feature of an object based on a ratio of vertices, faces, and elements of the extracted polygon by analyzing the shape elements of the extracted cell regions for each object.

Object Context Object Context 45 contains information about the context within a frame of an object.

Feature Set Feature Set 75 is a set that includes various attribute information of an object.

In this case, the feature set 75 may express all the feature sets of sub-objects included in one object and process the set of sets.

The sub object table 50 is a sub object of the object and is an object identifier 41, a sub object color identifier 51, a sub object cell coordinate region 52, a control command 72, a feature set 75, and a service anchor ( 73).

In this case, the sub-object cell coordinate area 52 is sub-object position coordinate information in the object area.

The inverse frame object set table 60 is a reversed mapping table for the frame table. This is to manage and retrieve the information of the object located at the corresponding coordinate in the frame.

The inverse frame object set table 60 includes a frame identifier 31, a control instruction 72, a frame context 34, an object summary offset 61, an object detection number 62, an object identifier and its coordinates 63. )

The object summary offset may be used for the purpose of searching for the same and similar information and managing the copyright of the corresponding frame by summarizing the overall object composition and background and image analysis information in a specific frame.

3 is a table for managing a service profile and feature set information. And a category table F10, a control command table F20, a context DB F30, an object index DB F40, a feature set DB F50, a service DB F60, and a polygon DB F70.

The category table F10 is a table for managing a corresponding service classification system required for providing a video based service.

The control command F20 is used to provide an interface to the terminal, which provides the screen with the coordinates selected from the scene, the frame or the object, or the function options to be provided in the frame. For this purpose, each control command has a unique identifier, and the control interface provided in the scene or frame can be defined differently.

For this purpose, the control command may include a command for displaying the control command on a user screen and options including parameter values necessary for executing the control command.

The context DB F30 may include a classification identifier for a context, a feature matching rule for extracting a context, a matching condition to be interpreted as a corresponding context, and a core context; Secondary context;

The object color DB (F40) is the object information for the frame identification object; and the condition information that the object has, a service identifier connected to the object; And index words for subtitles or additional document information of the video; And object connection information.

The object index feature DB (F50) is an object index feature dictionary that manages feature sets according to the feature classification system for each object. Feature dictionary for representative objects; object identifiers and general features; Relationship quality; Attribute qualities; Have special qualities.

For example, if the 32-bit feature is allocated to each feature property, the feature DB feature has 128-bit feature representation for one representative object. It is preferable to set and manage '1' if there is a feature for the corresponding feature by object feature classification.

In order to search for a context or search for an association through this, it is desirable to perform intelligent search and management of frames and objects in an image through a comparison operation on specific feature values of the object through Boolean operations on two related objects.

 In the service DB (F60), the service anchor value is preferably used as an identifier of the service DB in FIG. 2, and a parameter value that can be processed when the service is called is used as a condition, and the service is an object or a frame through a corresponding control command. The concept of calling a service defined in service DB by interpreting any input value in.

Polygon DB (F70) constructs the polygon value of the detected value as the number of vertices, the feature of the adjacent angle, and the ratio of the face ratio, and estimates it as an approximation of the object when it reaches a certain value. It is a reference database.

Index word DB (F80) is a language analysis reference dictionary database for context identification through morphological analysis and parsing and mapping of relevant contexts and events with objects in frames for language analysis and event processing of documents and subtitles included in the image. .

4 is a relationship diagram between the database table described above with reference to FIGS. 2 and 3. One video or multimedia may have an image main table 10 for managing image information, and an image may have a scene table 20 composed of several scenes. The scene table may include a plurality of frame tables 30. Each frame has an object table 40 composed of image attribute information and objects, and the object table may have a plurality of sub object tables 50.

When the pointing information for the cell area is selected from the terminal in the frame table 30, the inverse frame object set table 60 checks and processes object information about the pointing information of the frame and object summary information of the frame.

In addition, the scene table 20, the frame table 30, the object table 40 and the sub-object table 50 preferably refer to the polygon table F70 in order to analyze information and extract scenes, objects and sub-objects. Do.

The category table F10 refers to managing categories and classification schemes in the image main table and the service table F60 for providing a service.

The service table F60 defines a service in the frame table 30, the object table 40, and the sub object table 50 and has link information.

Command control (F20) manages interface control command information to be provided to a user in a scene or a frame, and generates and manages a user interface corresponding to a selection of an object or sub-object or a control command through search or voice input in a frame. It manages and rules, and is used to provide a user interface for providing scenes, frames, objects, and sub-objects and services.

The context table F30 is used to recognize and manage the context of scenes, frames, objects, and sub-objects. Various context-aware video services are possible through the calculation of context and object-related information and features.

The object index table (F40) manages cell coordinate information, service information, and identification information of object feature set information of an extracted object in a frame. Through this, an object corresponding to a specific pointing coordinate in a scene or a frame is extracted and related services are managed. Core table for searching and serving.

Index word DB (F80) is an index extraction DB that enables natural language or keyword search for relevant information by mapping the extraction index DB through morphological analysis and parsing when there is additional document information or subtitle information in video information. .

The feature table F50 is characterized by having property feature information required for object analysis and language analysis as having an object feature, and a feature DB for language processing and parsing.

5 is a general procedure of the present invention. According to this procedure, the image information search for confirming whether there is newly registered image information (S100); Image information analysis (S200) for analyzing a video format, screen information, etc. with respect to the new target image information; An image information index (S300) for analyzing the image information from the analyzed original image information beam and indexing the extraction information into the cell area; Object identification S400; Endowment (S500); Service profile generation (S600); and provides a video service search (S700).

6 is a flowchart illustrating image information retrieval. By checking whether there is an analysis target video information in the image information storage (S110) and checking whether an index target video exists (S111), if there is an index target video, it is determined whether the video format and codec support are supported and an appropriate codec is selected (S120). ) Analyze the video.

At this time, the first step is to check the video header and key frame information (S121) and the screen size and frame information analysis step (S122) of the video and determine whether the original video is required to encode the codec for the service (S130) After converting or re-encoding the video for the streaming service (S131), the highest cell division number and the maximum cell division number are determined (S140).

The determining of the highest cell division number and the maximum cell division number S140 may include determining a division step, that is, a division depth, of a virtual division cell for image analysis of a screen size and pixel information analyzed from an original image. Means. Here, the highest cell division number means a division number for dividing a cell from a frame, and the cell is divided into multiples of two.

In the cell division, the depth of division refers to the number of times the highest division cell is repeatedly divided by a multiple of 2. The maximum cell division number refers to the total number of divisions of the smallest cell repeatedly divided in the highest division cell.

The frame analysis cell information analyzed as described above is stored (S150) and returned.

7 is a flowchart of a process for analyzing an image image. By analyzing the size and image quality of the frame, the division and the depth of division of cells are determined (S210). The image size of the cell divided into cells is obtained from the frame image (S220). The image is analyzed in order from the divided cells (S230) to determine whether the image analysis information of the cell is single or has a plurality of analysis attributes (S240).

In this case, the image analysis information is image information within the pixel coordinate region of the image corresponding to the cell region. In addition, the information to be analyzed preferably includes a color, texture, image boundary, and the like.

In the cell image analysis information determination (S240), if the single image attribute information is included, the image attribute information corresponding to the cell is stored.

8 is a flowchart illustrating a process of storing an object identification result. In order to analyze the frame image, the cell size and the image quality are determined by analyzing the frame size and the image quality (S210), and the image size of the cell divided into cells is obtained from the frame image (S220). The image is analyzed in order of the divided cells (S230), and whether the image analysis information of the cell is single information (S240). If the single object is within the determination ratio of the single object, the image attribute information corresponds to the cell coordinate and the result is stored (S250). The analysis result value of the cell is set to '1' in the result table (S260). At this time, it is determined whether the current cell division depth (Depth) is the set maximum depth (S270), and if the maximum depth is determined, the next adjacent cell exists and the adjacent cell is not the last cell (S280).

If it is determined whether image analysis information of the cell is single information (S240) and whether a plurality of pieces of analysis information exist and the analysis information is within a single object determination ratio (S251), the analysis result value of the cell is set to '1' in the result table. If it is set (S260) and outside the object judgment ratio, it is determined whether or not the depth of the cell of the current cell is not the maximum division depth (S270). (S230).

In the above process, if the object determination ratio is outside and the current segmentation depth is maximum, it is determined whether the current analysis cell is the last cell (S270). If it is the last step (S280), the analysis step is terminated or the next cell is selected to analyze the image (S230). .

9 is an example of object analysis. The object index S300 searches for a cell having a cell identification area value of '1' in a virtual cell corresponding area in a frame (S310), searches for an object area (S320), and stores index information of the found object (S330).

In the object image region analysis information of the stored search cell region, the boundary is analyzed (S340), and if there is a warning, the boundary of the subcell of the object is searched and the boundary is set to '1' and the rest to '0' (S350).

The sub-object having a boundary is extracted from the continuous cell of the search sub-object (S360) to store the extracted sub-object information.

The above process is repeatedly performed until there is no subcell having a region value of '1' in the gourmet cell (S380), and the extracted object identification information is stored in the frame table 30 and finished.

The area of the whole divided cell for cell identification is an area corresponding to the actual pixel coordinates of the frame area of the image. In order to identify objects by determining image analysis information in each cell area, color, texture, and boundary line detection steps are performed. The identification value of the detected object region cell is set to '1', and each cell is the maximum divided cell region. In the cell division virtual description region corresponding cell C01 of the frame, there are a triangular object C02 and a cuboid object C03 composed of four sub-objects.

The entire cell area corresponding to the frame area is set to '0' and the index identification cell or the target cell area is set to '1'. If the sum of the cells in the top left arrows' 1 ',' 2 ',' 3 'and' 4 'is set as the maximum divided cell area, the area with each arrow is the area of the first divided depth and each' 1 'is divided into 4 multiples of 2 multiples. In other words, this division process shows that the top 12 cells have 12 subcells and 64 subcells.

Accordingly, in the method of describing a cell in FIG. 10, the x-axis length of the screen is referred to as 'X', the x-axis division number is 'n', the y-axis length of the screen is 'Y', and the y-axis division number is 'm'. In this case, the cell identification information of the virtual cell is preferably expressed as illustrated in [Equation 1].

When the cell identification information is set as illustrated in Equation 1, as shown in FIG. 11, the j and i values are changed, and an analysis process is performed on all the divided virtual cells.

: 'O'는 객체 식별자이며 객체를 포함하는 셀의 좌표(i,j)에 위치한 객체의 셀 분석 색인 값

'O' is the object identifier and the cell analysis index value of the object located at the coordinate (i, j) of the cell containing the object

: 셀 좌표 C의 (i,j)에서 대상 객체를 포함하는 식별 영역에서의 셀 분할 방법과 깊이를 정의.

: Define cell division method and depth in the identification area containing the target object at (i, j) of cell coordinate C.

  X: the length of the x coordinate cell in the highest base cell

  Y: the length of the y coordinate cell, including the topmost base cell

  n: number of subcell divisions in the upper cell

  m: Depth of sub cell

 I: I is the bit index value of the cell.

10 is a diagram illustrating an example of screen division for image processing, according to an exemplary embodiment.

내의 객체의 분석 색인정보는 도 11의 예시도에서 자세한 처리과정을 살펴보고자 한다. At this time

The analysis index information of the object in the example will be described in detail with reference to FIG. 11.

는 4개의 '1','2','3','4' 영역에 위치한 객체의 좌표이며 이때 셀 좌표 'C'는 도 9에서 보여지는 프레임 영역 기준 셀로부터 (i,j)에 위치한 최상위 셀을 의미한다.11 is an exemplary diagram of subcell division information processing. In the drawing

Is the coordinate of the object located in four '1', '2', '3', '4' areas, where the cell coordinate 'C' is the highest position at (i, j) from the frame area reference cell shown in FIG. It means a cell.

에 존재하며 이때 객체의 서브셀내의 분포는 영역을 포함하고 256개 셀은 서브셀을 나타내고 의미한다. In this case, the object

In this case, the distribution in the subcell of the object includes an area and 256 cells represent and mean the subcell.

 The object partition index information 'I' may be represented as in Equation 3 above.

객체 색인정보를 표현하면 수학식 4와 같이 나타날 수 있다. 이를 수학식 4를 통하여 객체 색인 방법을 상세하게 설명하면 Of FIG.

When the object index information is expressed, it may appear as shown in Equation 4. If the object index method is described in detail through Equation 4

In FIG. 11, the most significant cell '2' is divided into four subcells, that is, a multiple of two into 'A', 'B', 'C', and 'D', and each subcell 'A', ' B ',' C 'and' D 'are divided into four.

In this case, when the cell of the analysis information is extracted based on the similar adjacent analysis similar information through the image analysis of the object and other background information in the subdivided region C04, which is the 'C' of the cell '2' It may come out like a subcell C04.

This is represented by Equation 4 when the partition depth d = '3' in Equation 'I' and expressed as a matrix index value in units of subcells. At this time, the identification value of the subcell C04 is represented by {0,1,1,1}, {0,0,1,0}, {1,1,1,1}, and {1,0,1,1}. . In the hexadecimal representation of the bit value of each of the sub-cells d = '3', '0x72FB' and adjacent cells may also be represented in the form of [Equation 4].

객체 분할셀의 기준 좌표를 구한다(S401). 기준좌표에서 연속인접셀을 분석(S410)하여 분석 셀의 비트 값이 '1'인 서브셀을 판단(S420)하여 인접한 동실 속성의 서브셀들의 구성 빗면각과 거리를 삼각함수를 이용하여 구한다(S421). 상기 S421단계에서 기준셀과 인접셀이 이루는 각(∠)의 변이를 판단(S422)하여 기준값을 분석하여 꼭지점의 셀 좌표를 저장한다(S430).12 is an object identification and polygon recognition procedure. In the extracted object identification information stored in FIG. 8, the background and the object identification information and the image attribute information are read (S390).

The reference coordinate of the object division cell is obtained (S401). A continuous adjacent cell is analyzed in reference coordinates (S410), a subcell having a bit value of '1' of the analyzed cell is determined (S420), and a configuration of oblique angles and distances of adjacent subcells of a same room attribute is obtained using a trigonometric function (S421). ). In step S421, the difference between the angle formed by the reference cell and the adjacent cell is determined (S422), the reference value is analyzed, and cell coordinates of the vertices are stored (S430).

The reference value of the shift angle means that if the shift angle of a cell where two cells or a plurality of adjacent cells form the longest distance is within or above a certain value, such as 150 degrees or more, it is regarded as a straight line and the adjacent surface of each cell within 150 degrees is indexed as a vertex. I mean.

After the analysis between neighboring cells, the result is stored and searched for successive neighboring cells (S440). If there is no analysis target cell, the object identification polygon is stored and the center coordinate cell of the object is obtained (S450). If the detected polygon is found to be similar in the polygon DB (F70) (S460), if a similar polygon model is found, a representative object identifier is assigned and stored in the object table 40 (S470), or otherwise stored in the unregistered object table (S480). .

13 is a scene division processing procedure diagram. The first segmented object and the background analysis information are extracted from the stored frame information (S390), and the information of the adjacent index analysis frame is stored in the scene segmentation buffer (S491).

The frame index information stored in the scene division buffer is compared (S492) to determine the variation information and similarity between the reference frame and the analysis information (S493). If similarity is found, the next frame is read and analyzed. If similarity is insufficient, the scene is transformed, and the scene is divided into frames directly, and the segmentation information is stored in the scene division table 20 as scene section information (S495). Repeat until all analyzes (S495).

14 is a procedure of granting object qualities (S500). Using the object identified in FIG. 12 and the polygon information, an object feature value is assigned (S510), and whether the assigned feature value is appropriate or unregistered object is determined (S511). Values are given (S520), general attribute quality (S530), the component and the relationship quality (S531) is given and the special features for the object is confirmed by granting special features (S532).

After assigning the extracted feature value in the frame, search for an object in the same frame to determine whether there are additional feature grant objects (S540), and if the feature values for all the detected objects are given, grant the feature values for the background object (S550). do.

If there is an additional document file, it is determined whether there is a subtitle or additional description document related to the image (S560), and text is extracted from the document file to generate a feature vector through language analysis and processing procedure by referring to the index word DB (S561). . The event and situation information is mapped by analyzing the analyzed feature vector and frame object feature information (S569).

Event and situation information in the frame is generated through feature computation between objects (S570), and the analyzed result is stored (S580).

FIG. 15 is a diagram illustrating a logical association between detected object information according to an embodiment of the present invention. The bold lines represent logical objects and the rest represent simple objects.

16 is a flowchart of a service profile generation step (S600). The service profile includes the steps of generating association information between objects including logical associations and relationships of the objects as shown in FIG. 15 (S630), generating various service information on the objects and sub-objects (S650), and providing a service. The control command generation for the operation (S670) and the step of storing the generated service profile (S680).

17 is a diagram illustrating a service search process and sequence. Determining the input values (S720, S721) and generating the input query data format (Table 4), analyzing the query data format (S740, S751) and analyzing the query for image-based terminal search In the step of generating a control command code to determine the additional user intention and search options in the terminal (S760, S770) and the step of receiving additional input from the user (S780) may be included through this query search There is a step (S790, S791) to perform the operation and transmitting and displaying the result (S800).

The input value includes a cell ID (S720) and a binary input (S721) based on coordinates, and the type of binary query data may be text, voice, image, or video.

At this time, in the analysis of the data type in the binary input (S721), a process of 'A' or lower in FIG. 18 is called.

In the analyzing of the query data format (S740, S751), the message type of the message data format of Table 4 and the transfer values such as the cell ID and the payloader are analyzed to follow the processing procedure according to the input value.

The service anchor corresponding to the object index value of Equation 3b in the corresponding object table 40 and the sub-object table 50 by searching the object information located in a specific region in the index value in the frame search table 30 in the query search data. (73), the service code is generated (S760, S761).

At this time, it is determined whether an additional input is required to search the service code, and receives an additional control command option input (S780) from the terminal, compares the value of the condition information necessary in the search index DB (F40), and searches for the service code. The search procedure (S790) is performed on the corresponding service of the service DB F60 according to the control command information input condition using the code.

18 is a diagram for explaining a processing procedure for a binary query. In order to process the binary query input, binary query data is first analyzed (S722, S723, and S724).

In this case, if the query data type is an image-based query, an object pattern is extracted through image information analysis (S200) and image information index (S300), and a search query is generated (S291).

In this case, the query through the image or the video is to search or extract the context of the similar image or the video. For example, to search a server for a specific image and search for a scene or frame similar to the image. At this time, a specific object can be specified in the image or scene.

In addition, if the search is performed through voice (S723) in the binary query, voice recognition (S275) is performed. At this time, voice recognition transmitted from the terminal is performed by referring to HMI Human Human Machine Interface Data Base (MIB) and extracted voice. The keyword is extracted (S276).

In the analyzing of the binary query data property analysis, if the text is a search-based search query, the input query text preprocessing process (S278) is performed.

The text preprocessing S278 includes word division, stem-mother division, and the like of the input query statement.

The lexical analysis and key word extraction (S279) are performed by referring to the lexical dictionary and the rule dictionary. Weighting is given to similar lexical components.

The weighting process in the text search has a rule of extracting the feature of the lexical component existing in the text query that the user queries by searching the object feature information of the nearest frame when generating the text based query. it means.

In the extracting of the key word and the appended word, the scene context 28 and the frame of the scene partition table 20 are generated in order to generate a search query based on the similarity between the feature elements of the proximity frame object at the text query point (S291). It is preferable to extract and compare the features of the containing object with reference to the frame context 34 of the table 30.

The generated query result for the search is performed by analyzing the query request information through 'B' of FIG. 17 (S790).

In the binary search, when an unrecognized query that is not included in any query of image, video, voice, and text is generated, an error code is generated for the requested query (S795) and the result is transmitted to the terminal through 'C' of FIG. 17. .

19 is a diagram illustrating the above-described process of generating and processing a terminal control command of FIG. 16. Control commands are typically for the purpose of dynamically providing various search options in a video or image.

Storing search option information to be provided for a scene, a frame, or an object in a table to provide the dynamic search option; Searching and confirming the stored control command option information (S671); Extracting and transmitting control commands required for the corresponding video (S680 and S681); Displaying a control command on the terminal (S770); Selecting control command information in a frame or object search (S780); And generating a query data format by interpreting the selected control command cell or frame information (S900).

 The storing of the search option information in the table may be defined in the scene partition table 20, the frame table 30, the object and the sub-object table 40 and 50 of FIG. Conditions and the like are defined in the control command F20, the object index DB F40, and the service DB F60 in FIG.

The searching and confirming of the stored control command option information may include an environment analysis (S672) in order to check a provision condition of the defined control command information and option setting information (S671) and provide a control command code according to a frame to be provided. do.

In this case, the control command code is generated (S680). In this case, the generated control command transmits the general control command code at a specific period or when the image information is provided from the server to the terminal (S681).

The control command code transmitted to the terminal identifies the frame and cell coordinate control command data by identifying the frame and the cell coordinates (S682) when a search request using a pointing device or a search word input or a binary search method is performed while the video is being played (S720). In operation S683, an area for displaying control command option information is set. This is for dynamically allocating an area of the provision information or an arbitrary area to the user.

20 is an exemplary view of a service editor. The service editor includes an image display display area 810; Frame and object property and feature management area 820; A scene division display area 830; A video service preview and edit management area 840; Object and content search area 850; Attributes, qualities, and service editing and management areas 860 and 870; The control command code management area 880 is divided.

In the image display display area 810, an object identification area is displayed in the frame cell division area, and an object identified in the cell area may be modified, inputted, and deleted in the property and property management area 820.

The scene division display area 830 indicates a scene divided division part of an image as a scene start. Each scene can be selected to specify its attributes or attributes and used to edit the overall service connection.

The service preview and editing area 840 is an interface screen for confirming the suitability of service or feature assignment of the edited video.

The object and frame search 850 is a frame, object matching information search area for a representative object or title, context, and subtitle index of the object.

The feature and service editing and management areas 860 and 870 may include a video display display area 810; In the frame and object property and property management area 820, newly registered, modified, deleted, service statistics, etc. of the property property information, the object property information, and the service connection information of the edited and modified frame are checked. It is desirable to manage the markup language for the feature, the attribute information and the service mapping information, for example, the generated information expressed in XML (xXtensible Markup Language).

The control command code management area 880 is an edit screen for providing a variety of information or an interface required to provide a service on a screen or an image.

21 is a view illustrating a video service interface of a user terminal.

The image display area 910, a control command display interface 920, an object selection interface 930, a video control function interface 940, a search category selection unit 950, and a search window are included.

The control command display interface 920 may be set in various areas, i.e., by setting the user terminal or in various areas in consideration of the object selection area location, rather than being displayed in the fixed area.

22 is a service embodiment of a mobile terminal search terminal. In the mobile portable telephone, there is an image display unit 970, a control command display unit 971, a pointing device 972, and a numeric pad and cell region mapping display unit 973. In a portable terminal, a device, such as a mouse, required for selecting an area in an image or image information may be limited or difficult. For this reason, region mapping with the numeric pad can be set for the virtual cell region of the display image, thereby providing various functional interfaces.

When a corresponding number of a keypad is input to an image or an image provided to the image display unit 970, for example, when a keypad '3' is inputted, an object or service in the corresponding region is identified by selecting the area 3 of the upper right of the display image unit. Means search and provide. In addition, in the case of a mobile terminal providing a touch screen function, the above search request is possible when the corresponding area is pointed.

In the case of the control command display unit, it is possible to additionally provide a "*" "0" "#" key to provide various control command interface functions.

As described above, the present invention has an advantage of easy objective analysis of an image. In addition, the present invention has an effect of accurately determining the object of the position specified by the viewer on the display device regardless of the resolution or the screen size of the device on which the image is displayed by determining the relative position of the object included in the image.

Claims (106)

To provide video based browsing and service search Object indexing step; Generating and managing metadata; Creating a service profile; Searching and providing service; and Computer system and implementation method comprising an image processing apparatus for this. The method of claim 1, The object index step Analyzing an image of a frame extracted from the image; Mapping image pixel information to an area of a virtual cell; Analyzing image pixel image information allocated to the virtual cell; Identifying a set of adjacent cells among cells having the same image analysis information as one object; Analyzing the object identification information and indexing the object as an object; Classifying a scene using analysis information such as an object and a background identified using image identification information of an image frame; And storing the analysis result in a storage table. The method of claim 1, How to create and manage metadata Adding meta data to the object identification data table; Verifying an added feature and attribute information in the input metadata of the object; Creating a search rule by calculating attribute information and feature information of added metadata; Creating a hierarchical structure between objects using feature information of metadata; Storing hierarchical information including a logical object created by the hierarchical structure; Creating an association between an object area and a service required for information; And generating a service context by analyzing a plurality of object metadata in a frame. And a meta database including attributes and qualities of an object managing attribute information and metadata of the object. The method of claim 1, Creating a profile for the service Service metadata input process; Generating connection information between objects, including logical associations and relationships of the objects; Generating various service information about the object and the sub-object; Generating a control command for providing a service; and Storing the created service profile; and And a computer operating according to a program for performing a service editing function for inputting and managing desired metadata. The method of claim 1, To provide search and service Providing image information; Inputting search query information; Determining a search query input; Generating an input query data format; Transmitting query request data to an image server; Interpreting the query data; Retrieving service information of the query data; And providing a search result to the terminal. The method of claim 1, Image processing unit A video server providing video information; A database management system for managing metadata for providing a service; Including a video display terminal; The display terminal includes a control command code for dynamically generating control command information on the object selection information, and a management storage method and apparatus for managing cell information and pixel mapping information. Receives object selection information included in the image displayed on the screen and transmits the object selection information to the image server; The video server And an additional service according to the object selection based on a search result of a database management system for providing preset service connection information for the corresponding object based on the object selection information. The method of claim 2, The method of analyzing the image information from the image extracted from the image frame Detecting a size of a screen by checking pixel information of an image screen; Extracting a pixel unit suitable for the split information from the screen size having the width and the height of the screen based on the preset split information; Allocating a cell area of the virtual analysis table to manage image analysis information; Mapping pixel segmentation information of an image to cell information; And the number of divisions of the virtual cell corresponding to the frame is a multiple of an integer. The method of claim 2, A method of analyzing video image information pixels mapped to cells. Extracting a frame for image analysis with a predetermined rule; Analyzing the pixel coordinate region divided from the extracted frame by using cell mapping information using analysis information such as color, texture, and boundary lines; Dividing a cell by a multiple of two into subcells when a plurality of analysis information exists in one selected cell; Dividing and analyzing at a predetermined depth of division until a single image analysis property is detected; The result of the video image analysis may include storing image analysis information such as color, texture information, and boundary information and single analysis determination information; And when the analysis information of the lowest subcell approaches a single object determination ratio in the storing process, a plurality of analysis information are determined as single analysis information. The method of claim 2, The object identification process Analyzing arbitrary cell information in the frame image analysis information; Determining whether the cells have consecutive adjacent surfaces and have the same image analysis attribute information; Extracting polygons from cell determination information; Determining a simple object by analyzing object properties from the extracted polygons; In managing the object identification information in the frame unit, the object identification information is stored as binary summary information in the virtual cell area. And a polygon database for polygon extraction for identifying an object by calculating a relative distance between a connection angle between consecutive adjacent cells and a vertex at which an angle variation occurs. The method of claim 2 or 9, wherein the simple object index management method is A cell generation process corresponding to the frame; Identification information allocation in a cell region by analyzing image information; Expressed as cell coordinates and identifying information in the area this is

Represented by

Is index information of the object 'O' in the object top cell;

Is the reference coordinate of the divided subcell area of the object 'O';

Is the subcell division number,

Is the division depth of the subcell;

Is the area coordinate of the highest cell where the object is located;

The object index summary data; object index management method characterized in that. The structure of claim 1, wherein the structure for object indexing is For virtual cell assigned to target frame area Setting object identification information; A cell area coordinate generation process including an object based on the identification information; Assigning an object identifier to the cell region identification object; and Creating an object index data table for the identified object. The method of claim 7, wherein the cell segmentation in the image information analysis

The coordinate at which the cell is represented by

Has a number n of divisions per unit cell and a partition depth m in the area (x, y) of the cell containing the target object, where X is the number of x coordinate cells including the reference cell, and Y is the y coordinate cell containing the reference cell Wherein n is the number of subcell divisions in the upper cell and m is the division depth of the subcells. The method of claim 1, 2, 7, wherein the cell region mapping of the image analysis information is By mapping the segmentation cell information of the end point coordinate area from the start point coordinate of the frame pixel coordinate of the video, the mapping pixel coordinate area per cell is managed. A cell area mapping method of image information, characterized by having a function or procedure for mapping a pixel coordinate of a screen and a relative coordinate of an object. 13. The method of claim 7, wherein the dividing process of the cell When one or more pieces of analysis information exist as a result of analyzing image information of the cell After dividing the subcell into a predetermined number of lower-level subcells, analyzing the image information of each of the lower-level subcells, and repeating the analysis of the image information of the subcell with respect to the lower-level subcell having the single analysis information. Sub-cell division method characterized in that performing. The method of claim 12, wherein the upper cell division number is Cell division method that divides a cell from a frame and divides the cell into multiples of integers. The method of claim 15, wherein the upper cell division number Cell division method characterized in that divided into four according to the depth of division. The method of claim 8, The cell analysis method is And when at least one analysis information exists as a result of the image analysis of the maximum level sub-cell, storing any one of the image information of the corresponding cell as the image information of the corresponding cell. The method of claim 8, wherein the cell analysis process When one or more pieces of analysis information exist as a result of analyzing the color, texture information, and boundary information of the selected cell, one piece of analysis information that is greater than or equal to a predetermined single object determination ratio among the pieces of analysis information is processed as image information of the corresponding cell. Cell Analysis Process for Object Indexing on Images . The method according to claim 8, 17 and 18, Image analysis processing procedure Analyzing the size and image quality of the frame to determine cell division and division depth; Obtaining an image size of a cell divided into cells from the frame image; Analyzing images from the divided cells in order to determine whether the image analysis information of the cell is single or has a plurality of analysis attributes; Dividing the cell into multiples of an integer if the target image analysis information is less than or equal to an object determination ratio threshold; Determining cell image analysis information and storing image attribute information corresponding to the cell; Storing the analysis result of the subcell in a result table; And repeating the above process until the last subcell is found. The method of claim 9, wherein the simple object determination process In identifying the object, the upper cell comprises a multiple of two of the lower cell; Determining image analysis information of the selected cell; Dividing a cell according to an image analysis result into a plurality of sub-cells of a predetermined integer; Identifying a set of cells as an object based on an approximation of analysis information of adjacent cells; Storing the single image analysis information image information of the divided cell; And extracting an index frame from an image. The method of claim 20, Image analysis information A color, texture, or border boundary analysis checks for the presence of one or more pieces of analysis information. When upper cells have the same color, texture, or continuous outer border with at least one contiguous adjacent cell on one side, And determining that the corresponding cell has the same image analysis information as the object area within the frame. The method of claim 20 Object identification refers to an object identifying a set of cells in which the adjacent lowest or upper cells are adjacent cells having at least one surface and a contiguous surface, and having a same color, texture, or a series of cells having a continuous outer division boundary. Object identification method, characterized in that. The method of claim 9 wherein The object identification information storing process may include managing cell information about relative positions included in the image; In the method of storing the object identification information, the step of configuring the designated cell in the same frame And storing at least one object in a cell at a frame as binary summary information. 10. The method of claim 9, wherein the polygon extraction is A polygon model database for storing and managing polygon models; and The method of claim 1, further comprising: storing the polygon index information of the object by comparing the similarity of the polygon database with respect to the extracted simple object polygon. The method of claim 24, wherein the polygon coordinate management is Analyzing a continuous adjacent cell at a reference coordinate; Determining subcells of the analysis cell and obtaining the constructed oblique angles and distances of adjacent subcells having the same attribute by using a trigonometric function; Determining a variation of an angle formed by the reference cell and an adjacent cell in the step; and And storing ratio information of the cell coordinates of the vertices and the polygon planes. 25. The method of claim 9, 24, wherein object identification and polygon recognition are Obtaining reference coordinates of the object division cell including the object from the background and the object identification information and the image attribute information from the stored object identification information; and And indexing an adjacent surface of each cell within a critical transition angle to a vertex. 27. The method of claim 26, wherein the critical shift angle is 2. A method of determining a polygon according to a polygon, characterized in that a decision is made on a transition angle by a reference value for determining whether the plurality of adjacent cells to be identified is a straight line with respect to an angle transition formed from a starting point. 25. The method of claim 9 and 24, wherein storing the polygon extraction results is After the polygon analysis between adjacent cells is completed, storing the object identification polygon and obtaining the center coordinate cell of the object; and Determination process for similar polygons in the polygon database detected; and Assigning a representative object identifier to the identification polygon object; and And storing the identification result in the object table. In the method of extracting a frame for object identification and extraction in claim 8 Specifying processing options for frame sampling, and A method of sampling a frame by using a variation of predetermined frame analysis information before and after a predetermined frame selection period based on variation information of a predetermined frame. The scene division according to claim 2, Selecting a reference frame to select transition information of the front and rear frames of each frame; Setting reference analysis information of the object identification information; Determining whether the variation information of the frame is within an allowable range; and And dividing the scene by using the analysis result information. The method of claim 30, wherein the frame variation information analysis Identifying a detection index object in an adjacent frame; Disparity information analysis method comprising the process of analyzing the disparity information of the object in the frame. 32. The method of claim 31, further comprising: determining object variation information in an adjacent frame in a sampled frame. This includes the phase change of the center cell of the frame object, the change of the number of objects in the frame, the change of image property information, and the background object. Frame object variation information determination method for scene segmentation using the at least one analysis information. 31. The method of claim 9 wherein the scene segmentation is And determining a scene segmentation period of the frame by determining similarity of disparity information between the analysis information of the reference frame and the target frame. The method of claim 2 wherein the cell generation method Allocating to the most significant cell of multiple of two corresponding to the pixel area of the frame; And sub-cell dividing by determining image analysis information in a pixel area of a frame corresponding to each of the top-most cells. 35. The method of claim 34 wherein the subcell partitioning is Cell segmentation process based on the ratio of simple object determination to image analysis information; and A subcell division method comprising dividing the subcell into two multiples of the cell to be analyzed. 36. The method of claim 35, wherein And a dividing depth for dividing the subcell by a specified dividing depth in dividing into subcells. The method of claim 10, wherein the virtual cell management method Managed by binary representation of object identification in cell partitions divided by multiples of integers.

Each parent cell coordinate

The object identification information management method, characterized in that for managing the presence or absence of the object identification as a binary '0', '1'. 38. The method of claim 37 wherein object analysis result processing is An object analysis result processing method characterized by managing the object index information by the (X, Y) coordinate region of the upper cell and the subcell. 39. The method of claim 38, wherein the object partition index information of the cell is By managing object index information represented by binary bits equal to the number of divisions of a subcell in a matrix coordinate region corresponding to an upper cell. The index information of the top cell indicates whether or not the object of the top cell is detected.

Object detection upper cell expression method expressed as

Object sub-index information management method characterized in that the sub-cell representation method expressed as. The method of claim 23, wherein the identification cell information of the object Identification object management method characterized in that it is binary object pattern information and managed through similar frame object search, image copyright information index target information through this. 36. The index information processing of a cell according to claim 10 or 35, wherein And if the object detection index information of the upper cell includes the object index information in any cell of the subcell area of the lower cell, the upper cell indicates this. 12. The object index data structure of claim 11, wherein the object index data structure is A frame object extraction data table; Image main table; Scene dividing table; frame table; Object table; Sub-object table; Inverse frame object set table; to store and manage the image frame analysis information, In this case, the image analysis is a database configuration and implementation method for generating and managing a plurality of data for analyzing and managing the image information including a markup language such as XML (eXtensible Markup Language) format. 43. The method of claim 42 wherein the frame object extraction data table And a video title title, frame index identifier, object identifier, object identification cell vector, document caption information, and language analysis index information. 43. The method of claim 42 wherein the video main table is Video main information table is used to identify the target video in the Video_ID video server, Disp_Size Screen size of the video, Run_Time video running time, Cell_No_Depth Repetition split depth of the subcell that divides the video, Video_Title video title, Idx_Term video A method of constructing and processing a video main table including an interval for extracting and indexing frames, a number of divided scenes in a No_Scenes video, a total number of frames in a No_Frames video, and a category classification system for the corresponding video. 43. The system of claim 42, wherein the scene partition table is Scene_Id Scene Classification Identifier, Scene Start Frame Start Frame, Scene End Frame End_Frame, Scene Partitioned Time Scene Time, Key Objects in the Scene, Object Set Object Set, Control Command Control Command, And a scene context identification identifier, Scene Context Id, and including at least a plurality of the information. 43. The method of claim 42 wherein the frame table is Frame ID, Frame Index, Frame Index, Frame Time Frame Time, Frame Object Set Object Set, Control Command Control Command, Frame Context ID Code Frame Context ID and Service Anchor to handle the service. The frame identifier identifies a specific frame area in the image, The frame index manages the relative coordinates of the object in the region divided into cells in the frame. The frame time is time information where the corresponding index frame is located, The frame object set is a set of object detection cells in a cell, The control command dynamically manages various search options and functions in the terminal, The frame context ID is a context identification information management key of a corresponding frame; The service anchor is a service reference key for processing by referring to object information of the frame and service information provided to the frame area. And a frame table comprising at least a plurality of the information. 43. The method of claim 42 wherein the object table is It includes object identifiers, object description names, frame identifiers, object indexes, object patterns, polygon extraction types, control commands, object contexts, feature sets and service anchors. The object identifier is assigned to all objects extracted and identified in the frame. The object description name is an object name, the object index is an object index, index information such as polygon and image color attributes including coordinates of an object subcell, The object pattern Object Pattern is extraction information representing the pattern of the object detection subcell in binary bits, The polygon extraction type Polygon Ext. Type is the object feature extraction information based on the ratio of vertices, faces, and elements of the extracted polygons by analyzing the shape elements of the extracted cell area for each object, Object Context Object Context is information about the context within an object's frame, Feature set Feature set is set information including various property information of object. At this time, feature set expresses all feature sets of sub-objects included in one object and processes them as concept of set. Object table configuration method comprising at least a plurality of the information. 43. The method of claim 42, wherein the subobject table is A subconstructing object of an object, including object identifiers, subobject color identifiers, subobject cell coordinate areas, control commands, feature sets, and service anchors. And configuring at least a plurality of the information. 43. The method of claim 42, wherein the inverse frame object set table In order to manage and retrieve the information of the object located at the corresponding coordinates with a reversed mapping table for the frame table. It includes frame identifier, control command, frame context, object summary offset, number of object detection, object identifier and object identification coordinate. The object summary offset is used for the purpose of searching for the same and similar information and managing the copyright of the corresponding frame by summarizing the overall object composition and background and image analysis information in a specific frame.  And configuring at least a plurality of pieces of information. The method of claim 3, wherein the object feature property in the metadata is Representative object qualities including the object-specific representative attribute classification; General attribute of representative object; Attribute properties of relationships between objects and objects and subobjects; Component attributes including the object's behavior, time, place, parts, conditional components, etc .; And a special feature for defining a special or unique property value of the object. 51. The method of claim 3 or 50, wherein the classification feature attribution method Assigning representative object feature values, general attribute properties, components, and related qualities to the analysis object information on the extracted objects in the frame; Assigning a feature value according to the index similarity of the image analysis information and the polygon; Assigning a feature value to a background object if the feature values of all detection objects in the same frame are given; Through the process of determining whether the given feature value is appropriate or unregistered object; Managing the object attribute by the feature set pattern; and The feature attribute value is a method for processing the presence or absence of a feature attribute of a detailed item with respect to a corresponding feature category; And the feature detail attribute presence or absence includes a method of managing a feature attribute with a binary value. 52. The method of claim 51, wherein the feature-based calculation method of claim 51 Determining the presence or absence of detailed feature items for each feature classification of the object; Applying an association processing rule between the object and the object or the sub-objects of the object; An association rule processing process of a plurality of objects and a plurality of object features; Through the process of identifying the situation and event by the pattern matching operation rule for the feature pattern between the plurality of objects; Computing a property value of the qualities and producing a result. 53. The processing rule database of claim 52, wherein the processing rule database for feature-based operations Setting a feature pattern extraction condition between the plurality of objects; A process of applying a processing algorithm according to an extracted feature pattern to recognize a situation and process various additional services by analyzing a relationship between feature and an attribute feature of a feature; A database defining an algorithm processing generation rule according to the feature pattern condition; The rule database for object situation and event recognition is a calculation rule database generation method, characterized in that to calculate a condition according to the presence or absence of a plurality of objects as a rule to derive a result to recognize the situation or event. The method of claim 3, wherein the additional text document or subtitle processing method for the image is performed. Determining whether a subtitle or video-related additional explanatory text document exists; Extracting text from documents or subtitles; Generating feature vectors through linguistic analysis and processing; Interoperating the semantic feature information and the lexical feature information extracted from the image frame; Mapping event and situation information by analyzing the analyzed feature vectors and the feature information of the frame object; The object feature information analyzing process includes: analyzing semantic features of a plurality of index words; and Associating the index word with the object to the object that matches the context of the frame, that is, the context; Generating an event and context information in a corresponding frame through feature computation between objects; and A text document information processing method for an image comprising the step of storing the analyzed result. The service editor in claim 4 An image display display unit displaying an object identification area in the frame cell division area; A frame and object feature attribute management unit identified in an area of the cell; A scene division display unit of an image; Designating a context and feature property of a scene, a frame, and an object, and a video service preview editing management unit; A service information retrieval unit for a representative object, title, context, and text index; A registration service meta information editing management unit; And a control command code manager for setting a control command interface required to provide a service. The object association information of claim 4, wherein Creating a service context; Creating a service profile required by the user; Generating a hierarchical structure between objects by using information such as attribute information and relationship information in the feature information of the objects; Storing hierarchical information including a logical object generated by the hierarchical structure; and Service association information management method using object area cell selection information. The method of claim 4, wherein the service profile generation process Object-related service call result processing information detected in the indexing process, To call motion information or a specific context The service profile is characterized by including operation information for each condition. The method according to claim 2 or 5, to provide image retrieval service information for retrieval and service.  Object index data structure; and Object qualities and service information management data structures; It has a service attribute information management data structure. The quality and attribute information required for service according to claim 5 or 58 A category table that manages video taxonomy and service categories, Service table that defines and manages motion information and rules for connection service provided in the object or frame, A control command for dynamically providing an object selection or a user search option according to a scene on the image frame; A context table for storing and managing contextual information in a scene or frame through identification of the image frame or object; Identification index information of the object in the frame and cell coordinate information linking operation information object index table of the object; A language processing index DB for document subtitle information related to the scene or frame, And a plurality of pieces of information of the information for the service including the object attribute and the feature table for managing the feature information. 60. The control command table of claim 59 wherein Applicable information management table to provide various command input interfaces to service search or video server. And at least one of operation setting information according to cell selection of a frame, control command interface in a frame area, cell area setting information, control command interface management rules through cell selection, or voice input. 60. The context table of claim 59 wherein It is for the situation recognition service through the operation of the feature information of the object by managing the situation information of the scene, frame and object And a plurality of pieces of information of a context classification identifier, a feature matching rule, a matching condition, and a primary context assist context. 60. The method of claim 59 wherein the object index table A data table for managing motion information linked to object index information according to specific region cell selection information in an image. It manages cell coordinate information and service information of extracted object and identification information of feature set information of object. Through this, object corresponding to specific pointing coordinate is extracted from scene and frame to search and provide related services. A method of constructing an object index table comprising frame identification object information, condition information that the object has, a service identifier connected to the object, an index word for subtitles or additional document information of an image, and object connection information. 60. The method of claim 58 or 59, wherein the object qualities As a dictionary to manage the qualities according to the classification system A feature dictionary for a representative object, characterized by having an object identifier, a general feature, a related feature, an attribute feature, and a special feature.  64. The object feature table of claim 58, wherein And a detailed feature value for each feature classification, wherein the feature value is configured as a binary value to process Boolean and search for a plurality of objects in the image through binary operation. The feature table of claim 64, wherein A method for constructing an image object feature table comprising: processing attribute property information required for object analysis and language analysis by configuring a plurality of object features and language analysis feature data tables. The video information based service according to claim 1 or 5 An image information retrieval service method comprising providing information transmitted through an input means to an object or related information in an image. The means for inputting the object or related information in claim 5 or 66 An image object service method comprising allocating a cell region mapped to an object and pointing to the allocated cell as an input means 68. The service data table of claim 66 or 67 wherein the service data table is  And a service identifier, service condition information and service operation information for providing various services according to the condition. 60. The polygon data table of claim 59, wherein A resultant sample data table for object identification, comprising a number of vertices, features of adjacent angles, aspect ratios, and the like. 70. The polygon reference table of claim 69 And extracting an attribute reference value of an object by comparing an approximation of values of object polygon information extracted from an image. 60. The method of claim 59, wherein the image-related text information or subtitle information is Image text information processing method characterized in that the mapping with the context information, object index information of the extracted object in the image. 72. The method of claim 71 for processing text. And a linguistic analysis reference dictionary for mapping keyword, event, and context information generated through linguistic analysis processing to a frame, an object, a scene, and the like. 60. The method of claim 59 wherein the control command processing step is Generating a control command according to the image identification information; and Transmitting control command information according to video streaming; Generating control command information cell mapping information in an image frame; Generating control command information selection information; A control command processing method comprising the step of interpreting transmission control command information. 74. The method of claim 73, wherein the control command information display is In setting the area to display the control command code transmitted to the terminal, A control command display method, characterized in that the terminal dynamically allocates a display display area to an area of information provided or an object display cell area. The method of claim 2, wherein the image image pixel information is mapped to a cell region. Extracting pixel information of a video image; Determining cell division and division depth in an image frame; Analyzing a relative position of a cell corresponding to the image pixel; Extracting image pixel coordinates of the terminal screen and position mapping information of the cells; and Cell region mapping and processing method for detecting object coordinates from user cell selection information. The method according to claim 1 or 5, wherein the video information service profile is generated. Generating association information between objects; Generating various service information about the object and the sub-object; Generating a control command for providing a service; and And storing the generated service profile.  The image retrieval service of claim 1 or 5, Determining the input value for the terminal search; Generating an input query data format; Interpreting the query data format; Generating a control command code to the terminal; Performing a query search; A video service searching method comprising a plurality of steps of transmitting and displaying results. The image search service input of claim 1 or 5 is And a binary input method including inputting a cell ID, text, voice information, and image image information according to selection of a cell partition by detecting object identification information included in the image image. 78. The method of claim 1 wherein the data format for a search query is The generated data format follows the packet rules in the communication network, The protocol identifier, session ID, message type, reserved field, video ID, frame ID, cell ID, and payload length fields and payload, which are common header parts of the packet, include a user identifier (ID), natural language query text or voice query data, A method of generating a data format for a query, characterized by a plurality of data formats such as codes for authentication. 80. The method of claim 79, wherein the search query data in the image frame is A method for generating an image query data format characterized by virtual cell information corresponding to pixels of an image and subsidiary information thereof. 80. The method of claim 79 wherein the message type is And defining types of various input messages regarding cell pointing, control command processing, query input or voice input in the video. The object query information in the video service according to claim 1 or 5 is The object selection information in the frame includes a frame identifier and a cell identifier. 83. The method of claim 82 An object information management method comprising a function or procedure for mapping pixel coordinates of an image frame and relative coordinates of an object, and in particular, managing coordinate information of an object as a cell. 6. The service search of claim 5 wherein A service search method comprising searching for a service before and after a frame at which a search input information, that is, object selection pointing, voice, or a keyword occurs. 79. The method of claim 78 for preprocessing of text input. A process including word division, stem-mother division, etc. of the input query statement; and Lexical analysis and key word extraction process with reference to lexical dictionaries and rule dictionaries; And weighting the lexical components having similarities by referring to the extracted lexical component and attribute feature values and the feature value of the object attribute of the frame and the context of the proximity frame when generating a query in the terminal. In the weighting process in, the search query text input has a rule for extracting the feature of the lexical component existing in the text query query by the user by searching the object feature information of the nearest frame when generating the text-based query in the terminal. Pretreatment method. 86. The method of claim 85, for generating a search query A search query generation method, characterized in that the feature of the included object is compared and extracted by referring to the scene context of the scene partition table and the frame table frame context. About binary query input Analyzing the query data; Generating a search query; binary query input and processing method comprising a.    The method of claim 5, wherein the video service search Parsing a query data format having a processing procedure according to an input value by analyzing a message type of a message data format and a transfer value such as a cell ID and a payloader; Retrieving object information located in a specific area from an index value in a frame table in the query search data;  Generating a service code by searching the service anchor 73 corresponding to the object index value in the corresponding object table and the sub object table; Input information analysis step for performing a search; Searching for a service code corresponding to necessary condition information in a search index DB and using the searched service code; The service search system, characterized in that for performing the search procedure (S790) for the corresponding service of the service DB according to the control command information input conditions. The method of claim 6, wherein the video server An input unit for receiving image information; A video converting unit for converting the input image information into an encoding or an appropriate format and storing the image information in the image information storage unit; An information indexing unit for detecting object information from the stored image information and matching and storing a frame of the image information of the image information storage unit and an object area and connection information in the frame; An image information storage unit configured to store image information including the object information, cell information which is screen division information of each image, and quality property and service link information included in the object; An index information manager configured to manage the meta data of each of the object information and the hierarchical relationship between the object information; An object feature register for managing and assigning features and property values of object information; Object feature information management database unit; Index and service information database manager; A service register for connecting various services to an image and managing mapping; A search provider for searching the video information storage based on various request information; A service request interpreter for interpreting and processing a service request; A result output unit for extracting and processing the result to transmit the search result to the terminal; and And a network connection unit for providing an interface with a communication network. And a control unit for controlling the operation of each unit. 90. The apparatus of claim 89, wherein the image information storage unit Object identification information, image identification information containing the object, configuration cell information including identification information of each divided cell constituting the object, area of the object, coordinates of the center point of the object, phase transition information and image of the object Image information storage method for storing a simple object information including a property. 91. A method according to claim 89 or 90 for storing and managing image information metadata. Constructing object feature database and processing rule database as electronic dictionary Meta data characterized by storing and processing simple object-specific metadata including object identification information, image identification information including the object, classification information of the object, link information, object detail information and operation information according to the event. How data is stored and managed. 90. The apparatus of claim 89, wherein the image information storage unit And storing cell identification information including image identification information, identification information of a cell which is a screen division unit for the image, and start coordinates and end coordinates of the corresponding cell together with the corresponding image information. . 90. The apparatus of claim 89, wherein the image information storage unit Storing logical object information including logical object identification information, image identification information including the logical object, classification information of the logical object, identification information about simple objects included in the logical object, and motion information according to an event How to manage video service index information storage.  90. The index unit according to claim 1, wherein the index unit Detecting a relative position of objects included in the image, and storing simple object information including the position and image information of each of the objects represented by cell information in the image information storage unit. Indexing system.  95. The apparatus of claim 94, wherein the information index unit Detecting a base screen size for an image; Dividing the screen into a plurality of virtual cells based on preset split information; Analyzing image information of each of the divided cells and recognizing a set of adjacent cells among cells having the same analysis information as one object; And storing each of the recognized simple object information. 90. A subtitle or related document information in an image information index according to claim 1, A method of indexing text information in video information comprising linking index keywords extracted through language processing and analysis with video frames and object information, and including object feature information and semantic information including corresponding cells. 90. The system of claim 89, wherein the index information management unit Means for receiving simple object association information from metadata for each of the simple objects; and storing hierarchical information including virtual logical objects created by the simple objects hierarchy. 90. The logical object metadata of claim 89, wherein And a process of including pixel position mapping information of the screen with respect to the virtual cell of the object, attributes and feature information of the object, and feature properties necessary for extracting association information between the object and the object. 90. The apparatus of claim 89 wherein the service register is Generating metadata using image frame analysis image information, detected object cell information, polygon information, and object corresponding feature information; and A service registration system for storing the extracted results of the object, frame and scene context in the image information storage unit. 99. The method of claim 99, for generating and managing service metadata. A service metadata management system comprising a feature information management DB including an operation rule DB, a feature analysis DB, and a language analysis DB. 90. The apparatus of claim 89, wherein the service request interpreter A service request parsing system for parsing an input request information type having means such as object selection, query input, and voice input, and performing procedures of pointing, query and speech recognition according to the analysis result. 90. The method of claim 89, wherein the search results are Searching in an index database and a service information management database; Formatting the terminal output format through the result output unit; A method and system for outputting a result serviced to a user terminal through at least one of a mobile communication network and a wireless network connected to a network connection unit. The method of claim 6, wherein the terminal of the image processing apparatus A display image information output display unit including a display screen divided in cell units; A search information input unit providing a plurality of input means; An input information analyzer for generating a message data format with respect to the input information; An input information generator for generating query data for the input query intent input; A network connection for providing interfacing with a communication network; And Terminal system device and an implementation method comprising a result output unit for outputting a result transmitted from the video server. 103. The apparatus of claim 103, wherein the input information generator The input information includes an image identifier; Frame identifier or time information; Cell information of the object location; An input information generation system comprising binary query input information including control command selection information, keywords, voice and images, and the like. 107. The method of claim 104, wherein the input information generation is Identifying a frame or scene identifier; A method for generating input information, characterized in that the command control information to be provided in common and the command control information to be provided in a specific frame are divided and transmitted in synchronization with the video information of the terminal. 107. The method of claim 104, wherein the input information generation is And a means for additionally selecting a search condition for searching for information on a selected object, searching for a related video, and linking link information.

Images