JP2005276004A - Image information processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image information processing system in which a user is capable of efficiently acquiring various kinds of required image information from a large-scale image database. <P>SOLUTION: An image information processing system comprises a target knowledge database 150, a target knowledge database 152, an image database 152, a search processing section 120, a knowledge processing section 130 and an extraction processing section 140 for dealing with information of a target, and includes a means for registering query relating to the situation (context) of the target on an image or a time sequential pattern of the target spread over images at a plurality of photographing timings and reservation queries thereof. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image information processing system for acquiring information of an image and a target image taken on the image from an image database in which a large amount of image data is accumulated.

  In recent years, large-capacity image data having a large size has been widely used due to the improvement in processing performance of computers. For example, it is a field of satellite images, surveillance camera images, observation record images, medical images, and the like. In particular, satellite images are a typical example, and the data capacity is increasing dramatically due to higher resolution with increased sensor performance, increased number of bands, and increased number of bits associated with luminance gradation expansion. .

  In addition, large-scale and time-series images that automatically accumulate the above-mentioned large-capacity image data through development of hardware technologies such as large-capacity storage and large-capacity networks, or construction technologies for large-scale systems. It has become possible to construct a database system. In particular, for satellite images, since the satellites orbit the earth and continue to capture images, the database continues to grow due to the accumulation of such time-series data. In recent years, the launch of commercial high-resolution satellites and the like has increased, and the number of data source satellites itself has increased, so it is inevitable that the database will be enlarged.

  In order to utilize such a large-scale image database, a technique for efficiently acquiring image data required by the user is required. For example, Japanese Patent Laid-Open No. 11-282861 discloses a conventional technique related to an image search navigation function using satellite image attribute information and a map. Japanese Patent Application Laid-Open No. 2001-195565 discloses a conventional technique for searching for satellite images using target information as a key.

Japanese Patent Laid-Open No. 11-282861

JP 2001-195565 A JP 2001-307106 A Japanese Patent Laid-Open No. 11-312233

  In the time series image database, information of various targets photographed on the image, that is, image information is implicitly accumulated. If such image information can be efficiently extracted and used, advanced application of the image database, such as wide-area environmental research and decision support, becomes possible. However, in the prior art, search target image data can be narrowed down using attribute information, or individual targets on a single image can be searched. However, using knowledge information on a target, multiple target objects on the image can be searched. It was difficult to obtain information on “situation (context)” and “time series pattern” such as a change situation across images at a plurality of shooting periods. That is, there is a problem that it is difficult to efficiently acquire a wide variety of image information required by the user from a large-scale image database in which a large number of large-capacity image data is registered. It is an object of the present invention to provide an image database utilization technique and an image information processing system that solve these problems.

  In order to solve the above object, the outline of the invention disclosed in the present application will be described as follows. An image database that manages an image in association with attribute information including position information and time information of the image, a target knowledge database that manages an object name and target knowledge in association with each other, and the target knowledge Means for extracting information on a target on the image using a database, means for managing the image in association with the target information, means for constructing an index of the target information, and a search query Means for associating the search query with the target information using the target knowledge database, means for searching for the target information using the target information index, And a means for outputting a search result and a means for outputting the image associated with the search result.

Further, the target information search means of the present invention extracts target information to be searched using the target knowledge database and the target information extraction means, and constructs the target information index. A means for searching for the target information using an index is provided. Further, the target information search means of the present invention comprises spatial analysis means or spatio-temporal analysis means, and the search query registration means of the present invention includes a query regarding the status of the target on the image, and a target that spans images at a plurality of shooting periods And a means for registering the reservation queries.
With these configurations, it is possible to realize an image information processing system that can efficiently acquire an image and information about a target on the image from an image database.

  According to the present invention, it is possible to acquire image information and image data using the status of the target on the image and the time-series pattern as keys, and it is possible to efficiently use a large number of image data stored in a large-scale image database. . Further, by registering a reservation query, image data can be examined only when a scene required by the user is obtained, and a vast image range can be confirmed efficiently. In general, a large-scale image database requires a huge amount of processing time to search for target information. However, according to the present invention, a high-speed information search can be performed by utilizing a target information index. These effects make it possible to grasp the latest situation and overview past similar cases, and support advanced analysis and decision making by users.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Here, in this application, “target” refers to features and moving objects and their time series patterns, “features” refers to buildings, roads, vegetation, coastlines, etc. “moving objects” refers to airplanes and ships. A “time-series pattern” such as train, vehicle, etc. means deforestation or increase / decrease in traffic (number of moving objects).

  FIG. 1 is a diagram illustrating an example of a functional configuration according to an embodiment of the present invention. The system includes two basic components, that is, a user terminal 100 and an image information processing system 101. The image information processing system 101 further includes means such as a processing management unit 110, a search processing unit 120, a knowledge processing unit 130, an extraction processing unit 140, and several databases. The image information processing system 101 receives the search query 102 from the user terminal 100 at the processing management unit 110, interprets the query 102 via the knowledge processing unit 130, and executes the corresponding search processing at the search processing unit 120. Then, the processing result 103 is returned to the user terminal 100.

  The processing management unit 110 includes a user information management unit 111, a data input / output management unit 112, and a processing history management unit 113. The user information management unit 111 manages information such as customization status and processing status for each user. The data input / output management unit 112 receives an operation input or query input from the user terminal 100, starts a process associated therewith, and outputs a processing result to the user terminal 100. Furthermore, input and update of image data and knowledge data are also performed. The processing history management unit 113 manages processing history information related to the extraction processing in the extraction processing unit 140, and collectively manages the extraction time and space range, the extraction method used for the extraction processing, or version information of the extraction means. This makes it possible to check whether there is a process to be extracted when new image data is input. In the process history information, in addition to the version information of the extraction process, the process execution time and user name may be managed.

  The search processing unit 120 includes a management unit 121, a query processing unit 122, and a search processing engine 123. These functions execute a search process of the target information index 150 in cooperation with each other. The management unit 121 manages the target information index 150 and the status of search processing. The query processing unit 122 receives a query as a search request from the user terminal 100, interprets the search processing content appropriately, and starts the search processing engine 123. The search processing engine 123 searches the target information index 150 and returns a search result. Here, since the target information is not obtained directly from the image but is searched mainly using the target information index 150, it is not a so-called image search engine, but a space such as a geographic information system (GIS: Geographic Information System). An analysis engine or a spatio-temporal analysis engine such as a four-dimensional geographic information system (4D-GIS: 4 Dimensional-GIS) may be used as the search processing engine 123. A powerful spatio-temporal analysis function is obtained by the configuration of the image search function having a spatio-temporal analysis engine in the back end, and advanced search processing using the target situation, time series pattern, etc. as a key is possible.

  The knowledge processing unit 130 includes a management unit 131, a query processing unit 132, and a knowledge processing engine 133. The management unit 131 manages the knowledge database 151 and the status of knowledge processing. The query processing unit 132 interprets a query input by the user, identifies a target intended by the user, and activates the knowledge processing engine 133 to acquire knowledge data of the target. The knowledge processing engine 133 uses the knowledge database 151 and various inference engines to specify the knowledge data of the target that the user needs, specifies an appropriate search keyword in the target information index 150, and causes the search processing unit 120 to A process in which a plurality of functions of the system are linked is realized, such as starting or specifying an extraction method suitable for extracting the target and starting the extraction processing unit 140.

  The extraction processing unit 140 includes a management unit 141, a query processing unit 142, and an extraction processing engine 143. These functions cooperate with each other to realize target information extraction processing and target information index 150 construction processing. The management unit 141 manages the status of the extraction process. The query processing unit 142 interprets the query, cooperates with the knowledge processing unit 130 and the like to acquire an extraction processing method suitable for extracting target information required by the user, and starts the extraction processing engine 143. . The extraction processing engine 143 executes target information extraction processing and target information index 150 construction processing, and returns an extraction result. In addition to the search request from the search processing unit 120, an extraction instruction may be received directly from the user terminal 100. Of course, not only automatic extraction processing but also manual or semi-automatic extraction processing may be executed.

  The database means uses a plurality of databases in cooperation. The target information index 150 efficiently manages various types of index information. In particular, in the search processing by the search processing unit 120, high-speed processing is realized using the target information index 150. The knowledge database 151 accumulates knowledge data of the target and provides basic information such as extraction processing and search processing. The image database 152 may be a large-scale database so that a large amount of image data can be managed.

  With the above configuration, an efficient image search system can be realized. In general, as the volume of image data increases, if target information is directly extracted from an image and a search process is attempted, a huge amount of processing time is required only for the extraction process. Also, the larger the database, the more problematic the access performance will be. Therefore, the target information index 150 is constructed and stored when the target information is extracted, and thereafter, the search process is executed using only the target information index 150, and the image data itself is not accessed every time. The configuration can be completed with As a result, even for a large-scale image database 152, efficient search processing can be performed with only the minimum necessary access, and the processing performance of the entire system can be improved.

  Further, with the configuration having the knowledge database 151 and the knowledge processing unit 130 as a common base, the search processing unit 120 and the extraction processing unit 140 can be linked to realize an advanced system function. The user can seamlessly execute the search process and the extraction process using only the query 102 as an interface without being aware of the integration of the search process and the extraction process. For example, when the search target data does not exist, the search processing unit 120 requests the extraction processing unit 140 to extract target information from the image data via the knowledge processing unit 130, and the extraction processing unit 140 performs the extraction process. It is also possible to perform a search process using the target information index 150 generated automatically. Of course, processing control may be performed such that processing setting information or the like is given to the processing management unit 110 as necessary, and only the search processing is executed and the extraction processing is not executed.

  Note that the above system configuration is conceptual and is not actually limited to a plurality of physically divided computers. If a similar configuration is provided, two systems on one computer can be used. It may be a process, or may be realized as a different component in the same process on the same computer. Conversely, the function may be divided into two or more processes or computers.

  FIG. 2 is a diagram showing an example of query registration and search processing results in an embodiment of the present invention. FIG. 2A is an example of a search processing result 200 that uses a target situation (context) on an image as a query, such as “an image showing two or more large ships”. By the spatial analysis of the target information index 150, it is possible to search for an image that satisfies the requested situation (query condition) as if the image data itself was searched. As is clear from this example, the target 210 on the image is a map or a graphic object itself in the spatial data. Therefore, according to the present invention having the search processing engine 123 having a powerful spatial analysis function such as GIS in the back end, it is possible to perform a search process using various situations of the target as a key. In addition, it is possible to perform a search process using a complicated situation as a key, such as “an image in which a large ship and a small ship are shown at the same time” and “a building that is densely and regularly arranged”.

  FIG. 2B is an example of a search processing result 201 using a time series pattern of a target that spans images at a plurality of shooting periods, such as an “image with increased ship traffic” as a query. By the spatio-temporal analysis of the target information index 150, it is possible to search for an image group that satisfies a requested time series pattern as if a plurality of images were searched. Here, the time-series pattern on the image is obtained by finding an old image and a new image obtained by photographing the same area and comparing and evaluating them. According to the present invention having a powerful spatio-temporal analysis engine such as GIS in the back end, it becomes possible to search for a target using various time series patterns as a key. In addition, for the purpose of surveying the traffic of ships, “images of XX bays with the most ships in a year”, “regions where XX became △△ (eg, forest on road)” and “cutting area ○ ○ Search processing by complex queries including modeling of targets such as “Deforested areas over hectares” is also possible.

  FIG. 2C is an example of a search processing result 202 by a reservation query such as “an image in which ship traffic has increased within the next ○ weeks”. Here, by setting an expiration date of the query, it is possible to provide information to the user when the expiration date ends, and to review the search processing contents and manage the system load. Also, an alarm information presentation method suitable for the user's usage environment such as a dialog box or e-mail may be set in advance at the time of query registration. By using the query as described above, the user can efficiently acquire the image information related to the target and the local situation.

  FIG. 3 is a diagram illustrating an example of an image of search processing content according to an embodiment of the present invention. FIG. 3A is a schematic diagram of the database space 300 and the image data 301. The database space 300 includes a horizontal space axis and a vertical time axis, and forms a virtual time space. The image data 301 is managed in association with an appropriate place in the database space 300 according to the position on the earth and the time taken. Of course, the image data 301 shown here can be replaced by a set of target information on the target information index 150 described above. FIG. 3B is a schematic diagram highlighting image data that is a search result 302 by a situation query, which is one of the features of the present invention. FIG. 3C is a schematic diagram in which the concept of situation search is further expanded to search for a time series pattern, and the searched time series pattern 303 and image data constituting the pattern are highlighted.

  In the above example, since the image is searched from the whole database, an unexpected result may be obtained by chance at an unexpected time or place. However, there is a problem that the larger the database, the more time is required for processing. Therefore, an image of search processing by specifying the priority order, which is one of the features of the present invention, will be described with reference to FIGS. FIG. 3D is a schematic diagram in which time priority 304 is designated and the database range to be searched and the image data are highlighted. Thereby, “image data search in the past one year” or the like becomes possible. FIG. 3E is a schematic diagram in which the latest priority 305 of “only the latest image data in each region” is designated, and the database range to be searched and the image data are highlighted in the time priority. There are many uses that require the latest image information, and support for it enables efficient database use. FIG. 3F is a schematic diagram in which the space priority 306 is designated and the database range to be searched and the image data are highlighted. For example, priority can be set by using “Japan, Asian region, world” or “from airports all over the world” and map data or user drawing data. Instead of defining the “order of processing results” as in the prior art SQL (Structured Query Language), the “order of processing itself” can be defined, and arbitrary user control can be performed. Thus, even an application that requires long-time processing using a large-scale database can be processed to withstand actual use.

  FIG. 4 is a diagram showing an example of the structure of the knowledge database in one embodiment of the present invention. The knowledge database 151 is a database that accumulates and manages knowledge about the target. Knowledge is described as knowledge data 402, and is managed by associating various information related to the target object. For example, a knowledge database is like a dictionary, and knowledge data is equivalent to words. The knowledge data 402 is associated with each other by a target classification system and class structure, and is systematically managed 401 as a tree structure or a graph structure. Furthermore, the knowledge data of the target object can be expanded by aggregating the knowledge data of the child nodes to generate new knowledge data, and assigning the new knowledge data to the parent node as the expanded knowledge data 403. For example, common knowledge data obtained by averaging feature quantities of individual targets may be generated and held as parent node knowledge data. As a result, it is possible to determine whether or not the target object is satisfied by simply checking the knowledge data of the parent node at the time of the extraction process, without tracing all child nodes and performing trial and error. It becomes possible.

  Furthermore, an extraction process using the knowledge data of the parent node directly is also possible. For example, searching or extracting using all aircraft as keys. Here, the knowledge database 151 and the knowledge data 402 are assumed to be a database and data corresponding to variable length records. For example, a database that handles XML data that can be easily expanded may be used. By setting the variable length instead of the fixed length, when new knowledge is added or acquired, the knowledge can be freely added to the database. Since knowledge data is described in a general-purpose data format, it can be configured as a different tree structure or graph structure depending on the interpretation method. By dynamically changing the structure according to an arbitrary classification system designated by the user and presenting information in a format that is easy for the user to understand, it is possible to provide a system according to various usage forms. The knowledge database may be constructed as a unique database, but an existing interpretation database in which interpretation samples are stored may be used. In this case, the interpretation sample can be used as a model for extraction processing. Of course, a simple configuration such as a database that simply stores a plurality of images or a rule base that stores extraction rules may be used.

  FIG. 5 is a diagram showing an example of the structure of knowledge data in one embodiment of the present invention. In the knowledge data 402, class information 501 such as the name, keyword, or ID of the target can be managed in association with attribute information 502 indicating the details of the target. Further, as a feature of the present invention, the extraction means 503 can also be managed as knowledge, such as a rule for realizing the target extraction process and a model such as a template included in the rule. As a result, the target can be associated with a processing method or program suitable for the extraction process, and the knowledge database 151 can be a basic function that can be used for various processes. For example, in the search processing unit 120 and the extraction processing unit 140, it is necessary to interpret a query from a user and map it to an actual process from a name, a keyword, and the like. , All processes can be kept consistent. Also, a plurality of sample images can be managed, and the model may be updated by acquiring knowledge about the target using them. For example, the luminance feature amount of the target is knowledge that can be acquired relatively easily, but is an effective clue in the extraction process.

  Furthermore, in the present invention, rules and models specific to each type of image data, sample images, and the like can be individually managed as target knowledge. Thereby, the optimal extraction process according to various types of image data can be realized. As an example of the model, as a model common to the optical system image and all types of images, non-image data such as CAD (Computer Aided Design) data indicating the detailed shape of the target object, drawing data, or an image obtained by scanning them Data etc. can be used. As an example of the model, the template of Patent Document 3 may be used. In addition, for a hyperspectral image or the like, spectral library data relating to the surface material of the target may be held. Alternatively, the spectrum library may be managed as a separate database, and pointers and IDs for the data may be held. Also, for radar images such as SAR (Synthetic Aperture Radar), completely different models such as a statistical model can be handled uniformly. Of course, parameters of extraction processing may be described as a simple model.

  In particular, for time-series patterns, moving objects and features have different models in terms of time interpretation or meaning, such as “movement” and “shape change”. A knowledge data structure that can be handled. Also, more effective knowledge data can be configured by describing not only a set of models but also a model usage method by rules. For example, for a target such as a moving object, an extraction process can be realized by a rule that a template is generated from a drawing or a sample image and a template search is performed on the image data using the template as a model. In addition, for targets that have a wide range of shapes, such as features, a rule is to generate feature values (shape, spectrum, texture, context, etc.) from drawings and sample images, and use them as models to match feature values on image data. What should I do? In addition to the approach from the model side, on the image data side, the shadow situation is estimated from the sun direction and the solar altitude using the attribute information at the time of shooting, etc. It may be an advanced model or rule.

  Here, when the target is automatically extracted or certified by the user, the situation may be monitored, and the sample image, rule, or model may be updated as necessary. As a result, the knowledge database 151 can be realized as the knowledge is accumulated as the user uses it with a simple configuration. For example, as an example of sample image update, when the image is judged to be clear from the luminance feature amount of the extracted target area, or when it has different color (spectrum) characteristics from the target acquired so far For example, an image that can be used as a reference for extraction processing can be acquired. As an example of rule update, by acquiring and accumulating feature quantities for each region, it is possible to construct a target extraction rule suitable for a region with a completely different environment. For example, roads in the AA area are made of asphalt, roads in the BB area are made of soil, and so on. As an example of model update, the weighting coefficient of an edge region having a particularly high degree of matching is increased depending on the degree of matching of edge feature amounts in processing using a model as a template. Of course, it goes without saying that the user may manually update the knowledge.

  In addition, the user assigns an evaluation level (A: optimal, B: appropriate, C: reference, D: inappropriate) to the processing result, and outputs this information together with the processing result in the subsequent processing. Therefore, it may be used as a judgment material. As a result, a filter effect can be obtained in which knowledge data (extraction process parameters, rules, etc.) is reviewed and updated when improperness occurs frequently, or when the result is optimum, the result is referred to preferentially next time. As for the update of the knowledge data 402, the knowledge database 151 itself may be directly updated, but update information may be held independently for each user. Specifically, a means for holding setting information is provided for each user, and difference information from the knowledge database generated by the update process is managed on the setting information. As a result, a knowledge database customized for each user can be used, and the processing contents of search or extraction can be individually customized. The user setting information may be managed on the knowledge database side (such as the management unit 131), but the knowledge database 151 is common to all users and may be managed by the user information management unit 111 of the processing management unit 110. good. With the configuration as described above, an image information processing system that meets user needs can be realized by extending rules and models through trial and error processes such as extraction processing experiments.

  FIG. 6 is a diagram illustrating an example of the target information index 150 according to an embodiment of the present invention. It is roughly divided into a table 600 for accumulating target information and an index for each item in the table. As the image and target data accumulation progresses, the table 600 becomes larger, so indexes are assigned to various items that are frequently used for retrieval. Thereby, processing can be efficiently executed in an appropriate order such as ascending order or descending order of various items. For example, if the index 601 is assigned to the target type, the search process focusing on the specific target can be speeded up. Further, if the index 602 is assigned to the image ID, it is possible to speed up the search process focusing on the situation and time series pattern in units of images. Further, if an index 603 relating to the position of the target is given, the search process focusing on a specific region or country can be speeded up. Specifically, a simple index such as ascending longitude may be used, or a spatial search index such as a quadtree may be constructed. In addition, special indexes that are frequently used, such as by country or region, may be constructed. Moreover, if the index 604 is assigned to the target time, the search process focusing on the time can be speeded up. Here, information such as the time at which the target is photographed can be obtained by following the image attribute information from the image ID. However, since it is frequently searched and used, as shown here, the target information is indexed in advance. It may be held in the form. In addition, any item may be indexed.

  Further, for the position and time information of the target, the table and index may be managed by a spatiotemporal geographic information system (4D-GIS). For example, by adopting the graphic data management method disclosed in Patent Document 4, it is possible to construct a map database that is easy to use for time change analysis while compressing the data capacity. Conversely, the entire index may be constructed by 4D-GIS, and IDs and other non-map elements may be implemented as attribute information of map graphic data. As a result, the spatio-temporal analysis engine can be configured to be included in the index, and the search processing performance can be improved by saving memory space when executing complex spatial analysis. In addition, for the user and other processing units, there is an advantage that complicated spatial analysis processing is concealed, and execution of processing and operation becomes simple.

  Here, the target object ID and the image ID may be described as URI (Uniform Resource Identifiers), etc., so that they can be distributed and managed in a plurality of computers and databases. As a result, it is possible to easily manage each item separately in separate computers and databases, and to perform distributed processing in a plurality of databases according to the processing request contents. Further, even when a part of the system configuration needs to be changed such as addition of a computer or database resource, it is only necessary to appropriately replace the URI corresponding to the changed part, and a highly scalable system can be provided. Of course, it goes without saying that a URI design that does not cause replacement may be used.

  FIG. 7 is a diagram illustrating an example of target extraction accuracy information according to an embodiment of the present invention. The extraction accuracy information 700 can describe the possibility of a plurality of targets based on the assumption that the target is not completely extracted and that there may be an error in recognition even if a person visually confirms it. It is. As a result, in the aircraft target, there is no doubt that it is the BB model of AA company, but the details of whether it is CC model number or DD model number is unknown. Can describe the situation. In addition, when the authorization work is performed manually and it is authorized that “this is definitely a CC model number”, the process may be skipped in the subsequent target extraction process. As a configuration, a mechanism that can manage a plurality of pieces of target information in association with each other, a mechanism that holds extraction accuracy information (see Patent Document 3) at the time of target extraction, and a mechanism that holds authorization information may be provided. Regarding the certification information, flag information indicating that the certification has been performed, information on the certified user, certification date, version information of extraction processing, and the like may be described.

  FIG. 8 is a diagram showing an example of a processing screen in one embodiment of the present invention. The processing screen 800 provides information related to processing to the user, and includes a plurality of sub windows. The processing operation window 801 provides a man-machine interface that receives user operations such as processing start and processing end. In addition, a man-machine interface that accepts user operations such as temporary suspension and resumption of processing may be provided for processing that takes a long time. The input of user operation is mainly executed by a pointing device such as a mouse, but of course, a keyboard or various input interfaces may be used. The processing content window 802 provides the user with information indicating the content of processing currently being executed or about to be executed, such as a search request (query) or an extraction processing command. In addition to providing processing contents, input of processing contents from the user, correction thereof, and the like may be accepted. The processing status window 803 provides information indicating the progress status of the processing to the user. This makes it possible to appropriately grasp the processing status even in the course of processing that takes a long time. The processing result window 804 provides information indicating the processing result to the user. In particular, in processing that takes a long time, processing results obtained in the course of processing may be sequentially added and output. A man-machine interface 805 such as a scroll bar may be provided to cope with the enlargement of the size of the processing result window due to the additional output of the processing result.

  With the configuration of the processing screen 800 that combines a plurality of windows as described above, various types of processing information can be browsed by the user in the course of processing, and flexible user operations corresponding to the processing information can be performed. For example, the processing result can be browsed while grasping the processing status such as the overall status. As a result, it is possible to select and execute a process with thought and error, such as temporarily stopping or stopping the process as necessary to correct the process content, restarting the process, or starting the process again from the beginning. For example, a query can be test-executed for a short period and in a narrow range, the obtained result can be checked immediately, and if the processing result is satisfactory, an additional job for the entire image database can be submitted. If the obtained result is unsatisfactory, the job can be canceled immediately, the model or rule can be finely adjusted, and the test can be performed again. As for the combination configuration of a plurality of windows, an arbitrary combination may be selected as necessary to organize the screen configuration, or may be provided only for the windows that the user thinks necessary depending on the usage scene. For example, only the processing status window is displayed for monitoring the system usage status and the processing operation window is not displayed, or only a specific window such as a processing result window is displayed.

  FIG. 9 is a diagram illustrating an example of query generation according to an embodiment of the present invention. If the query obtained by the following description is displayed in the processing content window 802, the processing content can be accurately grasped. The query may be a query 900 in the form of a character string such as the conventional SQL, but may also be a visual query 901 including a thumbnail image 902 of a search key. This can be easily realized by a description format such as XML or HTML, but has the effect that a complicated query can be clearly expressed, and an error in a search key image can be easily discovered. Furthermore, a modeling tool may be provided, and for example, a selection tool 903 that can easily set an advanced query simply by selecting a candidate from a list box may be prepared. This may be used not only for the thumbnail image 902 but also for a frequently used keyword 904 or the like. In particular, since the description of the pattern query 910 such as the situation (context) and the time series pattern is likely to be complicated, a situation selection tool 914, a time series pattern selection tool 911, and the like may be prepared. As the situation selection tool 914, a situation function selection tool 915 or the like may be prepared.

  For example, “Num (*)” is a function for acquiring the number of target objects *, “Area (*)” is a function for acquiring the area of the target object *, and “Within (*)” is a function * for which the target object * is another map. A function indicating whether or not the data is included in the data. Specifically, when the number of ships on the image is used for the query, it is defined and used to acquire the number of ship models, such as “Number of ships (*) :: = Num (Model: Ship)”. Also good. In the case of a time series pattern, in addition to the situation selection tool 914, a relation setting tool 912 between two images, a setting tool 913 for the time interval, and the like may be prepared. In particular, for the relationship setting tool 912, using the definition such as “Num (ship)” in the situation function selection tool 915, “number of ships (old) <number of ships (new)” (the number of ships increases) A query of a time series pattern such as “number of ships (old)> number of ships (new)” (decrease in number of ships) can be described by selection setting. Here, “old” indicates old image data, and “new” indicates new image data.

  Between them is an “operator”. For example, “<” is an increase in the target, “>” is a decrease in the target, “≈” is almost the same as the target, and “Max” is the target. “Min” represents a scene or time-series pattern in which the number of objects is maximum and “Min” is the minimum number of objects. A specific model stored in the knowledge database may be used as “<model>” or the like. For example, if a definitive model such as “Deforestation: == Significantly decreased vegetation index” is used, “Old <Forest Logging> New” is a condition when the vegetation index between two images is significantly decreased. Is added to the search results. Of course, the above example may be a more advanced query by combining a plurality of contents. In addition, in a query in which a plurality of images may be included in the result, such as a time series pattern search, the new or old image data to be output may be described in the query. Further, if necessary, the visual format query 901 may be converted into a character string format query 900 and used.

  FIG. 10 is a diagram illustrating an example of a priority order setting screen according to an embodiment of the present invention. Although the contents of the query processing are set in FIG. 9, the priority of query processing can be set by the following means. Since setting the priority order requires complicated spatiotemporal operations, it is important to provide a setting means that is easy for the user to understand, as shown here. The priority setting screen 1000 includes three screens: a detailed setting window 1001, a command operation window 1006, and a setting list window 1007.

  The detailed setting window 1001 efficiently realizes priority setting using time, space, or attribute information. The time setting window 1002 makes it possible to designate “latest data” in each area, which is a feature of the present invention. Thereby, it is possible to search only the latest situation in the region without complicated setting operations. Here, in order to efficiently realize this function, when inputting image data into the image database 152, it is determined whether or not the latest data of the target area is obtained, and flag information indicating the latest data is displayed. You may hold | maintain on the image database 152 previously. In addition, a means for specifying a specific start time or end time, a time interval, or the like, or specifying a processing setting such as “use only an image during the day” which is important for an outdoor image may be provided. Moreover, although the example of only the date is shown in the figure, it is of course possible to input the time. At that time, a conversion function between the world standard time and the local standard time may be provided. Further, the above plural functions may be used in combination.

  For example, if it is possible to set “latest data and images from the past year”, “There are areas where images have not been taken for the past several years, but those images are not subject to processing even if they are the latest.” Application is possible. The space setting window 1003 can graphically set the processing coping range by the user drawing 1004 on the map. In addition, by making a list of map data registered in a map database 1302, which will be described later, and making it selectable, a simple space setting is possible. For example, frequently used map data such as “own country, neighboring countries, XX region”, “world's major airports”, and “world's major ports” may be registered. As described above, the processing accuracy of search and extraction can be improved by integrating the information of the image system and the map system. The attribute setting window 1005 can set information regarding data to be preferentially processed based on the attribute information by a simple operation such as selection of a list. Needless to say, this is also effective for narrowing down the data to be processed.

  The command operation window 1006 adds the content selected and set in the detailed setting window 1001 to the list of the setting list window 1007 described later, conversely deletes it from the list, and terminates and cancels the setting process. Provide an interface. Here, in order to deal with complicated priority setting, a function to automatically check whether there is a contradiction in the setting contents is provided at the end of the setting process, and if a contradiction is detected, warning information may be presented to the user. good. A setting list window 1007 displays a list registered by the above settings and operations. Further, each item in the list may be moved by dragging the mouse to raise or lower the priority, or the contents may be corrected in conjunction with the command operation window 1006 or the detailed setting window 1001. Here, even for complicated priority settings such as a long list in which a plurality of items are registered, the processing history management unit 113 processes images that have already been processed so that the same image data is not duplicated by mistake. Processing log information may be retained, and processing may be appropriately managed such as providing a function of checking these before execution and skipping them if processing has been completed. These functions enable efficient processing even for a large-scale image database.

  FIG. 11 is a diagram showing an example of a processing status window according to an embodiment of the present invention. The processing status window 803 is provided to the user in a form that makes it easy to grasp processing status information such as processing progress status and remaining processing amount. For example, the example of the processed database range 1100 shown in FIG. 11A is an example in which the processed amount is clearly indicated by outputting the processed ratio with respect to the entire image database 152 or the target object information index 150. In addition, the example of the processing time 1101 shown in FIG. 11B is an example in which the expected processing time is clearly indicated so that the user can work efficiently even with a very long processing. In addition, the example of the character screen image 1102 shown in FIG. 11C is an example in which the entire processing status can be grasped by arranging and outputting the processed space-time range. It can also be applied when displaying the situation on a character terminal or the like. In addition, the example of the graphic screen image 1103 shown in FIG. 11D is an example in which the overview of the processing result is enhanced by the presentation of information that is graphical and easy to understand. With the configuration of the processing status window described above, information regarding the processed space range, time range, or database range can be appropriately grasped, and appropriate decision-making regarding subsequent processing and operations can be made.

  FIG. 12 is a diagram showing an example of a processing result window in one embodiment of the present invention. The processing result window 804 provides the user with processing result information such as the number and contents of processing results in an easy-to-understand form. The example of the processing result thumbnail output 1200 shown in FIG. 12A is an example in which the processing result including the thumbnail image 1201 is output, so that the processing status can be grasped particularly in units of image data. Of course, only the thumbnail image may be output, or information such as the target symbol 1202 and the range 1203 obtained may be output on the thumbnail image. Further, the related information 1204 may be output using attribute information of the image data. Further, the positional information may be acquired from the image attribute information, and the related information 1204 such as the region name may be output by superimposition analysis with the map data. In addition, the example of the processing result list output 1205 illustrated in FIG. 12B is an output example in which the processing results are output in a text format to improve the list of processing results. Information related to the image such as the image acquisition sensor, the shooting date and time, the shooting location, or information such as the number and status of the target may be output.

  Further, the example of the processing result map output 1206 shown in FIG. 12C is an example in which the processing result focusing on the regional situation can be grasped. The map can be scaled freely and displayed for any area. In particular, the window range may be automatically changed according to the currently registered space priority setting. As in the conventional query language, these results may be output together at the end after all the processing is completed, but the results may be sequentially output and browsed even during the processing. Thereby, since the number of processing results is unknown, a man-machine interface 805 for scrolling the processing results may be prepared, and the window size may be variable. Further, a selection interface 1207 may be provided for each processing result so that the processing result can be confirmed in detail even during the processing, and the detailed result may be displayed in another window or the like by the user's selection operation. Here, as soon as the individual processing results are obtained, the results are output in accordance with the processing priority specified at the time of processing execution, such as time priority and space priority, when the results are sequentially output. Alternatively, the processing results may be rearranged according to the order of suitability to the search query.

  FIG. 13 is a diagram illustrating an example of a database hardware configuration according to an embodiment of the present invention. As the image database 152 increases in size, it is assumed that the capacity of a single database will be insufficient, and it is necessary to have a basic system design that takes scalability into consideration. In the present invention, by using a plurality of databases such as the target information index 150 in cooperation with each other, the load on the image database main body can be distributed, which is desirable from the viewpoint of the system configuration. In general, a high-speed image database 1300 such as a disk system has a restriction that the capacity is relatively small, and conversely, a large-capacity image database 1301 such as a medium system often has a restriction that processing is slow. In order to realize a system by appropriately combining these hardware, frequently used image data is stored in the high-speed image database 1300, and other less frequently used data is stored in the large-capacity database 1301. is required.

  One of the features of this patent is that the target information index 150 is used as a front end for image database search, so that it can also be stored in a high-speed database to provide a high-throughput system. However, even if the search process can be performed using only the target information index 150, it is finally necessary to access the image database and browse the image data itself. Therefore, when the target information index 150 detects that the image data associated with the search result is stored in the large-capacity database 1301, a message 1310 is transmitted to the large-capacity image database 1301, and the high-speed database The image data may be transferred 1311 to 1300 in advance. However, since the disk capacity of the high-speed image database 1300 has an upper limit, if the database is expected to be saturated, a backup can be transferred 1312 to the large-capacity database 1301 side.

  By performing this automatically as the search process progresses, a high-performance image information processing system can be obtained in which image data already exists at a database location that can be accessed at high speed when the user uses it. be able to. The other map database 1302 and the like may have the same configuration as the image data. A plurality of databases located at remote locations via a network may be used. In that case, a database at a remote location on the network may be treated as a low-speed database.

  FIG. 14 is a diagram illustrating an example of a network service according to an embodiment of the present invention. If the system described so far is connected to the network 1400 and an input / output function is provided, a network service can be provided. For example, the query 1401 may be received by operation input from a network browser or the like, and the search processing result may be browsed and provided, or the reservation query 1401 is received and the output of the processing result is sent as a notification 1402 by e-mail or the like. Also good. As a result, it is possible to appropriately output a reservation query from a user who has not established a network connection. At that time, the user's usage status can be estimated based on the number and length of queries or the estimated processing time per image. The user usage status may be used as mere system statistical information, but based on the information, the usage limit of the computing resource may be set or the user may be charged. Finally, the main processing flow of the present invention is shown.

  FIG. 15 is a diagram illustrating an example of a cooperative processing flow of target information search processing and target information extraction processing according to an embodiment of the present invention. When a search query is input, first, a processing result is initialized (S110). S111 interprets the query content, and further inquires the knowledge processing unit to identify target information and its index S112. Thereafter, as long as there is search target data and there is no cancel instruction, the subroutine Sub1 is loop-processed S113. In the subroutine Sub1, the priority order list is traced, the current priority order setting to be processed is acquired, S120, and as long as there is data to be processed, the subroutine Sub2 is loop processed S121. It should be noted that priority order data may be listed in advance at the time of query interpretation S111 so that complicated interpretation of the contents of the priority order is not executed each time, and only the list may be traced here. In the subroutine Sub2, the processing history information and the target information index data are confirmed S130, and it is determined whether extraction processing is necessary S131.

  Only when the target information has not been extracted and the index has not yet been constructed, a target information extraction process described later with reference to FIG. 16 is executed, S132, and processing history information is stored S133. Then, the target information index is traced and data matching the input query condition is searched using a spatio-temporal analysis engine or the like, S134, the processing history is stored, S135, and the processing result is returned. With the above flow, it is possible to realize target information search processing in cooperation with target information extraction processing. As a result, even when image data without a target information index exists in the image database, it is possible to dynamically build an index and continue the processing, and to stabilize the behavior and search function of the entire system. It should be noted that an image in which a problem has occurred during the extraction process or the like may not be forcibly automatically processed, and data may be moved to a temporary standby queue or the end of the processing queue to make a user decision.

  FIG. 16 is a diagram showing an example of the target information extraction processing flow in one embodiment of the present invention. When the extraction query is input, first, the knowledge processing unit is inquired to acquire the target target extraction technique, S200, and the extraction process is executed using the obtained extraction technique S201. For example, the extraction method described in Patent Document 3 can be used. Further, the target information table is updated S202, and the target information index is updated S203. Further, when the setting for performing the certification work is registered, S204, and processing for assigning certification information is performed S205. In addition, about the provision of certification | authentication information, a user's setting condition can be managed in a process management part, and it can be made to work as needed. As a result, it is possible to automate the process so that the authorization is not normally required, or to perform the authorization work collectively when time is available. With the above flow, it is possible to realize target information extraction processing that can be linked with target information search processing.

  FIG. 17 is a diagram illustrating an example of a target information search processing flow according to an embodiment of the present invention. Here, the processing flow of the spatio-temporal analysis engine corresponding to both the situation (context) and time-series pattern search flows will be described together. When a search query is input, first, data to be searched is specified by referring to various indexes, and the image ID or target object ID is acquired (S300). In the case of time series pattern search processing, pair image data matching the set time interval condition is identified from the image data of the same region obtained by referring to the spatial index and the like, and the old and new two The image ID or the target ID belonging to the image ID is used as search target data. When a plurality of search target data is acquired, loop processing is performed as long as there is search target data S301. In the loop, it is determined whether or not the current search target data matches the query condition using a spatio-temporal analysis engine or the like, and if it matches, the image ID or the target ID is added to the processing result data to obtain S303. Returns the processing result data. With the above flow, the target information search process using the target information index can be realized.

  FIG. 18 is a diagram showing an example of an image data addition processing flow in one embodiment of the present invention. When new image data is input to the image database, the image attribute information is analyzed and the spatio-temporal position is specified (S400). The processing history management unit searches for processing history information at the corresponding spatio-temporal position in S401, and acquires a list of target information extraction processing that needs to be handled in S402. A loop process is performed until there is no unexecuted process on the list (S403), the target information extraction process described with reference to FIG. 16 is executed (S404), and the process history information is stored (S405). With the above flow, even in the case of an image database in which new image data is input one after another, the database status such as which target extraction processing (including processing version) was executed in which spatio-temporal range is automatically maintained. The target information index can be automatically updated. Further, the processing can be continued in the same flow even when the processing is canceled or resumed.

  As mentioned above, although the Example which mainly set the moving object as the target was described, this invention is not limited to this embodiment. It goes without saying that the use of the knowledge database as a core can provide the same effect with respect to various things and events captured and observed in images, such as various features and natural phenomena.

Moreover, although the Example mainly using a satellite image was demonstrated, this invention is not limited to this embodiment. It can be used in an image database that stores various types of images such as surveillance camera images and observation recording images that handle spatio-temporal information as well as satellite images, as well as images acquired from aircraft. Also, in the medical image field where image databases such as periodic diagnostic images are accumulated, satellite images described above can be substituted by country / region = patient, place / position = part, target = affected area or disease, etc. It is obvious that the embodiment of the present invention can be applied.
The means for realizing the present invention may be hardware having the above functions, a program, or a medium storing the program.

The example of a function structure in one Embodiment of this invention. The example of the query registration and search process result in one Embodiment of this invention. The example of the image of the search processing content in one Embodiment of this invention. The example of a structure of the knowledge database in one Embodiment of this invention. The example of the structure of the knowledge data in one Embodiment of this invention. The example of the target information index in one Embodiment of this invention. The example of the target extraction accuracy information in one Embodiment of this invention. The example of the processing screen in one Embodiment of this invention. An example of query generation in one embodiment of the present invention. The example of the priority order setting screen in one Embodiment of this invention. The example of the process status window in one Embodiment of this invention. The example of the process result window in one Embodiment of this invention. 2 illustrates an example of database hardware according to an embodiment of the present invention. 2 is an example of a network service according to an embodiment of the present invention. The example of the cooperation processing flow of the target information search process and target information extraction process in one Embodiment of this invention. The example of the target information extraction process flow in one Embodiment of this invention. The example of the target information search processing flow in one Embodiment of this invention. The example of the image data addition process flow in one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... User terminal, 101 ... Image information processing system, 102 ... Query, 103 ... Processing result, 110 ... Processing management part, 111 ... User information management part, 112 ... Data input / output management part, 113 ... Processing history management part, 120 DESCRIPTION OF SYMBOLS ... Search processing part 121 ... Management part 122 ... Query processing part 123 ... Search processing engine 130 ... Knowledge processing part 131 ... Management part 132 ... Query processing part 133 ... Knowledge processing engine 140 ... Extraction processing part , 141 ... management unit, 142 ... query processing unit, 143 ... extraction processing engine, 150 ... target information index, 151 ... knowledge database, 152 ... image database, 200 ... situation query processing result, 201 ... time series pattern query processing result 202 ... Reservation query processing result, 210 ... Target, 300 ... Database 301 ... image data 302 ... search result 303 ... time series pattern 304 ... time priority 305 ... latest priority 306 ... spatial priority 401 ... knowledge system management 402 ... knowledge data 403 ... extended knowledge data 501 ... Class information, 502 ... Attribute information, 503 ... Extraction means, 600 ... Target object information table, 601 ... Target object type index, 602 ... Image ID index, 603 ... Target object position index, 604 ... Target time index, 700 ... target extraction accuracy information, 800 ... processing screen, 801 ... processing operation window, 802 ... processing content window, 803 ... processing status window, 804 ... processing result window, 805 ... man-machine interface, 900 ... character string format query, 901 ... Visual format query, 902 ... Thumbnail 903 ... Selection tool, 904 ... Keyword, 910 ... Pattern query, 911 ... Time series pattern selection tool, 912 ... Relationship setting tool, 913 ... Time interval setting tool, 914 ... Status selection tool, 915 ... Status function selection tool , 1000 ... Priority order setting screen, 1001 ... Detailed setting window, 1002 ... Time setting window, 1003 ... Space setting window, 1004 ... User drawing, 1005 ... Attribute setting window, 1006 ... Command operation window, 1007 ... Setting list window, 1100 ... Processed database range, 1101 ... Processing time, 1102 ... Character screen image, 1103 ... Graphic screen image, 1200 ... Processing result thumbnail output, 1201 ... Thumbnail image, 1202 ... Target symbol, 1203 ... Target range 1204 ... Related information 1205 ... Processing result list output 1206 ... Processing result map output 1207 ... Selection interface 1300 ... High-speed image database 1301 ... Large-capacity image database 1302 ... Map database 1310 ... Message, 1311 ... Image data transfer, 1312 ... Backup transfer, 1400 ... Network, 1401 ... Query, 1402 ... Notification.

Claims (17)

An image information processing system that acquires an image and information on a target on the image from an image database, the image database managing the image and attribute information including positional information and time information of the image, and the target A target knowledge database for managing the name of the target and the target knowledge in association with each other, means for extracting information on the target on the image using the target knowledge database, the image and the target information. Means for associating and managing, means for constructing an index of the target information, means for registering a search query, means for associating the search query with the target information using the target knowledge database, Means for searching for the target information using the target information index, means for outputting a search result of the target information, and the image associated with the search result Image information processing system, characterized in that it comprises a means for outputting. The target information search means extracts the target information to be searched using the target knowledge database and the target information extraction means to construct the target information index, and uses the target information index. The image information processing system according to claim 1, further comprising means for retrieving the target object information. 3. The target object information search unit includes a spatial analysis unit, and the search query registration unit includes a unit that registers a query related to the status of the target on the image. The image information processing system described. The target object information search unit includes a spatiotemporal analysis unit, and the search query registration unit includes a unit that registers a query related to a time series pattern of a target that covers images at a plurality of shooting periods. The image information processing system according to any one of 1 to 3. 5. The image information processing system according to claim 1, wherein the search query registration means includes means for registering a reservation query. 6. The target object information extracting unit or the target object information searching unit includes a processing priority designation unit based on time priority, space priority, attribute information priority, or a combination thereof. The image information processing system described in 1. The image information processing system according to any one of claims 1 to 6, wherein the search result output means includes means for sequentially outputting the results of processing according to the priority order during processing. The image information processing system according to any one of claims 1 to 7, wherein the time priority designation means includes means for designating the latest image data in each region. 9. The image information processing system according to claim 1, wherein the space priority designation unit includes a unit that designates a region of interest using map data. 10. The target information extracting means or the target information searching means comprises means for outputting a processed range in the processing priority order, a processed ratio with respect to the entire database, or a combination thereof. An image information processing system according to any one of the above. The target information extraction means or the target information search means includes means for outputting processing history information, means for canceling processing, and means for continuing processing with reference to the processing history information. The image information processing system according to claim 1. The image database includes means for referring to the processing history information when image data is input, and means for executing the processing by the target object information extracting means when corresponding processing history information exists. An image information processing system according to any one of claims 1 to 11. The image information processing system according to any one of claims 1 to 12, wherein the target knowledge database includes means for aggregating knowledge data of a plurality of targets to generate new knowledge data. The target knowledge database includes means for managing setting information related to target information search processing or target information extraction processing for each user, and feedback information from the user for the result of target information search processing or target information extraction processing. The image information processing system according to claim 1, further comprising a receiving unit and a unit that updates the user setting information using the feedback information. The target object information includes means for managing a plurality of target information in association with each other, and means for managing extraction accuracy information and authorization information in association with each target information. 14. The image information processing system according to any one of 14. The database includes high-speed hardware and low-speed hardware, and the target information index is stored in high-speed hardware and used for the search process, and an image associated with the search result is low-speed hardware. 16. The image information processing system according to claim 1, further comprising means for transferring the image from the low-speed hardware to the high-speed hardware. 17. The image information processing system according to claim 1, wherein said search query registration means and said search result output means comprise means for inputting and outputting via a network. Information processing service.

Images