WO2019176398A1 - Information processing device, information processing method, and program - Google Patents

Abstract

[Problem] To provide: an information processing device capable of further enhancing the convenience of use of a system in which a plurality of feature spaces are handled; an information processing method; and a program. [Solution] This information processing device is provided with a control unit which performs: a control for storing a registration object, included in registration request information, in association with unique first identification information that is common in a plurality of feature extracting units; a control for converting the registration object according to a definition of the modality of the registration object and generating conversion data for registration; and a control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality.

Description

Information processing apparatus, information processing method, and program

The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Conventionally, in the field of search technology and machine learning technology, various data such as text, images, tabular data, and time-series data are widely replaced with N-dimensional vectors that well represent the characteristics of the data (feature extraction). Has been done.

As an example of vectorization, in the field of natural language processing, a technique called BoW (Bag of Words) that represents a sentence using a vector in which the number of vocabulary N is the number of dimensions and the value of only the word that appears is generally used. It has been broken. In the field of image processing, in addition to a technique such as BoVW (Bag of Visual Words) that considers local features such as local binary patterns (LBP) as codewords, data is input and feature vectors are output. Many deep learning models have been devised. The tabular data is converted into one vector by the process of converting the category into a 1-hot number-dimensional vector and the process of normalizing each integer value or real value.

A space spanned by such vectors is called a feature space and is used as a feature of search technology and machine learning technology. The demand for feature spaces will increase further in the future, and there will be demands for handling multiple feature spaces and using them across and switching.

There is also a cross-sectional feature space that handles different modalities (input / output formats) in a unified manner. For example, the following Non-Patent Document 1 discloses a technique for mapping text and images to a semantic space (a technique for mapping multimodal to a single space).

Here, when performing feature extraction, it is important to process the data (preprocessing) until it is put into the feature extractor, and it is necessary to follow a predetermined process, but these differ for each feature extractor, When a plurality of feature spaces are handled, the same data must be preprocessed by different processes for each feature extractor, which is redundant.

Therefore, the present disclosure proposes an information processing apparatus, an information processing method, and a program that can further improve the convenience of a system that handles a plurality of feature spaces.

According to the present disclosure, control for storing a registration object included in registration request information in a storage unit in association with unique first identification information common to a plurality of feature extraction units, and the registration object modality of the registration object And converting the first identification information and the registration conversion data to a plurality of feature extractors corresponding to the modality. An information processing apparatus including a control unit is proposed.

According to the present disclosure, the processor stores the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units, and the registration object Control for converting according to the definition of the modality of the object and generating conversion data for registration; and control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality An information processing method including performing the above is proposed.

According to the present disclosure, the computer controls the registration object included in the registration request information to be stored in the storage unit in association with the unique first identification information common to the plurality of feature extraction units, and the registration object is registered in the registration unit. Control for converting according to the definition of the modality of the object and generating conversion data for registration; and control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality A program for functioning as a control unit for performing the above is proposed.

As described above, according to the present disclosure, it is possible to further improve the convenience of a system that handles a plurality of feature spaces.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is a figure explaining an outline of an information processing system by one embodiment of this indication. It is a figure which shows an example of the processing content in the main functional block of the information processing system by this embodiment. It is a figure which shows an example of the other system configuration example of the information processing system by this embodiment. It is a sequence diagram which shows an example of the flow of the registration process of the object in the information processing system by this embodiment. It is a sequence diagram which shows an example of the flow of the search process in the information processing system by this embodiment. It is a figure which shows an example of the search screen in this embodiment. It is a figure explaining modularization of the feature extractor in the 1st application example by this embodiment. It is a sequence diagram which shows an example of the registration process of the 1st application example by this embodiment. It is a sequence diagram which shows an example of the search process of the 1st application example by this embodiment. It is a figure which shows an example of the search screen in the 2nd application example by this embodiment. It is a figure which shows the other example of the search screen in the 2nd application example by this embodiment. It is a sequence diagram which shows an example of the search process of the 2nd application example by this embodiment. It is a functional block diagram which shows an example of a structure of the suggestion system of the 3rd application example by this embodiment. It is a sequence diagram which shows an example of the flow of the search process in the suggestion system of the 3rd application example by this embodiment. It is a figure which shows an example of the operation information and request information which are acquired from the application in the 3rd application example by this embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. 1. Overview of information processing system according to an embodiment of the present disclosure Configuration 2-1. Configuration of information processing apparatus 10 2-2. 2. Configuration of the feature management server 20 Operation processing 3-1. Registration process 3-2. Search processing Application example 4-1. First application example: definition of modality inclusion 4-2. Second application example: merge of search results 4-3. Third application example: Suggest system 5. Summary

<< 1. Overview of Information Processing System According to One Embodiment of Present Disclosure >>
FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. As illustrated in FIG. 1, the information processing system according to the present embodiment includes an information processing apparatus 10 and a feature management server 20.

The feature management server 20 replaces various data (hereinafter referred to as “objects”) such as text, images, tabular data, and time-series data with N-dimensional vectors that well represent the features of the data (feature extraction). A feature extraction unit 202 (an example of a feature extractor) that performs processing), and manages a space (feature space) spanned by such vectors. In the present specification, the feature extraction unit 202 and the feature space have a one-to-one relationship. A plurality of feature management servers 20 may exist as shown in FIG. 1, each having one feature extraction unit 202 (that is, each feature management server 20 has one feature space. It can be said that it is managing.

Here, the handling of a general feature space will be described below. In the feature space, the degree of similarity and relationship between objects can be expressed, and for example, search and recommendation by concept can be performed. In the feature space, the accuracy of other machine learning techniques can be improved, and for example, it can be used as a feature in a recognition system, data analysis, or the like. In the feature space, different modalities can be handled in a unified manner. For example, text, handwriting, and notes can be handled in one feature space. In this specification, the “modality” is a data input / output format, and for example, various modalities are assumed as described below.
-Text-Word, Sentence, HTML (HyperText Markup Language), etc.-Media-RGB image, depth image, vector image, video, audio, etc.-Compound document-Office document, PDF, Web page, e-mail, etc.-Metadata-User , Date, etc.-sensor data-current position, acceleration, heart rate, etc.-application data-startup log, file information being processed, etc. The feature space can be defined and expanded freely.

It is also possible to handle multiple feature spaces across. For example, there is a first feature space that can process text, notes, and Web pages from a viewpoint of meaning, and a second feature space that can process handwriting and an image from a viewpoint of shape These may be mapped through a predetermined function. This makes it possible to quickly generate a system that, for example, searches for images and handwriting by text, searches for notes related to the web page that you are currently viewing, and automatically assigns tags that are close to the note you are currently writing. It becomes possible.

Also, third-party extension (extension by plug-in) is assumed as a method of handling multiple feature spaces across. More specifically, it is possible for another person to associate another modality with the same feature space, or to configure another feature space from the same modality. It is also possible to expand by doing both of these (for example, by associating the modalities of “paper” and “case” and entering sensor data for each case, it is possible to search for cases close to the paper or case. it can). It can also be used in combination with the feature space handled by others (for example, searching for products from text or images).

In this way, it becomes possible to search across all data in multiple multimodal spaces.

(background)
Here, the following problems can be considered regarding the construction of a system for searching across such a plurality of feature spaces.

First, when performing feature extraction, it is important to process (pre-process) data until it is put into the feature extractor, and it is necessary to follow a predetermined process. For example, taking an image as an example of the pre-process,
-Accept only JPEG or PNG data-Unify to RGB 3 channel, 256x256px-Extend the short side when the aspect ratio is not 1: 1-Apply processing such as smoothing.

As an example of text preprocessing,
Cleaning process (removes noise in the text. For example, in the case of Web text, an HTML tag or a JavaScript (registered trademark) source code)
-Word division of sentences, word normalization, stop word removal, etc.

However, since these differ for each feature extractor, when handling a plurality of feature spaces, the same data must be pre-processed by different processes for each feature extractor, which is redundant.

Therefore, in the present embodiment, it is possible to further improve the convenience of a system that handles a plurality of feature spaces by sharing preprocessing in a plurality of feature extractors.

Specifically, a rule for conversion to a predetermined data format is defined for each modality, and converted data (hereinafter also referred to as “entity”) converted according to the definition of the object modality in the information processing apparatus 10 corresponds to the modality. To one or more feature management servers 20 (server devices having a feature extraction unit 202 which is an example of a feature extractor).

Also, by defining modality definitions for some data that cannot be handled as general programming types, it is possible to read difficult data and unify data preprocessing.

For example, demand with the following data is assumed.
(Pattern 1: Conversion / extraction between formats)
・ Rendering (drawing) vector data (normal text file) to treat it like other images ・ Extracting text to handle PDF, office documents, etc. ・ Schedule name from iCalendar (standard schedule format) And schedule, extract only participants (Pattern 2: Unification of format)
Each company has proposed a different format for handwritten data, and there is no standard format. Extract common terms of such data and convert them into a processable format.
(Pattern 3: Reading special data)
Data that is not yet handled digitally, such as blood pressure and heart rate, is often described as general data such as text. They are read and converted to a data format that is easy to process (eg, an integer string).
(Pattern 4: Data acquisition)
・ Acquire images from URLs ・ Use ID as input and extract data from specific external database

In addition, when a plurality of feature spaces are handled and registered in a search database (hereinafter referred to as a search DB), if data is stored for each feature space, the same data is registered in a plurality of databases, which is redundant. Therefore, it is conceivable to prepare a database capable of storing data and extracting data from the ID separately from the search DB, and storing only the ID in the search DB. At this time, if the user arbitrarily inputs the ID together with the data, there is a possibility that the following problem may occur and the user must take care.
-It is possible to associate multiple IDs with the same data-It is possible to associate the same ID with multiple data-Different searches when storing data in only some search DBs There is no guarantee that the IDs stored in the DB point to the same data (search results cannot be compared or summarized)
-As a result, there is no guarantee that the original data can be extracted from the ID.

Therefore, in the present embodiment, in the information processing apparatus 10, along with the common pre-processing, a unique (unique) ID (identification information) common to a plurality of feature extractors is given to the conversion data of the registered object. By saving the registered object, it is possible to solve the above problem and further improve the convenience of a system that handles a plurality of feature spaces.

As described above, in the information processing system according to the present embodiment, a plurality of feature spaces are created by uniformly converting objects for each modality and managing objects with unique IDs common to a plurality of feature spaces. It is possible to further improve the convenience of the handling system.

The configurations of the information processing apparatus 10 and the feature management server 20 included in the information processing system according to this embodiment will be described below.

<< 2. Configuration >>
<2-1. Configuration of Information Processing Device 10>
As illustrated in FIG. 1, the information processing apparatus 10 according to the present embodiment includes a control unit 100, a communication unit 120, an output unit 130, and a storage unit 140. The information processing apparatus 10 is a local terminal such as a smartphone, a tablet terminal, or a PC used by a user, for example.

(Control unit 100)
The control unit 100 functions as an arithmetic processing unit and a control unit, and controls overall operations in the information processing apparatus 10 according to various programs. The control unit 100 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example. The control unit 100 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 100 according to the present embodiment also functions as a feature space management unit 101 (Feature Space Manager) and a modality management unit 102 (Modality Manager).

The feature space management unit 101 performs feature space management (acquisition of one or more feature management server 20 IDs), object registration processing, search processing, and the like. Here, FIG. 2 shows an example of processing contents in the main functional blocks (the feature space management unit 101, the modality management unit 102, and the feature management unit 201) of the information processing system according to the present embodiment. As shown in FIG. 2, for example, the feature space management unit 101 according to the present embodiment can perform the following processing.
・ Get Manager (space ID: string): Feature Manager
・ Register Manager (manager: Feature Manager)
・ Get Vector (space ID: string, obj: object, modality: string): vector
・ Register Object (obj: object, modality: string, space ID: string = ANY)
・ Search (query: object, query Modality: string, target Modality: string = ANY): Search Result

More specifically, for example, when the registration request information is input, the feature space management unit 101 outputs the object and modality included in the registration request information to the modality management unit 102, and the modality management unit 102 follows the definition of the modality. The converted conversion data (entity) and ID are acquired, and the entity and ID are output to the feature management server 20. At this time, the feature space management unit 101 can output to one or more feature management servers 20 corresponding to the modality of the object. The “feature management server 20 corresponding to the modality” is the feature management server 20 capable of handling the modality. The feature space management unit 101 can grasp what information each feature space handles from the ID (space ID: string, etc.) of the feature management server 20. Therefore, for example, in the case of modality: “Text”, the feature space management unit 101 can refer to each space ID and identify the feature management server 20 that manages the feature space that handles “Text”. . Alternatively, the feature space management unit 101 may output the information to all the feature management servers 20 (in this case, the feature management server 20 may determine whether or not processing can be appropriately performed).

Further, when the search request information is input, the feature space management unit 101 outputs the object and modality included in the search request to the modality management unit 102, and the converted data (converted by the modality management unit 102 according to the definition of the modality ( entity), and outputs the entity to the feature management server 20. At this time, the feature space management unit 101 can output to one or more feature management servers 20 corresponding to the modality of the object. The feature space management unit 101 may output the feature request to the feature management server 20 corresponding to the designated space ID when the search request includes a space ID as a search condition. Alternatively, the feature space management unit 101 may output the information to all the feature management servers 20 (in this case, the feature management server 20 may determine whether or not processing can be appropriately performed).

Then, the feature space management unit 101 extracts a corresponding object (that is, original data) from the storage unit 140 based on one or more IDs searched in the feature management server 20, and outputs them as search results to the search request source. . Note that the search condition may further include filter information as an additional condition. Examples of the filter information include designation of a search DB (corresponding to a feature space to be searched), designation of the number of searches, and the like. For example, when the number of searches is specified, the feature space management unit 101 sets the similarity of each search result (similarity indicating the similarity between each search result and the feature amount of the conversion data (entity) of the search object). Based on this, a predetermined upper number of search results may be output to the search request source.

The modality management unit 102 manages modalities. For example, as shown in FIG. 2, the modality management unit 102 can perform the following processing.
・ Get Modalities (): string []
・ Register Modality (modality: string, definer: Modality Definer)
Create (obj: object, modality: string): entity

More specifically, for example, the modality management unit 102 registers the modality definition (data), or the modality definition unit 103 inputs the object (registered object) input from the feature space management unit 101 to the modality of the object. The data is converted into a predetermined data format according to the definition, and converted data (entity) is generated. The entity is shaped data that can be directly passed to the feature extraction unit 202. In addition, the modality definition unit 103 assigns a unique ID common to the plurality of feature spaces to the generated entity, and outputs the entity and ID to the feature space management unit 101. Further, the modality definition unit 103 stores the object (registered object) and the ID assigned to the entity of the registered object in the storage unit 140 in association with each other. Such an ID is a unique character string across a plurality of feature spaces that handle the same modality. A hash value or the like may be used so that the same value is returned for the same entity.

Modality definition data includes definition data in a format that can be received as input data (obj) (for example, file name (string), OpenCV Mat format) (multiple) and output data (entity) format definition data (Only one) (for example, char [3] [256] [256]). Modality definition data exists, for example, for each data format or for each pattern described above (for example, conversion between formats, unification of formats, reading of special data, etc.), and is stored in the storage unit 140. The modality management unit 102 converts the registration object (input data) using the definition data of the modality, and outputs the conversion data (entity). Also, the modality definition unit 103 saves the data corresponding to the ID as necessary or confirms that the data is saved, the function of retrieving the data corresponding to the ID, and deletes the data corresponding to the ID. Has the function of The modality definition unit 103 exists for each modality.

(Input unit 110)
The input unit 110 has a function of receiving user instruction content such as an operation input unit that receives an operation instruction from the user and a voice input unit that receives a voice instruction from the user, and outputs the instruction content to the control unit 100. The operation input unit may be a touch sensor, a pressure sensor, or a proximity sensor. Alternatively, the input unit 110 may be a physical configuration such as a button, a switch, and a lever.

(Communication unit 120)
The communication unit 120 is connected to an external device by wire or wireless, and transmits / receives data to / from the external device. For example, the communication unit 120 may be a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE: Long Term Evolution, 3G (third generation mobile communication)). For example, it is possible to connect to a network (not shown) and transmit / receive data to / from the feature management server 20 via the network.

(Output unit 130)
The output unit 130 has a function of presenting (outputting) information to the user, such as a display unit and an audio output unit. For example, the output unit 130 outputs a search screen or outputs a search result under the control of the control unit 100.

(Storage unit 140)
The storage unit 140 is realized by a ROM (Read Only Memory) that stores programs used in the processing of the control unit 100, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

For example, the storage unit 140 stores feature space management information, modality definition information, and an object (substance data) assigned with a unique ID.

The configuration of the information processing apparatus 10 according to the present embodiment has been specifically described above. The configuration of the information processing apparatus 10 is not limited to the example illustrated in FIG. For example, each process by the control unit 100 of the information processing apparatus 10 may be executed by a plurality of apparatuses or may be executed by a server on the network.

<2-2. Configuration of Feature Management Server 20>
As illustrated in FIG. 2, the feature management server 20 includes a control unit 200, a communication unit 210, and a feature amount database 220. In this embodiment, since the feature management server 20 has one feature extraction unit 202, it can be said that each feature management server 20 manages one feature space, but the present disclosure is limited to this. For example, if the feature management server 20 includes a plurality of feature extraction units 202, a plurality of feature spaces can be managed.

(Control unit 200)
The control unit 200 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the feature management server 20 according to various programs. The control unit 200 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example. The control unit 200 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

In addition, the control unit 200 according to the present embodiment also functions as the feature management unit 201. The feature management unit 201 performs feature extraction (for example, substitution processing for an N-dimensional vector) on the entity transmitted from the information processing apparatus 10 by using the feature extraction unit 202, and sets the extracted feature amount as the ID of the entity. A process of registering in the feature amount database 220 in association with each other is performed. An existing technique can be used for the feature quantity extraction algorithm, and is not particularly limited here. In addition, the feature extraction unit 202 can handle different modalities in a unified manner. The feature quantity extracted by the feature extraction unit 202 is registered in the feature quantity database 220. Here, the feature quantity database 220 exists for each modality. The feature management unit 201 registers the feature amount extracted by the feature extraction unit 202 in the feature amount database 220 corresponding to the modality of the original data (entity transmitted from the information processing apparatus 10) in association with the ID. For example, it is assumed that there is a feature extraction unit 202a (color feature feature space a) that can extract feature quantities from different modalities (for example, handwriting (Strokes) and image (Image)) from the viewpoint of color features. In this case, the feature amount extracted by the feature extraction unit 202a is stored in the feature amount database 220-1 corresponding to the handwritten data when extracted from the handwritten data, and the feature corresponding to the image data when extracted from the image data. It is stored in the quantity database 220-2.

The feature management unit 201 can also perform a search process (similarity search) using a feature space in response to a request from the feature space management unit 101. Since the feature quantity database 220 exists for each modality, the feature management unit 201 may perform a similarity search using the feature quantity database 220 corresponding to the target modality (search target modality). For example, the feature management unit 201 can perform the following processing as shown in FIG.
・ Get Space ID (): string
・ Register Database (modality: string, database: Feature Database)
・ Get Vector (entity: object, modality: string): vector
Add (id: string, entity: object, modality: string)
・ Most Similar (query: object, modality: string, target Modality: string): Search Result []

(Communication unit 210)
The communication unit 210 is connected to an external device by wire or wireless, and transmits / receives data to / from the external device. For example, the communication unit 210 may be a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE: Long Term Evolution, 3G (third generation mobile communication). By connecting to a network (not shown) and transmitting / receiving data to / from the information processing apparatus 10 via the network.

(Feature database 220)
The feature quantity database 220 accumulates the feature quantities extracted by the feature extraction unit 202. Each feature amount is associated with a unique ID assigned by the modality management unit 102. As described above, the feature amount database 220 exists for each modality.

The feature amount database 220 is stored in a storage unit (not shown) included in the feature management server 20. The storage unit of the feature management server 20 is realized by a ROM that stores programs and calculation parameters used for the processing of the control unit 200, and a RAM that temporarily stores parameters that change as appropriate.

The configuration of the feature management server 20 according to the present embodiment has been specifically described above. The configuration of the feature management server 20 shown in FIG. 1 is an example, and the present embodiment is not limited to this. For example, at least a part of the configuration of the feature management server 20 may be in an external device.

Here, FIG. 3 shows an example of another configuration example of the information processing system according to this embodiment. As shown in FIG. 3, for example, the feature extraction unit 240 and the feature quantity database 250 may be managed by separate servers (the feature management server 24 and the database server 25), respectively.

<< 3. Action processing >>
Next, the operation processing of the information processing system according to the present embodiment will be specifically described with reference to the drawings.

<3-1. Registration process>
FIG. 4 is a sequence diagram illustrating an example of a flow of object registration processing in the information processing system according to the present embodiment.

As illustrated in FIG. 4, first, the feature space management unit 101 of the information processing apparatus 10 acquires a registration request based on a user operation input or the like (step S103), and the object (obj) included in the registration request Along with the information on the modality (mdl) of the object, a request for generation (conversion data (entity)) is made to the modality management unit 102 (step S106).

Next, the modality management unit 102 performs conversion data (entity) generation, unique ID assignment, and obj and unique ID storage processing by the modality definition unit 103 (step S109). Specifically, the modality definition unit 103 performs processing (common preprocessing) for converting an object into data of a predetermined format in accordance with the definition of the modality. Specific examples of the processing include the following examples.
In the case of a still image: JPEG data is converted into char [3] [256] [256] (multidimensional array) and smoothing processing is performed.
・ For voice: Read mp3 data as short type arbitrary length array.
For text: Remove HTML tags and convert all uppercase letters to lowercase (no format).
・ Handwriting: Read the point sequence data and draw it with a white line of thickness 3 on a black image of char [3] [256] [256].

Next, the feature space management unit 101 acquires at least an ID and an entity from the modality management unit 102 (step S112). Further, the modality management unit 102 may notify that the ID and obj are saved.

Next, the feature space management unit 101 outputs a data addition (registration) request to all corresponding feature spaces (Feature Space) based on the acquired ID and entity (step S115). The addition request includes ID, entity, and modality (mdl). The corresponding feature space is a feature space (feature management server 20) that can handle the modality of the entity. When there are a plurality of feature spaces that can handle the modality of the entity, the processes shown in steps S115 to S121 are repeated for each feature space.

Next, the feature management unit 201 extracts feature amounts using the feature extraction unit 202 (step S118).

Then, the feature management unit 201 adds (registers) the extracted feature quantity to the feature quantity database 220 together with the acquired unique ID (step S121). At this time, the feature management unit 201 registers the feature amount database 220 corresponding to the modality of the extraction source entity.

The registration process according to this embodiment has been specifically described above. In this way, before inputting data to each feature space, the modality management unit 102 performs a predetermined conversion process for each modality so that the same data is preprocessed by a different process for each feature extractor. Therefore, the convenience of a system that handles a plurality of feature spaces can be improved. Further, managing the modality management unit 102 and the feature extraction unit 202 individually reduces the responsibility of each function (for example, it becomes easier to identify the cause at the time of an error).

Also, for example, if a general database that can store IDs and feature vectors is provided, the search system is completed by developing only the feature extractor, and the availability of the system increases. In the search DB (that is, the feature database 220), only the ID and the feature are registered, and the original data (data entity) is separately managed by the modality definition unit 103. Therefore, the same data is registered in a plurality of databases. Can be avoided. In addition, by assigning unique IDs across multiple feature spaces to the same data, multiple IDs can be associated with the same data, the same ID can be associated with multiple data, etc. Can be avoided.

In addition, the feature management unit 201 may register the feature data in the feature amount database 220 only when the original data is stored in the modality management unit 102. This ensures that the feature space management unit 101 can extract the original data from the ID when acquiring a search result to be described later.

In addition, since the preprocessing to a predetermined data format is managed separately, the feature space (search DB system) development side can develop the feature extraction unit 202 without worrying about the input format. .

Further, the registration process according to the present embodiment is not limited to the example shown in FIG. For example, in step S115 described above, it has been described that the addition instruction is given to the feature space (feature management server 20) corresponding to the modality. However, the present embodiment is not limited to this, and the feature space management unit 101 can execute all the features. An additional instruction may be given to the space (feature management server 20). In this case, the feature space (feature management server 20) can determine whether the entity is a processable entity based on the modality.

<3-2. Search process>
Next, search processing in the information processing system according to the present embodiment that constructs a feature space as described above will be described with reference to FIG. FIG. 5 is a sequence diagram showing an example of the flow of search processing in the information processing system according to the present embodiment.

As shown in FIG. 5, first, the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on a user operation input or the like (step S133). The search request includes an object (obj), a modality (mdl1) of the object, and a target modality (mdl2) indicating the modality to be searched.

Next, the feature space management unit 101 makes a generation request (conversion data (entity)) to the modality management unit 102 together with information on the object (obj) and the modality (mdl1) of the object included in the search request. (Step S136).

Next, the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity) and assign a unique ID (step S139). Specifically, the modality definition unit 103 performs processing (common preprocessing) for converting an object into data of a predetermined format in accordance with the definition of the modality.

Next, the feature space management unit 101 acquires the ID and entity from the modality management unit 102 (step S142).

Next, the feature space management unit 101 outputs a search request to all the corresponding feature spaces (Feature Space) based on the acquired entity (step S145). The search request includes entity, mdl1 (original data modality), and mdl2 (target modality). The corresponding feature space is a feature space (feature management server 20) that can handle mdl1 and mdl2. When there are a plurality of feature spaces that can handle mdl1 and mdl2, the processing shown in steps S145 to S157 is repeated for each feature space.

Next, the feature management unit 201 uses the feature extraction unit 202 to extract feature amounts (step S148).

Subsequently, the feature management unit 201 searches for a similar feature amount from the feature amount database 220 based on the extracted feature amount (step S151). At this time, the feature management unit 201 searches the feature amount database 220 corresponding to the requested target modality (mdl2). In the feature quantity database 220, the unique ID is associated with the feature quantity, and the feature management unit 201 searches the feature quantity database 220 for a feature quantity similar to the feature quantity of the requested entity, and is similar. The ID associated with the feature quantity and the similarity of the feature quantity: sim (similarity with the feature quantity of the requested entity, for example, the distance of the N-dimensional vector) is acquired. If there are a plurality of target modalities (mdl2), the processes shown in steps S151 to S154 are repeated for each feature quantity database 220.

Next, the feature space management unit 101 acquires a search result (searched feature value ID, searched modality: mdl, and searched feature value similarity: sim) from the feature management unit 201 (step S157). . The search result may include a plurality of IDs, mdl, and sim. When a singular number is obtained as a search result, the feature space management unit 101 specifies, for example, an ID of a feature amount having the highest similarity. When the predetermined number is obtained as the search result, the feature space management unit 101 specifies the ID of the upper predetermined number of feature amounts based on, for example, the similarity.

Next, the feature space management unit 101 makes a request for original data to the modality management unit 102 together with the identified ID and the corresponding modality information (step S160).

Next, the modality management unit 102 acquires the original data (that is, the object) associated with the ID by the modality definition unit 103 (step S163) and outputs it to the feature space management unit 101 (step S166). The processing shown in steps S160 to S166 can be performed for the number of search results to be output. When the original data cannot be acquired, the feature space management unit 101 deletes the record (ID, mdl, sim).

Then, the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S169). For example, the feature space management unit 101 may display a screen showing the search result on the output unit 130 and present it to the user.

The search processing according to this embodiment has been specifically described above. Thus, the feature space constructed according to the present embodiment can be used for cross-sectional search in a plurality of feature spaces that handle different modalities.

As shown in the last example above, the feature space (search DB) used for the search may be specified. The identified feature space is included in the search request in step S113 as a space ID. For example, “I want to use a search DB created by XX company”, “I want to use a XX search site”, or the like is assumed.

Here, FIG. 6 shows an example of a search screen in the present embodiment. The search screen 30 shown in FIG. 6 is presented by the output unit 130, for example. The user inputs a search object 301 and selects a search target (corresponding to a modality. For example, “photograph”, “illustration”, “document”, etc.), and the feature quantity of what is similar is searched for. Select (for example, “shape”, “color”, “meaning”, etc., and correspond to each feature space. For example, feature space constructed based on shape features, constructed based on color features When the search button 302 is selected, an object similar to the search object 301 acquired as a search result is presented. For example, when “illustration” is selected as the search target and “shape” (space ID1), “color” (space ID2), and “meaning” (space ID3) are selected as the feature quantities, the search object 301 is obtained as the search result. And an illustration having a similar shape, color, and / or meaning (for example, a “modality: illustration” feature quantity database possessed by the feature management server 20 that handles a feature space constructed based on the feature of the shape. 220) and presented. The feature amount search condition and / or may be arbitrarily selected by the user, or or may be set as a default.

<< 4. Application example >>
Subsequently, an application example of the information processing system according to the present embodiment will be described.

<4-1. First Application Example: Definition of Modality Inclusion Relationship>
First, as a first application example, the definition of the modality inclusion relationship will be described. The modality definition unit 103 according to the present embodiment may define a parent-child relationship (inclusion relationship) between modalities. Specific examples include the following.

-Modality "RGB image" has modality "grayscale image" as child-Modality "mail" has modality "text", "user", "date" as children

According to this application example, when a new modality is defined, if a child modality is defined, it can be easily incorporated into an existing feature space. In addition, it is possible to perform feature extraction for a new modality by combining a plurality of feature extractors (feature space, feature extraction unit 202).

For example, use cases that define modalities including existing modalities are assumed. More specifically, it is assumed that a modality “text” already exists and there is a feature space A (feature extraction unit 202A) that handles “text”. Here, it is assumed that a modality of “mail” including text (body) and a user (sender) and a feature space B (feature extraction unit 202B) that can handle “mail” are added. In this case, the first effect is that “can be registered in a plurality of feature spaces at the same time”. That is, since the text can be acquired from the mail, the same ID and object pair can be registered not only in the feature space B but also in the feature space A. That is, the feature space when the focus is on only the text is stored in the feature space A, and the feature space when the focus is on the text and the user is stored in the feature space B. In addition, a search can be performed across other texts (in this case, it is necessary to assign a unique ID across all modalities, not just the same modality).

Also, as a second effect, “a new feature extractor (feature extractor 202) can be easily constructed by reusing an existing feature extractor (feature extractor 202)” is mentioned. That is, since the feature space B can extract the features of the text using the feature space A, only the feature extraction of the user needs to be performed in the feature space B, and the implementation becomes easy. It is also possible to modularize existing feature extractors. FIG. 7 is a diagram illustrating modularization of the feature extractor.

As shown in the left of FIG. 7, for example, when extracting mail features, each modality using each feature extractor that can handle modalities such as texts and inclusive relations defined with mail and modalities, respectively. As shown in the right part of FIG. 7, it is possible to acquire a mail feature amount including a sentence feature amount (contents) and a user feature amount (sender).

Hereinafter, registration processing and search processing in this application example will be sequentially described with reference to FIG. 8 and FIG.

(Registration processing taking into account the definition of modality inclusion)
FIG. 8 is a sequence diagram illustrating an example of registration processing of the first application example according to the present embodiment.

As shown in FIG. 8, first, when the feature space management unit 101 of the information processing apparatus 10 acquires a registration request based on a user operation input or the like (step S203), the object (obj) included in the registration request and the object Along with information on the modality (mdl) of the object (for example, “Mail”), a request for generation of (entity) is made to the modality management unit 102 (step S206).

Next, the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity), assign a unique ID, and store obj and a unique ID, and have an inclusion relationship with the modality of obj. Based on the definition of modality (sub mdl), sub entity is generated (step S209). For example, when the modality of the object is “Mail” and the modality (sub mdl) having an inclusion relation with this is “Text”, the modality definition unit 103 includes the mail data (obj) according to the definition of “Text”. Converts text data into a predetermined data format and generates a sub entity.

Next, the feature space management unit 101 acquires at least ID, entity, and sub entity from the modality management unit 102 (step S212). Further, the modality management unit 102 may notify that the ID and obj are saved.

Next, the feature space management unit 101 outputs a data addition (registration) request to all corresponding feature spaces (eg, feature space B) based on the acquired ID and entity (eg, Mail Entity) ( Step S215). The subsequent processing related to feature amount extraction shown in steps S218 to S221 is the same as that in steps S118 to S121 shown in FIG. 4 and will not be described in detail. For example, in the feature space B handling mail, The feature quantity (Mail Vector) is registered. At this time, for the text (Text Vector) among the feature quantities of the mail, the Get Vector (Step S227) of the feature space A handling the text described below is used. It may be.

The feature space management unit 101 outputs a data addition (registration) request for all corresponding feature spaces (eg, feature space A) for the same ID and sub entity (eg, Text Entity) (steps S224 to 230). . The feature space A is a feature space corresponding to text only, and the feature amount (Text Vector) of the text extracted from the Text Entity is registered.

As described above, in this modification, in extracting feature quantities, a feature space having an inclusion relationship can be used, and a feature quantity can be registered in the feature space.

(Search processing taking into account the definition of modality inclusion)
Next, a search process according to this modification will be described with reference to FIG. FIG. 9 is a sequence diagram illustrating an example of search processing of the first application example according to the present embodiment.

As shown in FIG. 9, first, the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on a user operation input or the like (step S243). The search request includes an object (obj), a modality (mdl1) of the object, and a target modality (mdl2) indicating the modality to be searched. Here, for example, mdl1 = Mail and mdl2 = Text.

Next, the feature space management unit 101 requests the modality management unit 102 to generate (entity) together with information on the object (obj) and the modality (mdl1) of the object included in the search request (step S246). .

Next, the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity) and assign a unique ID, and based on the definition of the modality (sub mdl) having an inclusion relation with the modality of obj. Then, the sub entity is generated (step S249). For example, when the modality of the object is “Mail” and the modality (sub mdl) having an inclusion relation with this is “Text”, the modality definition unit 103 includes the mail data (obj) according to the definition of “Text”. Converts text data into a predetermined data format and generates a sub entity.

Next, the feature space management unit 101 acquires ID, entity, and sub entity from the modality management unit 102 (step S252).

Next, the feature space management unit 101 outputs a search request to all corresponding feature spaces (Feature Space) based on the acquired entity (for example, Mail Entity) (step S255). The search request includes entity, mdl1 (original data modality, for example, Mail), and mdl2 (target modality, for example, Text). The corresponding feature space is a feature space that can handle mdl1 and mdl2 (for example, a feature space corresponding to both mail and text).

The subsequent processes related to feature amount extraction shown in steps S258 to S267 are the same as steps S148 to S157 shown in FIG. Here, it is assumed that there is no corresponding feature space. For example, if there is a feature space B that handles e-mails and a feature space A that handles texts, both are not feature spaces that handle both e-mail and text, so no search results will be returned. When is used, a search result can be returned from the feature space A.

The feature space management unit 101 outputs a search request to all corresponding feature spaces (Feature Space) based on the ID and sub entity (for example, Text Entity) (step S270). The search request includes sub entity (eg, Text Entity), mdl1 (sub mdl, eg, Text), and mdl2 (target modality, eg, Text). The corresponding feature space is a feature space that can handle mdl1 and mdl2, and here mdl1 and mdl2 are the same “Text”, and therefore feature space A corresponding to text corresponds. A search is performed in the feature space A (steps S273 to S279), and the feature space management unit 101 acquires a search result from the feature management unit 201 (step S282).

Subsequently, the feature space management unit 101 sends a request for original data to the modality management unit 102 together with the acquired ID and the corresponding modality (for example, Text), as in steps S160 to S169 shown in FIG. In step S285, the object acquired based on the ID by the modality definition unit 103 (step S288) is output from the modality management unit 102 to the feature space management unit 101 (step S291).

Then, the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S294).

The search processing in consideration of the modality inclusion relation according to this application example has been specifically described above.

<4-2. Second Application Example: Merging Search Results>
Next, search result merging will be described as a second application example. The feature space management unit 101 can output the final search result to the search request source after re-evaluating the search result based on the similarity and weighting of the search result from each feature extractor. . The weighting is, for example, weighting of the feature space. Such weighting can be arbitrarily set by a search request source (for example, a user).

FIG. 10 is a diagram illustrating an example of a search screen in this application example. As shown in FIG. 10, the search screen 32 includes a search object 321, a search target selection region 322, a region 323 for selecting the feature amount of what is similar to search, and a search button 326. It is displayed. In the area 323 for selecting the feature amount, the weight of the selected feature amount can be set by operating the slide bar 324. For example, when it is desired to prioritize “color feature” among “shape feature” and “color feature”, the operation unit 325 of the slide bar 324 is moved toward “color feature”. Thereby, for example, the weight (w) is set as follows on the system side. Here, the color feature space: space1 and the shape feature space: space2.
w (weights) ＝ {space1: 0.8, space2: 0.2}

In this case, as shown in FIG. 10, search results giving priority to “color features” (search results giving priority to illustrations with similar colors) are displayed.

Note that the setting of the weighting of the feature space is not limited to the example illustrated in FIG. 10. For example, the weighting may be set based on what is selected by the user from the search results, and the search results may be presented again. An example is shown in FIG. For example, when the illustration 341 is selected from the search results presented on the search screen 34 in FIG. 11, the system outputs the illustration 341 as the search result, assuming that the illustration 341 is close to the user's intention. The weighting may be set so as to prioritize the feature space (feature extractor, that is, the feature extraction unit 202), and the search result may be presented again.

(Operation processing)
Next, operation processing of this application example will be described with reference to FIG. FIG. 12 is a sequence diagram illustrating an example of search processing of the second application example.

As shown in FIG. 12, first, the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on a user operation input or the like (step S303). The search request includes an object (obj), a modality (mdl1) of the object, a target modality (mdl2) indicating a modality to be searched, and a weight (w) of the feature space.

In subsequent steps S315 to S321, a search process similar to the process shown in steps S145 to 157 of FIG. 5 described above is performed, and thus detailed description thereof is omitted here. In step S318, processing similar to that shown in steps S148 to S154 in FIG. 5 is performed, but detailed illustration is omitted.

Next, the feature space management unit 101 ranks (re-evaluates) the search results according to the similarity and weighting of the search results (step S324). Specifically, for example, the feature space management unit 101 multiplies the similarity of the search result by the weight of the feature space (feature extractor, that is, the feature extraction unit 202) that outputs the search result, and creates a new similarity. Can be re-evaluated. Table 1 below shows an example of re-evaluation. Here, w (weights) = {space1: 0.8, space2: 0.2}.

As shown in Table 1 above, for example, the object A as a search result is weighted to the first feature space (sim (space1): 0.9) with the similarity (sim (space1): 0.9) when searched from the first feature space (space1). space1: 0.8) multiplied by the second feature space weight (space2: 0.2) multiplied by the similarity (sim (space2): 0.3) retrieved from the second feature space (space2) A value obtained by adding the values (sim (new): 0.78) is calculated as a new similarity. This is because an ID associated with the same data may be registered in a plurality of feature spaces. In addition, it is assumed that the search result is searched only from one feature space. In this case, as in the example of the object C in Table 1 above, for example, the second feature space has a similarity (sim (space2): 0.9) when the object C is searched from the second feature space (space2). A value (sim (new): 0.18) obtained by multiplying the weight (space2: 0.2) is calculated as a new similarity.

The feature space management unit 101 specifies, for example, a predetermined number of search results (IDs) on the basis of the new similarity.

Next, the feature space management unit 101 makes a request for original data to the modality management unit 102 together with the identified ID and the corresponding modality, similarly to steps S160 to S169 shown in FIG. 5 described above (step S327). The object acquired based on the ID by the modality definition unit 103 (step S330) is output from the modality management unit 102 to the feature space management unit 101 (step S333).

Then, the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S336).

So far, the search result merging according to this application example has been specifically described.

<4-3. Third application example: Suggest system>
Next, a suggestion system will be described with reference to FIGS. 13 to 15 as a third application example. The suggest system can be used on a system that runs multiple applications to search for content that suits the situation based on user operation information (browsing content, operating content, etc.) in each application. It is possible to make a proposal to the user.

Suppose, for example, that the user is planning a travel plan using various applications. When a user searches for a sightseeing spot with a web browser, searches for a local map with a map application, and further summarizes a plan in a notebook application, the suggestion system will meet the demand according to the usage status of these multiple applications. Content (Web page, text, image, etc.) can be proposed.

(Configuration example)
FIG. 13 is a functional block diagram showing an example of the configuration of the present system. As shown in FIG. 13, for example, a suggestion system can be realized by an information processing apparatus 10x. The information processing apparatus 10x functions as one or more applications 105, an information collection unit 106, a suggestion unit 107, a feature space management unit 101x, and a modality management unit 102x. These can be implemented by the control unit 100 of the information processing apparatus 10.

The application 105 is various application programs such as a web browser, a map application, and a notebook application.

The information collection unit 106 has a function of monitoring the operation of each application 105 and collecting and storing user operation information (that is, application usage status) in each application 105. The information collecting unit 106 may use an OS (Operating System).

The suggestion unit 107 generates a search request based on the operation information collected by the information collection unit 106, and makes a search request to the feature space management unit 101x. For example, the suggestion unit 107, based on the request of the modality (mdl1) and content (obj) of the content being browsed / edited acquired from each application 105 from the information collection unit 106, and the modality (mdl2) of the required content, A search request may be generated. For example, the following examples of content modality acquired from each application 105 and required content modality requests are assumed.
-Web browser ... Browsing: Web page, Request: Web page / Map application ... Browsing: Address, Request: None * Note app ... Editing: Text / Image, Request: Text / Image

The feature space management unit 101x performs a search process using one or more feature spaces in response to a request from the suggestion unit 107. The search process is the same as in the above-described embodiment. First, the feature space management unit 101x acquires the entity obtained by converting the obj by the modality management unit 102x, and the entity, mdl1 (for example, Web page, address, text), and mdl2 A search request is made to the feature management server 20 based on (for example, Web page, image). Then, the feature space management unit 101x outputs the search result to the suggestion unit 107.

The modality management unit 102x has the same function as that of the modality management unit 102 described with reference to FIG. 1, and the modality definition unit 103 performs processing for converting obj into a predetermined data format according to the definition of the modality of mdl1. The generated entity is output to the feature space management unit 101x.

The example of the configuration of the information processing apparatus 10x that executes the suggestion system according to this application example has been specifically described above.

(Operation processing)
Next, an operation process of the suggest system according to this application example will be described with reference to FIG. FIG. 14 is a sequence diagram showing an example of the flow of search processing in the suggestion system of this application example.

As shown in FIG. 14, first, when one or more applications 105 are operated by the user (step S403), the content being handled (post; obj, mdl1) and the required content request (request) Mdl2) is performed on the information collecting unit 106 (step S406). As an example of the post, for example, a web page of “Kinkakuji”, an address “Kita-ku, Kyoto ... 1-2-3”, a travel-related text, and the like can be cited. Further, examples of the request include a web page and an image.

Next, the information collection unit 106 outputs the collected information (post, request) to the suggestion unit 107 (step S412).

Next, the suggestion unit 107 makes a search request to the feature space management unit 101x (step S415). The search request includes the content included in post as obj, the modality thereof as mdl1, and the content modality included in request as mdl2.

Next, a search process is executed in the feature space management unit 101x (step S418). In step S418, processing similar to that in steps S136 to S166 of FIG. 5 (generation of entity from obj and mdl1, search based on entity, mdl1, and mdl2, acquisition of object from search result ID) is performed. Detailed description is omitted.

Next, the suggestion unit 107 acquires a search result from the feature space management unit 101x (step S421).

Next, the suggestion unit 107 may rank (re-evaluate) the search results according to the similarity and weighting (W) of the search results (step S424). For example, the suggestion unit 107 may set a weight as shown in Table 2 below for each input and output, and rank by multiplying the similarity. In this application example, such reevaluation may be skipped.

The suggestion unit 107 creates a display screen showing the search result (step S427) and presents it to the user (step S430).

Further, the suggestion unit 107 may update the weighting (W) used in the above step 424 when the usage status feedback is obtained from the user (step S433).

Note that the suggestion unit 107 may suggest to the user and obtain feedback from the user via the application 105.

Here, FIG. 15 shows an example of operation information and request information acquired from an application according to this application example. In this system, the operation information as shown on the left in FIG. 15 and the request information as shown on the right in FIG. 15 are acquired from each application, and the requested information is suggested based on the operation information.

<< 5. Summary >>
As described above, the information processing system according to the embodiment of the present disclosure can further improve the convenience of a system that handles a plurality of feature spaces.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the present technology is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

For example, a computer program for causing the information processing apparatus 10 or the feature management server 20 to perform the functions of the information processing apparatus 10 or the feature management server 20 on hardware such as a CPU, ROM, and RAM incorporated in the information processing apparatus 10 or the feature management server 20 described above. Can be created. A computer-readable storage medium storing the computer program is also provided.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

In addition, this technique can also take the following structures.
(1)
Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units;
Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration;
Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
An information processing apparatus comprising a control unit that performs the following.
(2)
The information processing apparatus according to (1), wherein the definition of the modality is a rule for conversion to a predetermined data format corresponding to the modality.
(3)
The controller is
A control for converting the search object included in the search request according to the definition of the modality of the search object, and generating conversion data for search;
Control to output the search conversion data to the feature extractor corresponding to the modality of the search object and the target modality included in the search request;
The information processing apparatus according to (1) or (2), wherein:
(4)
The controller is
Obtaining second identification information searched based on the search conversion data in the one or more feature extractors;
The information processing apparatus according to (3), wherein a corresponding object is acquired from the storage unit based on the second identification information, and is output as a search result.
(5)
The controller is
The degree of similarity indicating the degree of similarity of the feature is acquired from the feature extractor together with the second identification information associated with the feature similar to the feature extracted from the search conversion data, (4) The information processing apparatus described in 1.
(6)
The information processing apparatus according to (4) or (5), wherein the search request further includes filter information as a search condition.
(7)
The controller is
When the registration request information is input, according to the definition of the sub-modality having a parent-child relationship with the modality of the registration object, control for converting the registration object to generate sub-conversion data for registration;
Control for outputting the first identification information and the sub-transformed data to one or more feature extractors corresponding to the sub-modalities;
The information processing apparatus according to any one of (1) to (6), further performing:
(8)
The controller is
When the search request is input, sub-converted data for search is generated by converting data corresponding to the sub-modality among the search objects according to the definition of the sub-modality having a parent-child relationship with the modality of the search object. Control,
Control to output the sub-transform data for search to one or more feature extractors corresponding to the sub-modality and the target modality;
The information processing apparatus according to any one of (3) to (6), further performing:
(9)
The controller is
Control for ranking the plurality of second identification information based on the second identification information and similarity obtained from the feature extractor based on the search request, and the weight of the feature extractor;
A control for outputting the upper predetermined number of the second identification information as the search results;
The information processing apparatus according to any one of (4) to (6), further performing:
(10)
The controller is
Generating the search request for searching for content to be proposed to the user based on information including user operation information output from one or more applications;
The information processing apparatus according to any one of (4) to (6), wherein one or more pieces of the second identification information acquired from the feature extractor are output as the search result.
(11)
The controller is
The content included in the information and handled by the application is the search object,
The modality of the content is the modality of the search object,
The information processing apparatus according to (10), wherein the search request is generated using the modality of the content requested by the application as the target modality.
(12)
The information processing apparatus includes:
The information according to any one of (1) to (11), further including a communication unit that transmits the first identification information and the conversion data for registration to a feature management server having the feature extractor. Processing equipment.
(13)
Processor
Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units;
Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration;
Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
An information processing method including performing.
(14)
Computer
Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units;
Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration;
Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
A program for functioning as a control unit for performing

10, 10x information processing device 20, 24 feature management server 25 database server 100 control unit 101, 101x feature space management unit 102, 102x modality management unit 103 modality definition unit 105 application 106 information collection unit 107 suggestion unit 110 input unit 120 communication unit DESCRIPTION OF SYMBOLS 130 Output part 140 Storage part 200 Control part 201 Feature management part 202 Feature extraction part 210 Communication part 220 Feature quantity database 240 Feature extraction part 250 Feature quantity database

Claims (14)

1. Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units;
   Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration;
   Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
   An information processing apparatus comprising a control unit that performs the following.
2. The information processing apparatus according to claim 1, wherein the definition of the modality is a rule for conversion to a predetermined data format corresponding to the modality.
3. The controller is
   A control for converting the search object included in the search request according to the definition of the modality of the search object, and generating conversion data for search;
   Control to output the search conversion data to the feature extractor corresponding to the modality of the search object and the target modality included in the search request;
   The information processing apparatus according to claim 1, wherein:
4. The controller is
   Obtaining second identification information searched based on the search conversion data in the one or more feature extractors;
   The information processing apparatus according to claim 3, wherein a corresponding object is acquired from the storage unit based on the second identification information, and is output as a search result.
5. The controller is
   The degree of similarity indicating the degree of similarity of the feature is acquired from the feature extractor together with the second identification information associated with the feature similar to the feature extracted from the search conversion data. The information processing apparatus described.
6. The information processing apparatus according to claim 4, wherein the search request further includes filter information as a search condition.
7. The controller is
   When the registration request information is input, according to the definition of the sub-modality having a parent-child relationship with the modality of the registration object, control for converting the registration object to generate sub-conversion data for registration;
   Control for outputting the first identification information and the sub-transformed data to one or more feature extractors corresponding to the sub-modalities;
   The information processing apparatus according to claim 1, further performing:
8. The controller is
   When the search request is input, sub-converted data for search is generated by converting data corresponding to the sub-modality among the search objects according to the definition of the sub-modality having a parent-child relationship with the modality of the search object. Control,
   Control to output the sub-transform data for search to one or more feature extractors corresponding to the sub-modality and the target modality;
   The information processing apparatus according to claim 3, further performing:
9. The controller is
   Control for ranking the plurality of second identification information based on the second identification information and similarity obtained from the feature extractor based on the search request, and the weight of the feature extractor;
   A control for outputting the upper predetermined number of the second identification information as the search results;
   The information processing apparatus according to claim 4, further performing:
10. The controller is
    Generating the search request for searching for content to be proposed to the user based on information including user operation information output from one or more applications;
    The information processing apparatus according to claim 4, wherein one or more pieces of the second identification information acquired from the feature extractor are output as the search result.
11. The controller is
    The content included in the information and handled by the application is the search object,
    The modality of the content is the modality of the search object,
    The information processing apparatus according to claim 10, wherein the search request is generated with the content modality requested by the application as the target modality.
12. The information processing apparatus includes:
    The information processing apparatus according to claim 1, further comprising a communication unit that transmits the first identification information and the conversion data for registration to a feature management server having the feature extractor.
13. Processor
    Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units;
    Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration;
    Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
    An information processing method including performing.
14. Computer
    Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units;
    Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration;
    Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
    A program for functioning as a control unit for performing

Images