JP7255585B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

Conventionally, in the fields of search technology and machine learning technology, replacing various data such as text, images, tabular data, and time-series data with N-dimensional vectors that well represent the characteristics of the data (feature extraction) is widely used. It is done.

As an example of vectorization, in the realm of natural language processing, a method called BoW (Bag of Words) is commonly used in which the number of words N is the number of dimensions, and a vector having a value only for words that appear is used to represent sentences. It is In addition, in the area of image processing, in addition to techniques such as BoVW (Bag of Visual Words), in which local features such as local binary patterns (LBP) are regarded as codewords, data is input and feature vectors are output. Many deep learning models have also been devised. Tabular data is converted into one vector by a process of converting a category into a 1-hot number-dimensional vector and a process of normalizing each integer value and real value.

A space spanned by such vectors is called a feature space, and is used as features in search technology and machine learning technology. In the future, the demand for feature spaces will increase further, and it is expected that there will be demand for handling multiple feature spaces, cross-sectionally using them, and switching between them.

There is also a transversal feature space that handles different modalities (input/output formats) in a unified manner. For example, Non-Patent Document 1 below discloses a technique of mapping text and images to a semantic space (a technique of mapping multimodal to a single space).

"A New Approach to Cross-Modal Multimedia Retrieval" N. Rasiwasia, J. Costa Pereira, E. Coviello, G. Doyle, G. Lanckriet, R.Levy and N. Vasconcelos. Proceedings of the 18th ACM international conference on Multimedia, Pages 251-260, Florence, Italy - Oct. 2010.

Here, when performing feature extraction, it is important to process (preprocess) the data before entering it into the feature extractor, and it is necessary to go through a fixed process. When dealing with a plurality of feature spaces, the same data must be preprocessed by different processes for each feature extractor, which is redundant.

Therefore, the present disclosure proposes an information processing device, an information processing method, and a program that can further improve the convenience of a system that handles a plurality of feature spaces.

According to the present disclosure, control for storing a registered object included in registration request information in a storage unit in association with unique first identification information common to a plurality of feature extraction units, and modality of the registered object for the registered object and control to generate conversion data for registration, and control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality. An information processing apparatus including a control unit is proposed.

According to the present disclosure, a processor controls to store a registered object included in registration request information in a storage unit in association with unique first identification information common to a plurality of feature extraction units; Control for converting according to the definition of the modality of the object to generate conversion data for registration, and control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modalities. We propose a method for processing information, including:

According to the present disclosure, a computer controls to store a registered object included in registration request information in a storage unit in association with unique first identification information common to a plurality of feature extraction units; Control for converting according to the definition of the modality of the object to generate conversion data for registration, and control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modalities. We propose a program for functioning as a control unit that performs

As described above, according to the present disclosure, it is possible to further improve the convenience of a system that handles multiple feature spaces.

In addition, the above effects are not necessarily limited, and in addition to the above effects or instead of the above effects, any of the effects shown in this specification, or other effects that can be grasped from this specification may be played.

1 is a diagram describing an overview of an information processing system according to an embodiment of the present disclosure; FIG. It is a figure which shows an example of the processing content in the main functional blocks of the information processing system by this embodiment. It is a figure which shows an example of the other system configuration example of the information processing system by this embodiment. FIG. 7 is a sequence diagram showing an example of the flow of object registration processing in the information processing system according to the present embodiment; FIG. 4 is a sequence diagram showing an example of the flow of search processing in the information processing system according to the embodiment; It is a figure which shows an example of the search screen in this embodiment. It is a figure explaining modularization of the feature extractor in the 1st application example by this embodiment. FIG. 12 is a sequence diagram showing an example of registration processing of the first application example according to the present embodiment; FIG. 10 is a sequence diagram showing an example of search processing of the first application example according to the present embodiment; FIG. 11 is a diagram showing an example of a search screen in a second application example according to the embodiment; FIG. 10 is a diagram showing another example of a search screen in the second application example according to the embodiment; FIG. 12 is a sequence diagram showing an example of search processing in a second application example according to the present embodiment; It is a functional block diagram which shows an example of a structure of the suggestion system of the 3rd application example by this embodiment. FIG. 12 is a sequence diagram showing an example of the flow of search processing in the suggestion system of the third application example according to the present embodiment; FIG. 14 is a diagram showing an example of operation information and request information acquired from an application in a third application example according to the embodiment;

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

Also, the description shall be given in the following order.
1. Overview of an information processing system according to an embodiment of the present disclosure2. Configuration 2-1. Configuration of information processing apparatus 2-2. Configuration of feature management server 20 3 . Operation processing 3-1. Registration process 3-2. Search processing 4 . Application example 4-1. First Application Example: Definition of Inclusive Relationship of Modalities 4-2. Second Application Example: Merging Search Results 4-3. Third Application Example: Suggestion System5. summary

<<1. Outline of information processing system according to an embodiment of the present disclosure>>
FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the information processing system according to this embodiment has a configuration including an information processing device 10 and a feature management server 20 .

The feature management server 20 replaces various data such as texts, images, tabular data, and time-series data (hereinafter referred to as "objects") with N-dimensional vectors that well represent the features of the data (feature extraction). processing), and manages a space (feature space) spanned by such vectors. In this specification, there is a one-to-one relationship between the feature extraction unit 202 and the feature space. A plurality of feature management servers 20 may exist as shown in FIG. 1, each having one feature extraction unit 202. said to be in control).

Here, the handling of a general feature space will be described below. In the feature space, it is possible to express similarities and relationships between objects, and for example, to search and recommend concepts. In addition, in the feature space, the accuracy of other machine learning techniques can be improved, and for example, it can be used as features in recognition systems, data analysis, and the like. Also, in the feature space, different modalities can be handled in a unified manner, and for example, text, handwriting, and notes can be handled in one feature space. In this specification, the term "modality" refers to a data input/output format, and includes a wide variety of modalities, such as those listed below.
・Text - words, sentences, HTML (HyperText Markup Language), etc. ・Media - RGB images, depth images, vector images, videos, audio, etc. ・Compound documents - office documents, PDFs, web pages, emails, etc. ・Metadata - users , date, etc. Sensor data - current position, acceleration, heart rate, etc. Application data - startup log, file information being processed, etc. A feature space can be defined for these modalities, and can be expanded freely.

Also, it is possible to cross-handle multiple feature spaces. For example, if there is a first feature space that can process text, notes, and web pages from a semantic perspective, and a second feature space that can process handwriting and images from a shape perspective. , these may be mapped through a predetermined function. As a result, for example, it is possible to quickly create a system that searches for images and handwriting in text, searches for notes related to the web page you are currently viewing, and automatically adds tags similar to the notes you are currently writing. It becomes possible.

Further, extension by a third party (extension by plug-in) is also envisioned as a method of cross-cutting multiple feature spaces. More specifically, others can associate different modalities with the same feature space, or construct different feature spaces from the same modality. It is also possible to expand by doing both (for example, by associating the modalities of "papers" and "judicial precedents" and inserting sensor data for each case, it is possible to search for cases that are similar to the papers and judicial precedents. can). It can also be used in combination with feature spaces handled by others (for example, searching for products from text or images).

In this way, it is possible to traverse any data in multiple multimodal spaces.

(background)
Here, the following problems are conceivable with regard to the construction of such a system for cross-searching a plurality of feature spaces.

First, when performing feature extraction, it is important to process (pre-process) data before entering it into the feature extractor, and it is necessary to go through a fixed process. As a fixed process of preprocessing, for example, taking an image,
・Only JPEG or PNG data is accepted. ・Uniform to RGB 3 channels, 256x256px. ・If the aspect ratio is not 1:1, the short side is stretched.

Also, as an example of text preprocessing,
・Cleaning processing (remove noise in the text. For example, in the case of web text, HTML tags, JavaScript (registered trademark) source code, etc.)
・Sentence word segmentation, word normalization, stop word removal, etc.

However, since these are different for each feature extractor, when handling a plurality of feature spaces, the same data must be preprocessed by different processes for each feature extractor, which is redundant.

Therefore, in the present embodiment, by sharing preprocessing in a plurality of feature extractors, it is possible to further improve the convenience of a system that handles a plurality of feature spaces.

Specifically, rules for conversion into a predetermined data format are defined for each modality, and conversion data (hereinafter also referred to as "entity") converted according to the definition of the modality of the object in the information processing apparatus 10 is converted to correspond to the modality. output to one or more feature management servers 20 (a server device having a feature extraction unit 202, which is an example of a feature extractor).

In addition, by prescribing modality definitions for some data that cannot be handled as general programming types, it is possible to read data that is difficult to process, and to unify data preprocessing.

For example, the demand for the following data is assumed.
(Pattern 1: conversion/extraction between formats)
・Rendering (drawing) vector data (usually text files) to handle it like other images ・Extracting text from PDFs, office documents, etc. to handle them as text ・Name of appointment from iCalendar (standard format for schedules) , dates, and participants only (Pattern 2: Format unification)
Each company has proposed different formats for handwritten data, and there is no standard format. It extracts the common denominator of such data and transforms it into a processable format.
(Pattern 3: Reading special data)
Data such as blood pressure and heart rate, which are not handled digitally yet, are often described as general data such as text. Read them and convert them into a data format that is easier to process (eg, an integer string).
(Pattern 4: Acquisition of data)
・Acquire an image from a URL ・With an ID as an input, extract data from a specific external database

Also, when handling a plurality of feature spaces and registering them in a search database (hereinafter referred to as a search DB), if data is saved for each feature space, the same data will be registered in a plurality of databases, which is redundant. Therefore, it is conceivable to store data separately from the search DB and prepare a database from which data can be retrieved from IDs, and to store only IDs in the search DB. At this time, if the user arbitrarily inputs the ID together with the data, the following problems may arise, and the user must take care of them.
・It is possible to associate multiple IDs with the same data. ・It is possible to associate the same ID with multiple data. There is no guarantee that the IDs stored in the DB point to the same data (search results cannot be compared or summarized)
・As a result, there is no guarantee that the original data can be extracted from the ID.

Therefore, in the present embodiment, in the information processing apparatus 10, in addition to standardization of the preprocessing, a unique ID (identification information) common to a plurality of feature extractors is assigned to the transformation data of the registered object. , the above problem can be solved by saving registered objects, and the convenience of a system that handles a plurality of feature spaces can be further improved.

As described above, in the information processing system according to the present embodiment, a plurality of feature spaces are transformed by uniformly transforming objects for each modality and managing objects with unique IDs common to a plurality of feature spaces. It is possible to further improve the convenience of the system to be handled.

The configurations of the information processing apparatus 10 and the feature management server 20 included in the information processing system according to this embodiment will be described below.

<<2. Configuration>>
<2-1. Configuration of Information Processing Device 10>
As shown in FIG. 1, the information processing apparatus 10 according to this embodiment has a control unit 100, a communication unit 120, an output unit 130, and a storage unit 140. FIG. The information processing device 10 is, for example, a local terminal such as a smart phone, a tablet terminal, or a PC used by a user.

(control unit 100)
The control unit 100 functions as an arithmetic processing device and a control device, and controls general operations within the information processing device 10 according to various programs. The control unit 100 is realized by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like. The control unit 100 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 100 according to this embodiment also functions as a feature space manager 101 and a modality manager 102 .

The feature space management unit 101 manages the feature space to be handled (acquisition of IDs of one or more feature management servers 20, etc.), object registration processing, search processing, and the like. Here, FIG. 2 shows an example of processing contents in the main functional blocks (feature space management unit 101, modality management unit 102, and feature management unit 201) of the information processing system according to this embodiment. As shown in FIG. 2, for example, the feature space management unit 101 according to this embodiment can perform the following processing.
・Get Manager (space ID: string): Feature Manager
・Register Manager (manager: Feature Manager)
・GetVector(space ID: string, obj: object, modality: string): vector
・Register Object (obj: object, modality: string, space ID: string=ANY)
・Search(query: object, query Modality: string, target Modality: string=ANY): Search Result

More specifically, for example, when the registration request information is input, the feature space management unit 101 outputs the object and modality included in the registration request information to the modality management unit 102, and the modality management unit 102 modality modality definition. The converted data (entity) and the ID are acquired, and the entity and the ID are output to the feature management server 20 . At this time, the feature space management unit 101 can output to one or more feature management servers 20 corresponding to the modality of the object. The “feature management server 20 corresponding to the modality” is the feature management server 20 capable of handling the modality. The feature space management unit 101 can grasp what kind of information each feature space handles from the ID (space ID: string, etc.) of the feature management server 20 . Therefore, for example, in the case of modality: "Text", the feature space management unit 101 can refer to each space ID and specify the feature management server 20 that manages the feature space that handles "Text". . Alternatively, the feature space management unit 101 may output to all the feature management servers 20 (in this case, the feature management server 20 side can appropriately determine whether or not processing is possible).

Further, when the search request information is input, the feature space management unit 101 outputs the object and modality included in the search request to the modality management unit 102, and the converted data ( entity) and outputs the entity to the feature management server 20 . At this time, the feature space management unit 101 can output to one or more feature management servers 20 corresponding to the modality of the object. Also, when a space ID is included as a search condition in a search request, the feature space management unit 101 may output to the feature management server 20 corresponding to the designated space ID. Alternatively, the feature space management unit 101 may output to all the feature management servers 20 (in this case, the feature management server 20 side can appropriately determine whether or not processing is possible).

Based on one or more IDs retrieved by the feature management server 20, the feature space management unit 101 retrieves the corresponding object (that is, the original data) from the storage unit 140, and outputs it as a search result to the search requester. . Note that the search condition may further include filter information as an additional condition. Examples of filter information include specification of a search DB (corresponding to a feature space to be searched) and specification of the number of searches. For example, when the number of searches is specified, the feature space management unit 101 calculates the similarity of each search result (similarity indicating the similarity of each search result to the feature amount of the transformation data (entity) of the search object). Based on this, a predetermined number of top search results may be output to the search request source.

The modality management unit 102 manages modalities. For example, as also shown in FIG. 2, the modality management unit 102 can perform the following processing.
・Get Modalities(): string[]
・Register Modality (modality: string, definer: Modality Definer)
・Create(obj: object, modality: string): entity

More specifically, for example, the modality management unit 102 registers the modality definition (data), and the modality definition unit 103 registers the object (registered object) input from the feature space management unit 101 as the modality of the object. It converts to a predetermined data format according to the definition and generates converted data (entity). An entity is shaped data that can be passed directly to the feature extractor 202 . Further, modality definition section 103 assigns a unique ID common to a plurality of feature spaces to the generated entity, and outputs the entity and the ID to feature space management section 101 . Further, the modality definition unit 103 associates the object (registered object) with the ID assigned to the entity of the registered object and stores them in the storage unit 140 . Such an ID is a unique character string across multiple feature spaces that deal with the same modality. Also, a hash value or the like may be used so that the same value is returned for the same entity.

In addition, the modality definition data consists of definition data in a format that can be received as input data (obj) (e.g., file name (string), OpenCV Mat format) (multiple possible) and definition data in the format of output data (entity) (only one) (eg char[3][256][256]). The modality definition data exists, for example, for each data format or for each pattern described above (conversion between formats, unification of formats, reading of special data, etc.), and is stored in the storage unit 140 . The modality management unit 102 converts the registered object (input data) using the modality definition data, and outputs the converted data (entity). In addition, the modality definition unit 103 has a function of saving data corresponding to the ID as required, a function of confirming that the data is saved, a function of retrieving data corresponding to the ID, and a function of deleting the data corresponding to the ID. It has the function to A modality definition unit 103 exists for each modality.

(Input unit 110)
The input unit 110 has a function of receiving instructions from the user, such as an operation input unit that receives operation instructions from the user and a voice input unit that receives voice instructions from the user, and outputs the instructions to the control unit 100. The operation input unit may be a touch sensor, pressure sensor, or proximity sensor. Alternatively, input 110 may be a physical structure such as buttons, switches, and levers.

(Communication unit 120)
The communication unit 120 connects to an external device by wire or wirelessly, and transmits and receives data to and from the external device. For example, the communication unit 120 can be a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE: Long Term Evolution, 3G (third generation mobile communication method)), etc., to transmit and receive data to and from the feature management server 20 via the network (not shown).

(Output unit 130)
The output unit 130 has a function of presenting (outputting) information to the user, such as a display unit and an audio output unit. For example, the output unit 130 outputs a search screen or a search result under the control of the control unit 100 .

(storage unit 140)
The storage unit 140 is implemented by a ROM (Read Only Memory) that stores programs, calculation parameters, and the like used in the processing of the control unit 100, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

For example, the storage unit 140 stores feature space management information, modality definition information, and objects (entity data) assigned unique IDs.

The configuration of the information processing apparatus 10 according to the present embodiment has been specifically described above. Note that the configuration of the information processing apparatus 10 is not limited to the example shown in FIG. For example, each process by the control unit 100 of the information processing apparatus 10 may be executed by a plurality of apparatuses, or may be executed by a server on a network.

<2-2. Configuration of Feature Management Server 20>
As shown in FIG. 2 , the feature management server 20 has a control section 200 , a communication section 210 and a feature amount database 220 . In the present embodiment, since each feature management server 20 has one feature extraction unit 202, it can be said that each feature management server 20 manages one feature space, but the present disclosure is limited to this. Instead, for example, if the feature management server 20 has a plurality of feature extraction units 202, it is possible to manage a plurality of feature spaces.

(control unit 200)
The control unit 200 functions as an arithmetic processing device and a control device, and controls overall operations within the feature management server 20 according to various programs. The control unit 200 is realized by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like. The control unit 200 may also include a ROM (Read Only Memory) for storing programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) for temporarily storing parameters and the like that change as appropriate.

The control unit 200 according to this embodiment also functions as a feature management unit 201 . The feature management unit 201 performs feature extraction (for example, substitution processing to an N-dimensional vector) by the feature extraction unit 202 for the entity transmitted from the information processing apparatus 10, and assigns the extracted feature quantity to the ID of the entity. A process of associating and registering in the feature amount database 220 is performed. Algorithms for feature quantity extraction can use existing techniques, and are not particularly limited here. Also, the feature extraction unit 202 can handle different modalities in a unified manner. The feature amount extracted by the feature extraction unit 202 is registered in the feature amount database 220. Here, the feature amount database 220 exists for each modality. The feature management unit 201 registers the feature amount extracted by the feature extraction unit 202 in the feature amount database 220 corresponding to the modality of the original data (the entity transmitted from the information processing apparatus 10) in association with the ID. For example, assume that there is a feature extraction unit 202a (feature space a of color features) capable of extracting feature amounts from different modalities (for example, handwriting (Strokes) and image (Image)) from the viewpoint of color features. In this case, the feature amount extracted by the feature extraction unit 202a is stored in the feature amount database 220-1 corresponding to the handwritten data when extracted from the handwritten data, and is stored in the feature amount database 220-1 corresponding to the handwritten data, and when extracted from the image data, the feature amount corresponding to the image data. Stored in quantity database 220-2.

Further, the feature management unit 201 can also perform search processing (similarity search) using the feature space in response to a request from the feature space management unit 101 . Since the feature amount database 220 exists for each modality, the feature management unit 201 may perform similarity search using the feature amount database 220 corresponding to the target modality (modality to be searched). For example, the feature management unit 201 can perform the following processing as shown in FIG.
・GetSpaceID(): string
・Register Database (modality: string, database: Feature Database)
・GetVector(entity: object, modality: string): vector
・Add(id: string, entity: object, modality: string)
・Most Similar(query: object, modality: string, target Modality: string): Search Result[]

(Communication unit 210)
The communication unit 210 connects to an external device by wire or wirelessly, and transmits and receives data to and from the external device. For example, the communication unit 210 is a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE: Long Term Evolution, 3G (third generation mobile communication method)) or the like to connect to a network (not shown) and transmit/receive data to/from the information processing apparatus 10 via the network.

(Feature database 220)
The feature amount database 220 accumulates the feature amounts extracted by the feature extraction unit 202 . A unique ID assigned by the modality management unit 102 is associated with each feature amount. The feature quantity database 220 exists for each modality as described above.

Also, the feature amount database 220 is stored in a storage unit (not shown) of the feature management server 20 . The storage unit of the feature management server 20 is implemented by a ROM that stores programs, calculation parameters, and the like used in the processing of the control unit 200, and a RAM that temporarily stores parameters that change as appropriate.

The configuration of the feature management server 20 according to this embodiment has been specifically described above. Note that the configuration of the feature management server 20 shown in FIG. 1 is an example, and the present embodiment is not limited to this. For example, at least part of the configuration of the feature management server 20 may be in an external device.

Here, FIG. 3 shows an example of another configuration example of the information processing system according to this embodiment. As shown in FIG. 3, for example, the feature extraction unit 240 and the feature amount database 250 may be managed by separate servers (the feature management server 24 and the database server 25).

<<3. Operation processing >>
Next, operation processing of the information processing system according to this embodiment will be specifically described with reference to the drawings.

<3-1. Registration process>
FIG. 4 is a sequence diagram showing an example of the flow of object registration processing in the information processing system according to this embodiment.

As shown in FIG. 4, first, when the feature space management unit 101 of the information processing apparatus 10 acquires a registration request based on a user's operation input or the like (step S103), the object (obj) included in the registration request and the relevant Along with the information of the modality (mdl) of the object, the modality management unit 102 is requested to generate (converted data (entity)) (step S106).

Next, the modality management unit 102 causes the modality definition unit 103 to generate conversion data (entity), assign a unique ID, and store obj and the unique ID (step S109). Specifically, the modality definition unit 103 performs processing (common preprocessing) for converting an object into data in a predetermined format according to the modality definition. Specific examples of processing include the following examples.
・For still images: JPEG data is converted to char[3][256][256] (multidimensional array) and smoothed.
・For audio: Read mp3 data as a short type arbitrary length array.
- For text: Remove HTML tags and convert all uppercase letters to lowercase (do not convert format).
・In the case of handwriting: read the point sequence data and draw a white line with a thickness of 3 on a black image of char[3][256][256].

Next, the feature space manager 101 acquires at least the ID and entity from the modality manager 102 (step S112). Also, the modality management unit 102 may notify that the ID and obj have been saved.

Next, the feature space management unit 101 outputs a data addition (registration) request to all corresponding feature spaces based on the acquired ID and entity (step S115). The add request includes ID, entity and modality (mdl). The corresponding feature space is the feature space (feature management server 20) that can handle the modality of the entity. Note that if there are a plurality of feature spaces that can handle the modality of the entity, the processing shown in steps S115 to S121 is repeated for each feature space.

Next, the feature management unit 201 causes the feature extraction unit 202 to extract feature amounts (step S118).

Then, the feature management unit 201 adds (registers) the extracted feature amount to the feature amount database 220 together with the acquired unique ID (step S121). At this time, the feature management unit 201 registers in the feature amount database 220 corresponding to the modality of the extraction source entity.

The registration processing according to the present embodiment has been specifically described above. In this way, before inputting data into each feature space, the modality management unit 102 performs a predetermined conversion process for each modality. can be omitted, and the convenience of a system that handles multiple feature spaces can be improved. Also, by managing the modality management unit 102 and the feature extraction unit 202 separately, the responsibility of each function is reduced (for example, it becomes easier to identify the cause of an error).

Also, for example, if a general database capable of storing IDs and feature vectors is provided, a search system can be completed by developing only a feature extractor, and the availability of the system can be improved. Also, in the search DB (that is, the feature amount database 220), only IDs and feature amounts are registered, and the original data (the substance of the data) is managed separately by the modality definition unit 103. Therefore, the same data can be registered in a plurality of databases. verbosity can be avoided. In addition, by assigning a unique ID to the same data across multiple feature spaces, it is possible to associate multiple IDs with the same data, or associate the same ID with multiple data. can be avoided.

Also, the feature manager 201 may register in the feature quantity database 220 only when the original data is stored in the modality manager 102 . This guarantees that the feature space management unit 101 can retrieve the original data from the ID when obtaining a search result, which will be described later.

In addition, since the preprocessing to a predetermined data format is managed separately, the feature space (search DB system) development side can develop the feature extraction unit 202 without worrying about the input format. .

Further, the registration process according to this embodiment is not limited to the example shown in FIG. For example, in the above step S115, it has been explained that an additional instruction is given to the feature space (feature management server 20) corresponding to the modality, but the present embodiment is not limited to this, and the feature space management unit 101 allows all feature An additional instruction may be given to the space (feature management server 20). In this case, the feature space (feature management server 20) side can determine whether or not the entity can be processed based on the modality.

<3-2. Search processing>
Next, search processing in the information processing system according to this embodiment, which constructs the feature space as described above, will be described with reference to FIG. FIG. 5 is a sequence diagram showing an example of the flow of search processing in the information processing system according to this embodiment.

As shown in FIG. 5, first, the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on user's operation input or the like (step S133). The search request includes the object (obj), the modality (mdl1) of the object, and the target modality (mdl2) indicating the modality to be searched.

Next, the feature space management unit 101 requests the modality management unit 102 to generate (converted data (entity)) together with the information of the object (obj) and the modality (mdl1) of the object included in the search request. (Step S136).

Next, the modality management unit 102 causes the modality definition unit 103 to generate conversion data (entity) and assign a unique ID (step S139). Specifically, the modality definition unit 103 performs processing (common preprocessing) for converting an object into data in a predetermined format according to the modality definition.

Next, the feature space manager 101 acquires the ID and entity from the modality manager 102 (step S142).

Next, the feature space management unit 101 outputs a search request to all corresponding feature spaces based on the acquired entity (step S145). The search request includes entity, mdl1 (original data modality), and mdl2 (target modality). The corresponding feature space is the feature space (feature management server 20) that can handle mdl1 and mdl2. Note that if there are a plurality of feature spaces that can handle mdl1 and mdl2, the processing shown in steps S145 to S157 is repeated for each feature space.

Next, the feature management unit 201 causes the feature extraction unit 202 to extract feature amounts (step S148).

Subsequently, the feature management unit 201 searches for similar feature amounts from the feature amount database 220 based on the extracted feature amounts (step S151). At this time, the feature manager 201 searches the feature amount database 220 corresponding to the requested target modality (mdl2). In the feature amount database 220, the feature amount is associated with the above-mentioned unique ID. An ID associated with a feature amount and the similarity of the feature amount: sim (similarity between the feature amount of the requested entity and, for example, the distance of an N-dimensional vector) are obtained. Note that if there are a plurality of target modalities (mdl2), the processing shown in steps S151 to S154 is repeated for each feature amount database 220. FIG.

Next, the feature space management unit 101 acquires the search result (the ID of the retrieved feature amount, the retrieved modality: mdl, and the similarity of the retrieved feature amount: sim) from the feature management unit 201 (step S157). . Search results may include multiple ids, mdl, and sim. When a singular number is obtained as a search result, the feature space management unit 101 identifies, for example, the ID of the feature quantity with the highest similarity. When a predetermined number are obtained as search results, the feature space management unit 101 identifies IDs of a predetermined number of top feature amounts based on similarity, for example.

Next, the feature space management unit 101 requests the modality management unit 102 for the original data together with the identified ID and the corresponding modality information (step S160).

Next, the modality management unit 102 acquires the original data (that is, the object) associated with the ID by the modality definition unit 103 (step S163), and outputs it to the feature space management unit 101 (step S166). The processing shown in steps S160 to S166 can be performed for the number of search results to be output. If the original data cannot be acquired, the feature space management unit 101 deletes the record (ID, mdl, sim).

Then, the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S169). For example, the feature space management unit 101 may display a screen showing search results on the output unit 130 and present it to the user.

The search processing according to the present embodiment has been specifically described above. In this way, the feature space constructed according to this embodiment can be used for cross-sectional searches in a plurality of feature spaces dealing with different modalities.

As shown in the last example above, the feature space (search DB) used for searching may be specified. The specified feature space is included in the search request in step S113 as a space ID. For example, "I want to use the search DB created by XX company", "I want to use the XX search site", etc. are assumed.

Here, FIG. 6 shows an example of the search screen in this embodiment. The search screen 30 shown in FIG. 6 is presented by the output unit 130, for example. A user inputs a search object 301, selects a search target (corresponding to a modality; for example, "photograph", "illustration", "document", etc.), and specifies the feature amount of what is similar to what he/she wants to search. Select (e.g., 'shape', 'color', 'meaning', etc., corresponding to each feature space. For example, feature space constructed based on shape features, color feature ), and when a search button 302 is selected, an object similar to the search object 301 obtained as a search result is presented. For example, if "illustration" is selected as a search target, and "shape" (space ID1), "color" (space ID2), and "meaning" (space ID3) are selected as feature quantities, a search object 301 , illustrations similar in shape, color, and/or meaning are obtained (for example, the feature amount database of "modality: illustration" held by the feature management server 20 that handles feature spaces constructed based on the features of the shape 220) and presented. The user may arbitrarily select and/or of the search condition of the feature amount, or the default may be set to or.

<<4. Application example >>
Next, an application example of the information processing system according to this embodiment will be described.

<4-1. First Application Example: Definition of Inclusive Relationship of Modalities>
First, as a first application example, the definition of the inclusion relationship of modalities will be described. The modality definition unit 103 according to this embodiment may define a parent-child relationship (inclusive relationship) between modalities. Specific examples include the following.

・The modality “RGB image” has the modality “grayscale image” as a child. ・The modality “mail” has the modalities “text”, “user”, and “date” as children.

According to this application example, when defining a new modality, if a child modality is defined, it can be easily incorporated into an existing feature space. Also, for new modalities, feature extraction can be performed by combining a plurality of feature extractors (feature space, feature extraction unit 202).

For example, consider a use case that defines modalities that include existing modalities. More specifically, it is assumed that the modality "text" already exists and there is a feature space A (feature extraction unit 202A) that handles "text". It is assumed here that a modality "email" including text (text) and a user (sender) and a feature space B (feature extraction unit 202B) capable of handling "email" are added. In this case, the first effect is that "it can be registered in a plurality of feature spaces at the same time". That is, since the text can be obtained from the mail, the same ID/object pair can be registered not only in the feature space B but also in the feature space A. That is, the feature space A stores the feature amount when focusing only on the text, and the feature space B stores the feature amount when focusing on the text and the user. In addition, it is possible to search across other texts (in this case, it is necessary to assign a unique ID across all modalities, not just the same modality).

A second effect is that "a new feature extractor (feature extraction unit 202) can be easily constructed by reusing an existing feature extractor (feature extraction unit 202)." That is, since the feature space B can use the feature space A to extract the feature of the text, only the feature of the user needs to be extracted in the feature space B, which facilitates implementation. It is also possible to use an existing feature extractor as a module. FIG. 7 is a diagram explaining modularization of the feature extractor.

As shown on the left side of FIG. By extracting the feature amount of , it is possible to obtain the mail feature amount including the text feature amount (content) and the user feature amount (sender), as shown in the right side of FIG.

Registration processing and search processing in this application example will be sequentially described below with reference to FIGS. 8 and 9, respectively.

(Registration processing considering definition of modality inclusion relationship)
FIG. 8 is a sequence diagram showing an example of registration processing of the first application example according to this embodiment.

As shown in FIG. 8, first, when the feature space management unit 101 of the information processing apparatus 10 acquires a registration request based on a user's operation input or the like (step S203), the object (obj) included in the registration request and the relevant Along with the information of the modality (mdl) of the object (for example, "Mail"), the modality management unit 102 is requested to generate (entity) (step S206).

Next, the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity), assign a unique ID, and store obj and the unique ID. A sub entity is generated based on the definition of the modality (sub mdl) (step S209). For example, if the modality of the object is "Mail" and the modality (sub mdl) having an inclusion relationship with it is "Text", the modality definition unit 103, according to the definition of "Text", It converts text data into a predetermined data format and performs processing to generate it as a sub entity.

Next, the feature space manager 101 acquires at least the ID, entity, and sub entity from the modality manager 102 (step S212). Also, the modality management unit 102 may notify that the ID and obj have been saved.

Next, the feature space management unit 101 outputs a data addition (registration) request to all corresponding feature spaces (eg, feature space B) based on the acquired ID and entity (eg, Mail Entity) ( step S215). The processing related to extraction of the feature amount shown in subsequent steps S218 to S221 is the same as steps S118 to S121 shown in FIG. 4, so a detailed description will be omitted. The feature quantity (Mail Vector) is registered. At this time, for the text (Text Vector) among the features of the mail, the Get Vector (step S227) of the feature space A that handles the text, which will be described below, is used. can be

The feature space management unit 101 outputs a data addition (registration) request to all feature spaces (for example, feature space A) corresponding to the same ID and sub entity (for example, Text Entity) (steps S224 to 230). . Feature space A is a feature space that corresponds only to text, and a text feature quantity (Text Vector) extracted from Text Entity is registered.

As described above, in this modified example, it is possible to use a feature space having an inclusive relationship in extracting feature amounts, and to register feature amounts in the feature space as well.

(Search processing considering definition of modality inclusion relationship)
Next, search processing according to this modification will be described with reference to FIG. FIG. 9 is a sequence diagram showing an example of search processing of the first application example according to this embodiment.

As shown in FIG. 9, first, the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on the user's operation input (step S243). The search request includes the object (obj), the modality (mdl1) of the object, and the target modality (mdl2) indicating the modality to be searched. Here, for example, mdl1=Mail and mdl2=Text.

Next, the feature space management unit 101 requests the modality management unit 102 to generate (entity) together with the information of the object (obj) and the modality (mdl1) of the object included in the search request (step S246). .

Next, the modality management unit 102 causes the modality definition unit 103 to generate conversion data (entity), assign a unique ID, and based on the definition of a modality (sub mdl) having an inclusion relationship with the modality of obj. Then, a sub entity is generated (step S249). For example, if the modality of the object is "Mail" and the modality (sub mdl) having an inclusion relationship with it is "Text", the modality definition unit 103, according to the definition of "Text", It converts text data into a predetermined data format and performs processing to generate it as a sub entity.

Next, the feature space manager 101 acquires the ID, entity, and sub entity from the modality manager 102 (step S252).

Next, the feature space management unit 101 outputs a search request to all corresponding feature spaces based on the acquired entity (for example, Mail Entity) (step S255). The search request includes entity, mdl1 (original data modality, such as Mail), and mdl2 (target modality, such as Text). A corresponding feature space is a feature space that can handle mdl1 and mdl2 (for example, a feature space that supports both mail and text).

Processing related to the extraction of the feature quantity shown in subsequent steps S258 to S267 is the same as steps S148 to S157 shown in FIG. 5, so detailed description thereof will be omitted. Here, it is assumed that there is no corresponding feature space. For example, if there are a feature space B that handles email and a feature space A that handles text, neither feature space handles both email and text, so no search results are returned. , the search results may be returned from feature space A.

The feature space management unit 101 outputs a search request to all corresponding feature spaces based on the same ID and sub entity (for example, Text Entity) (step S270). The search request includes sub entity (eg Text Entity), mdl1 (sub mdl, eg Text), mdl2 (target modality, eg Text). The corresponding feature space corresponds to a feature space that can handle mdl1 and mdl2, here, since mdl1 and mdl2 are the same "Text", feature space A corresponding to text corresponds. A search is performed in the feature space A (steps S273 to S279), and the feature space management unit 101 acquires search results from the feature management unit 201 (step S282).

Subsequently, the feature space management unit 101 requests the modality management unit 102 for the original data together with the acquired ID and the corresponding modality (for example, Text) in the same manner as in steps S160 to S169 shown in FIG. (step S285), the object acquired by the modality definition unit 103 based on the ID (step S288) is output from the modality management unit 102 to the feature space management unit 101 (step S291).

Then, the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S294).

The search processing in consideration of the inclusion relationship of modalities according to this application example has been specifically described above.

<4-2. Second Application Example: Merging of Search Results>
Next, merging of search results will be described as a second application example. The feature space management unit 101 can re-evaluate the search results based on the similarity and weighting of the search results from each feature extractor, and then output the final search results to the search requester. . Weighting is, for example, weighting of feature space. Such weighting can be arbitrarily set by the search requester (for example, the user).

FIG. 10 is a diagram showing an example of a search screen in this application example. As shown in FIG. 10, the search screen 32 includes a search object 321, a search target selection area 322, an area 323 for selecting a similar feature amount to search for, and a search button 326. is displayed. In the area 323 for selecting the feature amount, it is possible to operate the slide bar 324 to set the weighting of the selected feature amount. For example, if it is desired to give priority to the "color feature" out of the "shape feature" and the "color feature", the operation section 325 of the slide bar 324 is moved to the "color feature". Accordingly, for example, the system side sets the weighting (w) as follows. Here, the color feature space: space1 and the shape feature space: space2.
w(weights)＝{space1: 0.8, space2: 0.2}

In this case, as shown in FIG. 10, a search result giving priority to "color characteristics" (search results giving priority to illustrations with similar colors) is displayed.

Note that the setting of the weighting of the feature space is not limited to the example shown in FIG. 10. For example, the weighting may be set based on what the user selects from the search results, and the search results may be presented again. An example is shown in FIG. For example, when the illustration 341 is selected from among the search results presented on the search screen 34 of FIG. 11, the system outputs the illustration 341 as the search result, assuming that the illustration 341 is a result close to the user's intention. A weighting may be set so as to give priority to the feature space (feature extractor, that is, the feature extraction unit 202), and the search result may be presented again.

(operation processing)
Next, operation processing of this application example will be described with reference to FIG. FIG. 12 is a sequence diagram showing an example of search processing of the second application example.

As shown in FIG. 12, first, the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on the user's operation input (step S303). The search request includes the object (obj), the modality (mdl1) of the object, the target modality (mdl2) indicating the modality to be searched, and the weighting (w) of the feature space.

In subsequent steps S315 to S321, search processing similar to the processing shown in steps S145 to S157 in FIG. 5 described above is performed, so detailed description thereof will be omitted here. In step S318, the same processing as that shown in steps S148 to S154 in FIG. 5 is performed, but detailed illustration is omitted.

Next, the feature space management unit 101 ranks (re-evaluates) the search results according to the degree of similarity and weighting of the search results (step S324). Specifically, for example, the feature space management unit 101 multiplies the similarity of the search result by the weight of the feature space (feature extractor, that is, the feature extraction unit 202) that outputs the search result, and obtains a new similarity can be calculated and re-evaluated. Table 1 below shows an example of re-evaluation. Here, w (weights) = {space1: 0.8, space2: 0.2}.

As shown in Table 1 above, for example, the similarity (sim(space1): 0.9) when object A, which is the retrieval result, is retrieved from the first feature space (space1) and the weight of the first feature space ( space1: 0.8) and the similarity when searched from the second feature space (space2) (sim(space2): 0.3) multiplied by the weight of the second feature space (space2: 0.2) value (sim(new): 0.78) is calculated as a new degree of similarity. This is because IDs associated with the same data may be registered in multiple feature spaces. It is also assumed that search results are obtained from only one feature space. In this case, as in the example of object C in Table 1 above, for example, when object C is retrieved from the second feature space (space2), the similarity (sim(space2): 0.9) to the second feature space A value (sim(new): 0.18) obtained by multiplying the weight (space2: 0.2) is calculated as a new degree of similarity.

Based on the new degree of similarity, the feature space management unit 101 identifies, for example, a predetermined number of top search results (IDs).

Next, the feature space management unit 101 requests original data from the modality management unit 102 together with the specified ID and the corresponding modality, similarly to steps S160 to S169 shown in FIG. 5 (step S327). The object acquired based on the ID by the modality definition unit 103 (step S330) is output from the modality management unit 102 to the feature space management unit 101 (step S333).

Then, the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S336).

The merging of search results according to this application example has been specifically described above.

<4-3. Third Application Example: Suggestion System>
Next, as a third application example, a suggestion system will be described with reference to FIGS. 13 to 15. FIG. The suggestion system is used on a system running multiple applications to search for content that matches the situation based on the user's operation information (content being viewed, content being operated, etc.) in each application. and make it possible to propose to the user.

For example, assume that the user is planning a trip using various applications. If the user searches for tourist spots on a web browser, searches for local maps on a map app, and organizes plans on a note app, the suggestion system will be able to match demand according to the usage of these multiple applications. It is possible to propose content (web pages, text, images, etc.) that is unique to the user.

(Configuration example)
FIG. 13 is a functional block diagram showing an example of the configuration of this system. As shown in FIG. 13, for example, the suggestion system can be realized by the information processing device 10x. The information processing device 10x functions as one or more applications 105, an information collection unit 106, a suggestion unit 107, a feature space management unit 101x, and a modality management unit 102x. These can be performed by the control unit 100 of the information processing device 10 .

The application 105 is various application programs such as a web browser, a map application, and a notebook application.

The information collection unit 106 has a function of monitoring the operation of each application 105 and collecting and accumulating user operation information (i.e. application usage status) in each application 105 . Also, an OS (Operating System) may be used for the information collecting unit 106 .

The suggestion unit 107 generates a search request based on the operation information collected by the information collection unit 106, and issues the search request to the feature space management unit 101x. For example, the suggestion unit 107, based on the modality (mdl1) and content (obj) of the content being viewed/edited acquired from each application 105 from the information collection unit 106, and the modality (mdl2) of the required content, A search request can be generated. The modalities of content acquired from each application 105 and the modalities of required content are assumed to be, for example, as follows.
・Web browser…Browsing: Web page, Request: Web page ・Map application…Browsing: Address, Request: None ・Note application…Editing: Text/image, Request: Text/image

In response to a request from the suggestion unit 107, the feature space management unit 101x performs search processing using one or more feature spaces. The search processing is the same as in the above-described embodiment. First, the feature space management unit 101x obtains the entity obtained by converting obj by the modality management unit 102x, and obtains the entity, mdl1 (for example, Web page, address, text), and mdl2. (for example, web page, image), a search request is made to the feature management server 20 . The feature space management unit 101 x then outputs the search result to the suggestion unit 107 .

The modality management unit 102x has the same function as the modality management unit 102 described with reference to FIG. and outputs the generated entity to the feature space management unit 101x.

An example of the configuration of the information processing device 10x that executes the suggestion system according to the application example has been specifically described above.

(operation processing)
Next, operation processing of the suggestion system according to this application example will be described with reference to FIG. 14 . FIG. 14 is a sequence diagram showing an example of the flow of search processing in the suggestion system of this application example.

As shown in FIG. 14, first, when one or more applications 105 are operated by a user (step S403), the content being handled is sent (post; obj, mdl1) and the required content is requested (request ;mdl2) to the information collection unit 106 (step S406). Examples of posts include a web page of “Kinkakuji”, an address of “1-2-3 Kita-ku, Kyoto”, and travel-related texts. Also, the request includes, for example, a web page and an image.

Next, the information collection unit 106 outputs the collected information (post, request) to the suggestion unit 107 (step S412).

Next, the suggestion unit 107 makes a search request to the feature space management unit 101x (step S415). The search request includes the content included in the post as obj, its modality as mdl1, and the modality of the content included in the request as mdl2.

Next, search processing is executed in the feature space management unit 101x (step S418). In step S418, the same processing as steps S136 to S166 in FIG. 5 (generating an entity from obj and mdl1, searching based on entity, mdl1 and mdl2, and obtaining an object from the ID of the search result) is performed. Detailed description is omitted.

Next, the suggestion unit 107 acquires search results from the feature space management unit 101x (step S421).

Next, the suggestion unit 107 may rank (re-evaluate) the search results according to the degree of similarity and weighting (W) of the search results (step S424). For example, the suggestion unit 107 may set weights as shown in Table 2 below for each input and output, and rank them by multiplying the degrees of similarity. Note that in this application example, such re-evaluation may be skipped.

Then, the suggestion unit 107 creates a display screen showing the search results (step S427) and presents it to the user (step S430).

In addition, the suggestion unit 107 may update the weighting (W) used in step 424 (step S433) when feedback on the usage status is obtained from the user.

Note that the suggestion to the user by the suggestion unit 107 and the acquisition of feedback from the user may be performed via the application 105 .

Here, FIG. 15 shows an example of operation information and request information acquired from the application according to this application example. This system acquires operation information shown on the left side of FIG. 15 and request information shown on the right side of FIG. 15 from each application, and suggests requested information based on the operation information.

<<5. Summary>>
As described above, in the information processing system according to the embodiment of the present disclosure, it is possible to further improve the convenience of a system that handles multiple feature spaces.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is obvious that those who have ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. is naturally within the technical scope of the present disclosure.

For example, a computer program for causing hardware such as a CPU, ROM, and RAM incorporated in the information processing apparatus 10 or the feature management server 20 described above to exhibit the functions of the information processing apparatus 10 or the feature management server 20. can be created. A computer-readable storage medium storing the computer program is also provided.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the present technology can also take the following configuration.
(1)
control to associate a registration object included in the registration request information with unique first identification information common to a plurality of feature extraction units and store the registration object in a storage unit;
a control for converting the registration object according to the modality definition of the registration object to generate conversion data for registration;
Control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modalities;
An information processing device comprising a control unit that performs
(2)
The information processing apparatus according to (1), wherein the definition of the modality is a conversion rule into a predetermined data format corresponding to the modality.
(3)
The control unit
control for converting a search object included in a search request in accordance with the modality definition of the search object to generate conversion data for the search;
control for outputting the transformation data for retrieval to the feature extractor corresponding to the modality of the retrieval object and the target modality included in the retrieval request;
The information processing apparatus according to (1) or (2) above.
(4)
The control unit
Acquiring second identification information retrieved based on the transformation data for retrieval in one or more of the feature extractors;
The information processing apparatus according to (3), wherein the corresponding object is acquired from the storage unit based on the second identification information and output as a search result.
(5)
The control unit
Acquiring from the feature extractor the second identification information associated with the feature similar to the feature extracted from the conversion data for search and the degree of similarity indicating the degree of similarity of the feature; (4) The information processing device according to .
(6)
The information processing apparatus according to (4) or (5), wherein the search request further includes filter information as a search condition.
(7)
The control unit
control for generating sub-converted data for registration by converting the registered object according to the definition of a submodality having a parent-child relationship with the modality of the registered object when the registration request information is input;
Control for outputting the first identification information and the sub-transformed data to one or more feature extractors corresponding to the sub-modalities;
The information processing apparatus according to any one of (1) to (6), further performing
(8)
The control unit
When the search request is input, data corresponding to the submodality of the search object is converted to generate sub-converted data for search according to the definition of the submodality having a parent-child relationship with the modality of the search object. control and
Control for outputting the sub-transformed data for searching to one or more of the feature extractors corresponding to the submodality and the target modality;
The information processing apparatus according to any one of (3) to (6), further performing
(9)
The control unit
Control for ranking the plurality of pieces of second identification information based on the second identification information and the degree of similarity obtained from the feature extractor based on the search request and the weighting of the feature extractor;
Control for outputting the second identification information of a predetermined number of higher ranks as the search result;
The information processing apparatus according to any one of (4) to (6), further performing
(10)
The control unit
generating the search request for searching for content to be proposed to the user based on information including user operation information output from one or more applications;
The information processing apparatus according to any one of (4) to (6), wherein the one or more pieces of second identification information acquired from the feature extractor are output as the search result.
(11)
The control unit
The content handled by the application, which is included in the information, is defined as the search object;
Let the modality of the content be the modality of the search object,
The information processing apparatus according to (10), wherein the search request is generated with the modality of the content requested by the application as the target modality.
(12)
The information processing device is
The information according to any one of (1) to (11) above, further comprising a communication unit that transmits the first identification information and the conversion data for registration to a feature management server having the feature extractor. processing equipment.
(13)
the processor
control to associate a registration object included in the registration request information with unique first identification information common to a plurality of feature extraction units and store the registration object in a storage unit;
a control for converting the registration object according to the modality definition of the registration object to generate conversion data for registration;
Control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modalities;
A method of processing information, including performing
(14)
the computer,
control to associate a registration object included in the registration request information with unique first identification information common to a plurality of feature extraction units and store the registration object in a storage unit;
a control for converting the registration object according to the modality definition of the registration object to generate conversion data for registration;
Control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modalities;
A program that functions as a control unit that performs

10, 10x information processing device 20, 24 feature management server 25 database server 100 control unit 101, 101x feature space management unit 102, 102x modality management unit 103 modality definition unit 105 application 106 information collection unit 107 suggestion unit 110 input unit 120 communication unit 130 output unit 140 storage unit 200 control unit 201 feature management unit 202 feature extraction unit 210 communication unit 220 feature amount database 240 feature extraction unit 250 feature amount database

Claims (14)

In an information processing device connected to a plurality of feature extractors,
Control for storing the registration object included in the registration request information in a storage unit in association with unique first identification information common to a plurality of feature extractors corresponding to modalities among the plurality of feature extractors ;
a control for converting the registration object according to the modality definition of the registration object to generate conversion data for registration;
Control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality among the plurality of feature extractors;
An information processing device comprising a control unit that performs 2. The information processing apparatus according to claim 1, wherein said modality definition is a conversion rule into a predetermined data format corresponding to the modality. The control unit
control for converting a search object included in a search request in accordance with the modality definition of the search object to generate conversion data for the search;
control for outputting the transformation data for retrieval to the feature extractor corresponding to the modality of the retrieval object and the target modality included in the retrieval request;
The information processing apparatus according to claim 1, wherein The control unit
Acquiring second identification information retrieved based on the transformation data for retrieval in one or more of the feature extractors;
4. The information processing apparatus according to claim 3, wherein a corresponding object is acquired from said storage unit based on said second identification information and output as a search result. The control unit
5. The method according to claim 4, wherein from the feature extractor, the second identification information associated with the feature similar to the feature extracted from the conversion data for search and the similarity indicating the degree of similarity of the feature are obtained. The information processing device described. 5. The information processing apparatus according to claim 4, wherein said search request further includes filter information as a search condition. The control unit
control for generating sub-converted data for registration by converting the registered object according to the definition of a submodality having a parent-child relationship with the modality of the registered object when the registration request information is input;
Control for outputting the first identification information and the sub-transformed data to one or more feature extractors corresponding to the sub-modalities;
The information processing apparatus according to claim 1, further comprising: The control unit
When the search request is input, data corresponding to the submodality of the search object is converted to generate sub-converted data for search according to the definition of the submodality having a parent-child relationship with the modality of the search object. control and
Control for outputting the sub-transformed data for searching to one or more of the feature extractors corresponding to the submodality and the target modality;
4. The information processing apparatus according to claim 3, further comprising: The control unit
Control for ranking the plurality of pieces of second identification information based on the second identification information and the degree of similarity obtained from the feature extractor based on the search request and the weighting of the feature extractor;
Control for outputting the second identification information of a predetermined number of higher ranks as the search result;
5. The information processing apparatus according to claim 4, further comprising: The control unit
generating the search request for searching for content to be proposed to the user based on information including user operation information output from one or more applications;
5. The information processing apparatus according to claim 4, wherein said one or more pieces of said second identification information acquired from said feature extractor are output as said search result. The control unit
The content handled by the application, which is included in the information, is defined as the search object;
Let the modality of the content be the modality of the search object,
11. The information processing apparatus according to claim 10, wherein the search request is generated with the modality of the content requested by the application as the target modality. The information processing device is
2. The information processing apparatus according to claim 1, further comprising a communication unit that transmits said first identification information and said conversion data for registration to a feature management server having said feature extractor. In an information processing method for an information processing device connected to a plurality of feature extractors,
the processor
Control for storing the registration object included in the registration request information in a storage unit in association with unique first identification information common to a plurality of feature extractors corresponding to modalities among the plurality of feature extractors ;
a control for converting the registration object according to the modality definition of the registration object to generate conversion data for registration;
Control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality among the plurality of feature extractors;
A method of processing information, including performing In a computer program connected to a plurality of feature extractors,
said computer,
Control for storing the registration object included in the registration request information in a storage unit in association with unique first identification information common to a plurality of feature extractors corresponding to modalities among the plurality of feature extractors ;
a control for converting the registration object according to the modality definition of the registration object to generate conversion data for registration;
Control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality among the plurality of feature extractors;
A program that functions as a control unit that performs

Images