WO2019176398A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et programme - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et programme Download PDF

Info

Publication number
WO2019176398A1
WO2019176398A1 PCT/JP2019/004534 JP2019004534W WO2019176398A1 WO 2019176398 A1 WO2019176398 A1 WO 2019176398A1 JP 2019004534 W JP2019004534 W JP 2019004534W WO 2019176398 A1 WO2019176398 A1 WO 2019176398A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature
modality
search
registration
information processing
Prior art date
Application number
PCT/JP2019/004534
Other languages
English (en)
Japanese (ja)
Inventor
光平 西村
Original Assignee
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニー株式会社 filed Critical ソニー株式会社
Priority to JP2020505682A priority Critical patent/JP7255585B2/ja
Publication of WO2019176398A1 publication Critical patent/WO2019176398A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor

Definitions

  • the present disclosure relates to an information processing apparatus, an information processing method, and a program.
  • BoW BoW
  • N the number of vocabulary
  • LBP local binary patterns
  • a space spanned by such vectors is called a feature space and is used as a feature of search technology and machine learning technology.
  • the demand for feature spaces will increase further in the future, and there will be demands for handling multiple feature spaces and using them across and switching.
  • Non-Patent Document 1 discloses a technique for mapping text and images to a semantic space (a technique for mapping multimodal to a single space).
  • the present disclosure proposes an information processing apparatus, an information processing method, and a program that can further improve the convenience of a system that handles a plurality of feature spaces.
  • An information processing apparatus including a control unit is proposed.
  • the processor stores the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units, and the registration object Control for converting according to the definition of the modality of the object and generating conversion data for registration; and control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality
  • An information processing method including performing the above is proposed.
  • the computer controls the registration object included in the registration request information to be stored in the storage unit in association with the unique first identification information common to the plurality of feature extraction units, and the registration object is registered in the registration unit.
  • Control for converting according to the definition of the modality of the object and generating conversion data for registration; and control for outputting the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality A program for functioning as a control unit for performing the above is proposed.
  • FIG. 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present disclosure. As illustrated in FIG. 1, the information processing system according to the present embodiment includes an information processing apparatus 10 and a feature management server 20.
  • the feature management server 20 replaces various data (hereinafter referred to as “objects”) such as text, images, tabular data, and time-series data with N-dimensional vectors that well represent the features of the data (feature extraction).
  • objects such as text, images, tabular data, and time-series data
  • a feature extraction unit 202 an example of a feature extractor that performs processing
  • the feature extraction unit 202 and the feature space have a one-to-one relationship.
  • a plurality of feature management servers 20 may exist as shown in FIG. 1, each having one feature extraction unit 202 (that is, each feature management server 20 has one feature space. It can be said that it is managing.
  • the handling of a general feature space will be described below.
  • the degree of similarity and relationship between objects can be expressed, and for example, search and recommendation by concept can be performed.
  • the accuracy of other machine learning techniques can be improved, and for example, it can be used as a feature in a recognition system, data analysis, or the like.
  • different modalities can be handled in a unified manner. For example, text, handwriting, and notes can be handled in one feature space.
  • the “modality” is a data input / output format, and for example, various modalities are assumed as described below.
  • first feature space that can process text, notes, and Web pages from a viewpoint of meaning
  • second feature space that can process handwriting and an image from a viewpoint of shape
  • third-party extension extension by plug-in
  • third-party extension is assumed as a method of handling multiple feature spaces across. More specifically, it is possible for another person to associate another modality with the same feature space, or to configure another feature space from the same modality. It is also possible to expand by doing both of these (for example, by associating the modalities of “paper” and “case” and entering sensor data for each case, it is possible to search for cases close to the paper or case. it can). It can also be used in combination with the feature space handled by others (for example, searching for products from text or images).
  • pre-process when performing feature extraction, it is important to process (pre-process) data until it is put into the feature extractor, and it is necessary to follow a predetermined process.
  • pre-process -Accept only JPEG or PNG data-Unify to RGB 3 channel
  • Cleaning process removes noise in the text.
  • Web text an HTML tag or a JavaScript (registered trademark) source code) -Word division of sentences, word normalization, stop word removal, etc.
  • a rule for conversion to a predetermined data format is defined for each modality, and converted data (hereinafter also referred to as “entity”) converted according to the definition of the object modality in the information processing apparatus 10 corresponds to the modality.
  • entity converted data
  • To one or more feature management servers 20 server devices having a feature extraction unit 202 which is an example of a feature extractor).
  • search database (hereinafter referred to as a search DB)
  • data is stored for each feature space, the same data is registered in a plurality of databases, which is redundant. Therefore, it is conceivable to prepare a database capable of storing data and extracting data from the ID separately from the search DB, and storing only the ID in the search DB. At this time, if the user arbitrarily inputs the ID together with the data, there is a possibility that the following problem may occur and the user must take care.
  • a unique (unique) ID (identification information) common to a plurality of feature extractors is given to the conversion data of the registered object.
  • a plurality of feature spaces are created by uniformly converting objects for each modality and managing objects with unique IDs common to a plurality of feature spaces. It is possible to further improve the convenience of the handling system.
  • the information processing apparatus 10 includes a control unit 100, a communication unit 120, an output unit 130, and a storage unit 140.
  • the information processing apparatus 10 is a local terminal such as a smartphone, a tablet terminal, or a PC used by a user, for example.
  • Control unit 100 The control unit 100 functions as an arithmetic processing unit and a control unit, and controls overall operations in the information processing apparatus 10 according to various programs.
  • the control unit 100 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example.
  • the control unit 100 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the control unit 100 also functions as a feature space management unit 101 (Feature Space Manager) and a modality management unit 102 (Modality Manager).
  • the feature space management unit 101 performs feature space management (acquisition of one or more feature management server 20 IDs), object registration processing, search processing, and the like.
  • FIG. 2 shows an example of processing contents in the main functional blocks (the feature space management unit 101, the modality management unit 102, and the feature management unit 201) of the information processing system according to the present embodiment.
  • the feature space management unit 101 according to the present embodiment can perform the following processing.
  • the feature space management unit 101 when the registration request information is input, the feature space management unit 101 outputs the object and modality included in the registration request information to the modality management unit 102, and the modality management unit 102 follows the definition of the modality.
  • the converted conversion data (entity) and ID are acquired, and the entity and ID are output to the feature management server 20.
  • the feature space management unit 101 can output to one or more feature management servers 20 corresponding to the modality of the object.
  • the “feature management server 20 corresponding to the modality” is the feature management server 20 capable of handling the modality.
  • the feature space management unit 101 can grasp what information each feature space handles from the ID (space ID: string, etc.) of the feature management server 20.
  • the feature space management unit 101 can refer to each space ID and identify the feature management server 20 that manages the feature space that handles “Text”. .
  • the feature space management unit 101 may output the information to all the feature management servers 20 (in this case, the feature management server 20 may determine whether or not processing can be appropriately performed).
  • the feature space management unit 101 when the search request information is input, the feature space management unit 101 outputs the object and modality included in the search request to the modality management unit 102, and the converted data (converted by the modality management unit 102 according to the definition of the modality ( entity), and outputs the entity to the feature management server 20.
  • the feature space management unit 101 can output to one or more feature management servers 20 corresponding to the modality of the object.
  • the feature space management unit 101 may output the feature request to the feature management server 20 corresponding to the designated space ID when the search request includes a space ID as a search condition.
  • the feature space management unit 101 may output the information to all the feature management servers 20 (in this case, the feature management server 20 may determine whether or not processing can be appropriately performed).
  • the feature space management unit 101 extracts a corresponding object (that is, original data) from the storage unit 140 based on one or more IDs searched in the feature management server 20, and outputs them as search results to the search request source.
  • the search condition may further include filter information as an additional condition. Examples of the filter information include designation of a search DB (corresponding to a feature space to be searched), designation of the number of searches, and the like. For example, when the number of searches is specified, the feature space management unit 101 sets the similarity of each search result (similarity indicating the similarity between each search result and the feature amount of the conversion data (entity) of the search object). Based on this, a predetermined upper number of search results may be output to the search request source.
  • the modality management unit 102 manages modalities. For example, as shown in FIG. 2, the modality management unit 102 can perform the following processing. ⁇ Get Modalities (): string [] ⁇ Register Modality (modality: string, definer: Modality Definer) Create (obj: object, modality: string): entity
  • the modality management unit 102 registers the modality definition (data), or the modality definition unit 103 inputs the object (registered object) input from the feature space management unit 101 to the modality of the object.
  • the data is converted into a predetermined data format according to the definition, and converted data (entity) is generated.
  • the entity is shaped data that can be directly passed to the feature extraction unit 202.
  • the modality definition unit 103 assigns a unique ID common to the plurality of feature spaces to the generated entity, and outputs the entity and ID to the feature space management unit 101.
  • the modality definition unit 103 stores the object (registered object) and the ID assigned to the entity of the registered object in the storage unit 140 in association with each other.
  • Such an ID is a unique character string across a plurality of feature spaces that handle the same modality.
  • a hash value or the like may be used so that the same value is returned for the same entity.
  • Modality definition data includes definition data in a format that can be received as input data (obj) (for example, file name (string), OpenCV Mat format) (multiple) and output data (entity) format definition data (Only one) (for example, char [3] [256] [256]).
  • Modality definition data exists, for example, for each data format or for each pattern described above (for example, conversion between formats, unification of formats, reading of special data, etc.), and is stored in the storage unit 140.
  • the modality management unit 102 converts the registration object (input data) using the definition data of the modality, and outputs the conversion data (entity).
  • the modality definition unit 103 saves the data corresponding to the ID as necessary or confirms that the data is saved, the function of retrieving the data corresponding to the ID, and deletes the data corresponding to the ID. Has the function of The modality definition unit 103 exists for each modality.
  • the input unit 110 has a function of receiving user instruction content such as an operation input unit that receives an operation instruction from the user and a voice input unit that receives a voice instruction from the user, and outputs the instruction content to the control unit 100.
  • the operation input unit may be a touch sensor, a pressure sensor, or a proximity sensor.
  • the input unit 110 may be a physical configuration such as a button, a switch, and a lever.
  • the communication unit 120 is connected to an external device by wire or wireless, and transmits / receives data to / from the external device.
  • the communication unit 120 may be a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE: Long Term Evolution, 3G (third generation mobile communication)).
  • LAN local area network
  • Wi-Fi registered trademark
  • Bluetooth registered trademark
  • LTE Long Term Evolution
  • 3G third generation mobile communication
  • the output unit 130 has a function of presenting (outputting) information to the user, such as a display unit and an audio output unit.
  • the output unit 130 outputs a search screen or outputs a search result under the control of the control unit 100.
  • the storage unit 140 is realized by a ROM (Read Only Memory) that stores programs used in the processing of the control unit 100, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • the storage unit 140 stores feature space management information, modality definition information, and an object (substance data) assigned with a unique ID.
  • each process by the control unit 100 of the information processing apparatus 10 may be executed by a plurality of apparatuses or may be executed by a server on the network.
  • the feature management server 20 includes a control unit 200, a communication unit 210, and a feature amount database 220.
  • the feature management server 20 since the feature management server 20 has one feature extraction unit 202, it can be said that each feature management server 20 manages one feature space, but the present disclosure is limited to this. For example, if the feature management server 20 includes a plurality of feature extraction units 202, a plurality of feature spaces can be managed.
  • Control unit 200 The control unit 200 functions as an arithmetic processing unit and a control unit, and controls the overall operation in the feature management server 20 according to various programs.
  • the control unit 200 is realized by an electronic circuit such as a CPU (Central Processing Unit) or a microprocessor, for example.
  • the control unit 200 may include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.
  • ROM Read Only Memory
  • RAM Random Access Memory
  • control unit 200 also functions as the feature management unit 201.
  • the feature management unit 201 performs feature extraction (for example, substitution processing for an N-dimensional vector) on the entity transmitted from the information processing apparatus 10 by using the feature extraction unit 202, and sets the extracted feature amount as the ID of the entity.
  • a process of registering in the feature amount database 220 in association with each other is performed.
  • An existing technique can be used for the feature quantity extraction algorithm, and is not particularly limited here.
  • the feature extraction unit 202 can handle different modalities in a unified manner.
  • the feature quantity extracted by the feature extraction unit 202 is registered in the feature quantity database 220.
  • the feature quantity database 220 exists for each modality.
  • the feature management unit 201 registers the feature amount extracted by the feature extraction unit 202 in the feature amount database 220 corresponding to the modality of the original data (entity transmitted from the information processing apparatus 10) in association with the ID.
  • a feature extraction unit 202a color feature feature space a
  • the feature amount extracted by the feature extraction unit 202a is stored in the feature amount database 220-1 corresponding to the handwritten data when extracted from the handwritten data, and the feature corresponding to the image data when extracted from the image data. It is stored in the quantity database 220-2.
  • the feature management unit 201 can also perform a search process (similarity search) using a feature space in response to a request from the feature space management unit 101. Since the feature quantity database 220 exists for each modality, the feature management unit 201 may perform a similarity search using the feature quantity database 220 corresponding to the target modality (search target modality). For example, the feature management unit 201 can perform the following processing as shown in FIG. ⁇ Get Space ID (): string ⁇ Register Database (modality: string, database: Feature Database) ⁇ Get Vector (entity: object, modality: string): vector Add (id: string, entity: object, modality: string) ⁇ Most Similar (query: object, modality: string, target Modality: string): Search Result []
  • the communication unit 210 is connected to an external device by wire or wireless, and transmits / receives data to / from the external device.
  • the communication unit 210 may be a wired / wireless local area network (LAN), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE: Long Term Evolution, 3G (third generation mobile communication).
  • LAN local area network
  • Wi-Fi registered trademark
  • Bluetooth registered trademark
  • LTE Long Term Evolution
  • 3G third generation mobile communication
  • the feature quantity database 220 accumulates the feature quantities extracted by the feature extraction unit 202. Each feature amount is associated with a unique ID assigned by the modality management unit 102. As described above, the feature amount database 220 exists for each modality.
  • the feature amount database 220 is stored in a storage unit (not shown) included in the feature management server 20.
  • the storage unit of the feature management server 20 is realized by a ROM that stores programs and calculation parameters used for the processing of the control unit 200, and a RAM that temporarily stores parameters that change as appropriate.
  • the configuration of the feature management server 20 according to the present embodiment has been specifically described above.
  • the configuration of the feature management server 20 shown in FIG. 1 is an example, and the present embodiment is not limited to this.
  • at least a part of the configuration of the feature management server 20 may be in an external device.
  • FIG. 3 shows an example of another configuration example of the information processing system according to this embodiment.
  • the feature extraction unit 240 and the feature quantity database 250 may be managed by separate servers (the feature management server 24 and the database server 25), respectively.
  • FIG. 4 is a sequence diagram illustrating an example of a flow of object registration processing in the information processing system according to the present embodiment.
  • the feature space management unit 101 of the information processing apparatus 10 acquires a registration request based on a user operation input or the like (step S103), and the object (obj) included in the registration request Along with the information on the modality (mdl) of the object, a request for generation (conversion data (entity)) is made to the modality management unit 102 (step S106).
  • the modality management unit 102 performs conversion data (entity) generation, unique ID assignment, and obj and unique ID storage processing by the modality definition unit 103 (step S109).
  • the modality definition unit 103 performs processing (common preprocessing) for converting an object into data of a predetermined format in accordance with the definition of the modality.
  • Specific examples of the processing include the following examples. In the case of a still image: JPEG data is converted into char [3] [256] [256] (multidimensional array) and smoothing processing is performed.
  • Handwriting Read the point sequence data and draw it with a white line of thickness 3 on a black image of char [3] [256] [256].
  • the feature space management unit 101 acquires at least an ID and an entity from the modality management unit 102 (step S112). Further, the modality management unit 102 may notify that the ID and obj are saved.
  • the feature space management unit 101 outputs a data addition (registration) request to all corresponding feature spaces (Feature Space) based on the acquired ID and entity (step S115).
  • the addition request includes ID, entity, and modality (mdl).
  • the corresponding feature space is a feature space (feature management server 20) that can handle the modality of the entity.
  • the processes shown in steps S115 to S121 are repeated for each feature space.
  • the feature management unit 201 extracts feature amounts using the feature extraction unit 202 (step S118).
  • the feature management unit 201 adds (registers) the extracted feature quantity to the feature quantity database 220 together with the acquired unique ID (step S121). At this time, the feature management unit 201 registers the feature amount database 220 corresponding to the modality of the extraction source entity.
  • the registration process according to this embodiment has been specifically described above.
  • the modality management unit 102 performs a predetermined conversion process for each modality so that the same data is preprocessed by a different process for each feature extractor. Therefore, the convenience of a system that handles a plurality of feature spaces can be improved. Further, managing the modality management unit 102 and the feature extraction unit 202 individually reduces the responsibility of each function (for example, it becomes easier to identify the cause at the time of an error).
  • the search system is completed by developing only the feature extractor, and the availability of the system increases.
  • the search DB that is, the feature database 220
  • only the ID and the feature are registered, and the original data (data entity) is separately managed by the modality definition unit 103. Therefore, the same data is registered in a plurality of databases. Can be avoided.
  • assigning unique IDs across multiple feature spaces to the same data multiple IDs can be associated with the same data, the same ID can be associated with multiple data, etc. Can be avoided.
  • the feature management unit 201 may register the feature data in the feature amount database 220 only when the original data is stored in the modality management unit 102. This ensures that the feature space management unit 101 can extract the original data from the ID when acquiring a search result to be described later.
  • the feature space (search DB system) development side can develop the feature extraction unit 202 without worrying about the input format. .
  • the registration process according to the present embodiment is not limited to the example shown in FIG.
  • step S115 described above it has been described that the addition instruction is given to the feature space (feature management server 20) corresponding to the modality.
  • the present embodiment is not limited to this, and the feature space management unit 101 can execute all the features.
  • An additional instruction may be given to the space (feature management server 20).
  • the feature space (feature management server 20) can determine whether the entity is a processable entity based on the modality.
  • FIG. 5 is a sequence diagram showing an example of the flow of search processing in the information processing system according to the present embodiment.
  • the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on a user operation input or the like (step S133).
  • the search request includes an object (obj), a modality (mdl1) of the object, and a target modality (mdl2) indicating the modality to be searched.
  • the feature space management unit 101 makes a generation request (conversion data (entity)) to the modality management unit 102 together with information on the object (obj) and the modality (mdl1) of the object included in the search request. (Step S136).
  • the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity) and assign a unique ID (step S139). Specifically, the modality definition unit 103 performs processing (common preprocessing) for converting an object into data of a predetermined format in accordance with the definition of the modality.
  • the feature space management unit 101 acquires the ID and entity from the modality management unit 102 (step S142).
  • the feature space management unit 101 outputs a search request to all the corresponding feature spaces (Feature Space) based on the acquired entity (step S145).
  • the search request includes entity, mdl1 (original data modality), and mdl2 (target modality).
  • the corresponding feature space is a feature space (feature management server 20) that can handle mdl1 and mdl2.
  • the processing shown in steps S145 to S157 is repeated for each feature space.
  • the feature management unit 201 uses the feature extraction unit 202 to extract feature amounts (step S148).
  • the feature management unit 201 searches for a similar feature amount from the feature amount database 220 based on the extracted feature amount (step S151). At this time, the feature management unit 201 searches the feature amount database 220 corresponding to the requested target modality (mdl2). In the feature quantity database 220, the unique ID is associated with the feature quantity, and the feature management unit 201 searches the feature quantity database 220 for a feature quantity similar to the feature quantity of the requested entity, and is similar. The ID associated with the feature quantity and the similarity of the feature quantity: sim (similarity with the feature quantity of the requested entity, for example, the distance of the N-dimensional vector) is acquired. If there are a plurality of target modalities (mdl2), the processes shown in steps S151 to S154 are repeated for each feature quantity database 220.
  • the feature space management unit 101 acquires a search result (searched feature value ID, searched modality: mdl, and searched feature value similarity: sim) from the feature management unit 201 (step S157).
  • the search result may include a plurality of IDs, mdl, and sim.
  • the feature space management unit 101 specifies, for example, an ID of a feature amount having the highest similarity.
  • the feature space management unit 101 specifies the ID of the upper predetermined number of feature amounts based on, for example, the similarity.
  • the feature space management unit 101 makes a request for original data to the modality management unit 102 together with the identified ID and the corresponding modality information (step S160).
  • the modality management unit 102 acquires the original data (that is, the object) associated with the ID by the modality definition unit 103 (step S163) and outputs it to the feature space management unit 101 (step S166).
  • the processing shown in steps S160 to S166 can be performed for the number of search results to be output.
  • the feature space management unit 101 deletes the record (ID, mdl, sim).
  • the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S169).
  • the feature space management unit 101 may display a screen showing the search result on the output unit 130 and present it to the user.
  • the search processing according to this embodiment has been specifically described above.
  • the feature space constructed according to the present embodiment can be used for cross-sectional search in a plurality of feature spaces that handle different modalities.
  • the feature space (search DB) used for the search may be specified.
  • the identified feature space is included in the search request in step S113 as a space ID. For example, “I want to use a search DB created by XX company”, “I want to use a XX search site”, or the like is assumed.
  • FIG. 6 shows an example of a search screen in the present embodiment.
  • the search screen 30 shown in FIG. 6 is presented by the output unit 130, for example.
  • the user inputs a search object 301 and selects a search target (corresponding to a modality. For example, “photograph”, “illustration”, “document”, etc.), and the feature quantity of what is similar is searched for.
  • Select for example, “shape”, “color”, “meaning”, etc., and correspond to each feature space. For example, feature space constructed based on shape features, constructed based on color features
  • an object similar to the search object 301 acquired as a search result is presented.
  • the search object 301 is obtained as the search result.
  • an illustration having a similar shape, color, and / or meaning for example, a “modality: illustration” feature quantity database possessed by the feature management server 20 that handles a feature space constructed based on the feature of the shape. 220) and presented.
  • the feature amount search condition and / or may be arbitrarily selected by the user, or or may be set as a default.
  • the modality definition unit 103 may define a parent-child relationship (inclusion relationship) between modalities. Specific examples include the following.
  • modalities For example, use cases that define modalities including existing modalities are assumed. More specifically, it is assumed that a modality “text” already exists and there is a feature space A (feature extraction unit 202A) that handles “text”.
  • a modality of “mail” including text (body) and a user (sender) and a feature space B (feature extraction unit 202B) that can handle “mail” are added.
  • the first effect is that “can be registered in a plurality of feature spaces at the same time”. That is, since the text can be acquired from the mail, the same ID and object pair can be registered not only in the feature space B but also in the feature space A.
  • the feature space when the focus is on only the text is stored in the feature space A, and the feature space when the focus is on the text and the user is stored in the feature space B.
  • a search can be performed across other texts (in this case, it is necessary to assign a unique ID across all modalities, not just the same modality).
  • FIG. 7 is a diagram illustrating modularization of the feature extractor.
  • each modality using each feature extractor that can handle modalities such as texts and inclusive relations defined with mail and modalities, respectively.
  • a mail feature amount including a sentence feature amount (contents) and a user feature amount (sender).
  • FIG. 8 is a sequence diagram illustrating an example of registration processing of the first application example according to the present embodiment.
  • step S203 when the feature space management unit 101 of the information processing apparatus 10 acquires a registration request based on a user operation input or the like (step S203), the object (obj) included in the registration request and the object Along with information on the modality (mdl) of the object (for example, “Mail”), a request for generation of (entity) is made to the modality management unit 102 (step S206).
  • the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity), assign a unique ID, and store obj and a unique ID, and have an inclusion relationship with the modality of obj.
  • conversion data entity
  • sub entity is generated (step S209). For example, when the modality of the object is “Mail” and the modality (sub mdl) having an inclusion relation with this is “Text”, the modality definition unit 103 includes the mail data (obj) according to the definition of “Text”. Converts text data into a predetermined data format and generates a sub entity.
  • the feature space management unit 101 acquires at least ID, entity, and sub entity from the modality management unit 102 (step S212). Further, the modality management unit 102 may notify that the ID and obj are saved.
  • the feature space management unit 101 outputs a data addition (registration) request to all corresponding feature spaces (eg, feature space B) based on the acquired ID and entity (eg, Mail Entity) ( Step S215).
  • the subsequent processing related to feature amount extraction shown in steps S218 to S221 is the same as that in steps S118 to S121 shown in FIG. 4 and will not be described in detail.
  • the feature quantity (Mail Vector) is registered.
  • the Get Vector (Step S227) of the feature space A handling the text described below is used. It may be.
  • the feature space management unit 101 outputs a data addition (registration) request for all corresponding feature spaces (eg, feature space A) for the same ID and sub entity (eg, Text Entity) (steps S224 to 230).
  • the feature space A is a feature space corresponding to text only, and the feature amount (Text Vector) of the text extracted from the Text Entity is registered.
  • a feature space having an inclusion relationship can be used, and a feature quantity can be registered in the feature space.
  • FIG. 9 is a sequence diagram illustrating an example of search processing of the first application example according to the present embodiment.
  • the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on a user operation input or the like (step S243).
  • the search request includes an object (obj), a modality (mdl1) of the object, and a target modality (mdl2) indicating the modality to be searched.
  • mdl1 Mail
  • mdl2 Text.
  • the feature space management unit 101 requests the modality management unit 102 to generate (entity) together with information on the object (obj) and the modality (mdl1) of the object included in the search request (step S246). .
  • the modality management unit 102 uses the modality definition unit 103 to generate conversion data (entity) and assign a unique ID, and based on the definition of the modality (sub mdl) having an inclusion relation with the modality of obj. Then, the sub entity is generated (step S249). For example, when the modality of the object is “Mail” and the modality (sub mdl) having an inclusion relation with this is “Text”, the modality definition unit 103 includes the mail data (obj) according to the definition of “Text”. Converts text data into a predetermined data format and generates a sub entity.
  • the feature space management unit 101 acquires ID, entity, and sub entity from the modality management unit 102 (step S252).
  • the feature space management unit 101 outputs a search request to all corresponding feature spaces (Feature Space) based on the acquired entity (for example, Mail Entity) (step S255).
  • the search request includes entity, mdl1 (original data modality, for example, Mail), and mdl2 (target modality, for example, Text).
  • the corresponding feature space is a feature space that can handle mdl1 and mdl2 (for example, a feature space corresponding to both mail and text).
  • steps S258 to S267 are the same as steps S148 to S157 shown in FIG.
  • the feature space management unit 101 outputs a search request to all corresponding feature spaces (Feature Space) based on the ID and sub entity (for example, Text Entity) (step S270).
  • the search request includes sub entity (eg, Text Entity), mdl1 (sub mdl, eg, Text), and mdl2 (target modality, eg, Text).
  • the corresponding feature space is a feature space that can handle mdl1 and mdl2, and here mdl1 and mdl2 are the same “Text”, and therefore feature space A corresponding to text corresponds.
  • a search is performed in the feature space A (steps S273 to S279), and the feature space management unit 101 acquires a search result from the feature management unit 201 (step S282).
  • the feature space management unit 101 sends a request for original data to the modality management unit 102 together with the acquired ID and the corresponding modality (for example, Text), as in steps S160 to S169 shown in FIG.
  • the object acquired based on the ID by the modality definition unit 103 is output from the modality management unit 102 to the feature space management unit 101 (step S291).
  • the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S294).
  • the feature space management unit 101 can output the final search result to the search request source after re-evaluating the search result based on the similarity and weighting of the search result from each feature extractor.
  • the weighting is, for example, weighting of the feature space. Such weighting can be arbitrarily set by a search request source (for example, a user).
  • FIG. 10 is a diagram illustrating an example of a search screen in this application example.
  • the search screen 32 includes a search object 321, a search target selection region 322, a region 323 for selecting the feature amount of what is similar to search, and a search button 326. It is displayed.
  • the weight of the selected feature amount can be set by operating the slide bar 324. For example, when it is desired to prioritize “color feature” among “shape feature” and “color feature”, the operation unit 325 of the slide bar 324 is moved toward “color feature”. Thereby, for example, the weight (w) is set as follows on the system side.
  • w (weights) ⁇ space1: 0.8, space2: 0.2 ⁇
  • search results giving priority to “color features” are displayed.
  • the setting of the weighting of the feature space is not limited to the example illustrated in FIG. 10.
  • the weighting may be set based on what is selected by the user from the search results, and the search results may be presented again.
  • An example is shown in FIG.
  • the system outputs the illustration 341 as the search result, assuming that the illustration 341 is close to the user's intention.
  • the weighting may be set so as to prioritize the feature space (feature extractor, that is, the feature extraction unit 202), and the search result may be presented again.
  • FIG. 12 is a sequence diagram illustrating an example of search processing of the second application example.
  • the feature space management unit 101 of the information processing apparatus 10 acquires a search request based on a user operation input or the like (step S303).
  • the search request includes an object (obj), a modality (mdl1) of the object, a target modality (mdl2) indicating a modality to be searched, and a weight (w) of the feature space.
  • step S315 to S321 a search process similar to the process shown in steps S145 to 157 of FIG. 5 described above is performed, and thus detailed description thereof is omitted here.
  • step S318 processing similar to that shown in steps S148 to S154 in FIG. 5 is performed, but detailed illustration is omitted.
  • the object A as a search result is weighted to the first feature space (sim (space1): 0.9) with the similarity (sim (space1): 0.9) when searched from the first feature space (space1).
  • a value obtained by adding the values (sim (new): 0.78) is calculated as a new similarity. This is because an ID associated with the same data may be registered in a plurality of feature spaces.
  • it is assumed that the search result is searched only from one feature space.
  • the second feature space has a similarity (sim (space2): 0.9) when the object C is searched from the second feature space (space2).
  • a value (sim (new): 0.18) obtained by multiplying the weight (space2: 0.2) is calculated as a new similarity.
  • the feature space management unit 101 specifies, for example, a predetermined number of search results (IDs) on the basis of the new similarity.
  • the feature space management unit 101 makes a request for original data to the modality management unit 102 together with the identified ID and the corresponding modality, similarly to steps S160 to S169 shown in FIG. 5 described above (step S327).
  • the object acquired based on the ID by the modality definition unit 103 (step S330) is output from the modality management unit 102 to the feature space management unit 101 (step S333).
  • the feature space management unit 101 outputs the search result (object, modality, and similarity) to the search request source (step S336).
  • the suggestion system will meet the demand according to the usage status of these multiple applications.
  • Content Web page, text, image, etc.
  • FIG. 13 is a functional block diagram showing an example of the configuration of the present system.
  • a suggestion system can be realized by an information processing apparatus 10x.
  • the information processing apparatus 10x functions as one or more applications 105, an information collection unit 106, a suggestion unit 107, a feature space management unit 101x, and a modality management unit 102x. These can be implemented by the control unit 100 of the information processing apparatus 10.
  • the application 105 is various application programs such as a web browser, a map application, and a notebook application.
  • the information collection unit 106 has a function of monitoring the operation of each application 105 and collecting and storing user operation information (that is, application usage status) in each application 105.
  • the information collecting unit 106 may use an OS (Operating System).
  • the suggestion unit 107 generates a search request based on the operation information collected by the information collection unit 106, and makes a search request to the feature space management unit 101x. For example, the suggestion unit 107, based on the request of the modality (mdl1) and content (obj) of the content being browsed / edited acquired from each application 105 from the information collection unit 106, and the modality (mdl2) of the required content, A search request may be generated. For example, the following examples of content modality acquired from each application 105 and required content modality requests are assumed. -Web browser ... Browsing: Web page, Request: Web page / Map application ... Browsing: Address, Request: None * Note app ... Editing: Text / Image, Request: Text / Image
  • the feature space management unit 101x performs a search process using one or more feature spaces in response to a request from the suggestion unit 107.
  • the search process is the same as in the above-described embodiment.
  • the feature space management unit 101x acquires the entity obtained by converting the obj by the modality management unit 102x, and the entity, mdl1 (for example, Web page, address, text), and mdl2
  • mdl1 for example, Web page, address, text
  • mdl2 A search request is made to the feature management server 20 based on (for example, Web page, image).
  • the feature space management unit 101x outputs the search result to the suggestion unit 107.
  • the modality management unit 102x has the same function as that of the modality management unit 102 described with reference to FIG. 1, and the modality definition unit 103 performs processing for converting obj into a predetermined data format according to the definition of the modality of mdl1.
  • the generated entity is output to the feature space management unit 101x.
  • FIG. 14 is a sequence diagram showing an example of the flow of search processing in the suggestion system of this application example.
  • step S403 when one or more applications 105 are operated by the user (step S403), the content being handled (post; obj, mdl1) and the required content request (request) Mdl2) is performed on the information collecting unit 106 (step S406).
  • the post for example, a web page of “Kinkakuji”, an address “Kita-ku, Kyoto ... 1-2-3”, a travel-related text, and the like can be cited.
  • examples of the request include a web page and an image.
  • the information collection unit 106 outputs the collected information (post, request) to the suggestion unit 107 (step S412).
  • the suggestion unit 107 makes a search request to the feature space management unit 101x (step S415).
  • the search request includes the content included in post as obj, the modality thereof as mdl1, and the content modality included in request as mdl2.
  • step S4108 a search process is executed in the feature space management unit 101x (step S418).
  • step S4108 processing similar to that in steps S136 to S166 of FIG. 5 (generation of entity from obj and mdl1, search based on entity, mdl1, and mdl2, acquisition of object from search result ID) is performed. Detailed description is omitted.
  • the suggestion unit 107 acquires a search result from the feature space management unit 101x (step S421).
  • the suggestion unit 107 may rank (re-evaluate) the search results according to the similarity and weighting (W) of the search results (step S424).
  • the suggestion unit 107 may set a weight as shown in Table 2 below for each input and output, and rank by multiplying the similarity. In this application example, such reevaluation may be skipped.
  • the suggestion unit 107 creates a display screen showing the search result (step S427) and presents it to the user (step S430).
  • the suggestion unit 107 may update the weighting (W) used in the above step 424 when the usage status feedback is obtained from the user (step S433).
  • suggestion unit 107 may suggest to the user and obtain feedback from the user via the application 105.
  • FIG. 15 shows an example of operation information and request information acquired from an application according to this application example.
  • the operation information as shown on the left in FIG. 15 and the request information as shown on the right in FIG. 15 are acquired from each application, and the requested information is suggested based on the operation information.
  • the information processing system can further improve the convenience of a system that handles a plurality of feature spaces.
  • a computer program for causing the information processing apparatus 10 or the feature management server 20 to perform the functions of the information processing apparatus 10 or the feature management server 20 on hardware such as a CPU, ROM, and RAM incorporated in the information processing apparatus 10 or the feature management server 20 described above.
  • a computer-readable storage medium storing the computer program is also provided.
  • this technique can also take the following structures.
  • An information processing apparatus comprising a control unit that performs the following.
  • the controller is A control for converting the search object included in the search request according to the definition of the modality of the search object, and generating conversion data for search; Control to output the search conversion data to the feature extractor corresponding to the modality of the search object and the target modality included in the search request;
  • the information processing apparatus according to (3), wherein a corresponding object is acquired from the storage unit based on the second identification information, and is output as a search result.
  • the controller is The degree of similarity indicating the degree of similarity of the feature is acquired from the feature extractor together with the second identification information associated with the feature similar to the feature extracted from the search conversion data, (4) The information processing apparatus described in 1. (6) The information processing apparatus according to (4) or (5), wherein the search request further includes filter information as a search condition.
  • the controller is When the registration request information is input, according to the definition of the sub-modality having a parent-child relationship with the modality of the registration object, control for converting the registration object to generate sub-conversion data for registration; Control for outputting the first identification information and the sub-transformed data to one or more feature extractors corresponding to the sub-modalities;
  • the information processing apparatus according to any one of (1) to (6), further performing: (8)
  • the controller is When the search request is input, sub-converted data for search is generated by converting data corresponding to the sub-modality among the search objects according to the definition of the sub-modality having a parent-child relationship with the modality of the search object.
  • Control Control to output the sub-transform data for search to one or more feature extractors corresponding to the sub-modality and the target modality;
  • the information processing apparatus according to any one of (3) to (6), further performing: (9)
  • the controller is Control for ranking the plurality of second identification information based on the second identification information and similarity obtained from the feature extractor based on the search request, and the weight of the feature extractor; A control for outputting the upper predetermined number of the second identification information as the search results;
  • the information processing apparatus according to any one of (4) to (6), further performing: (10)
  • the controller is Generating the search request for searching for content to be proposed to the user based on information including user operation information output from one or more applications;
  • the information processing apparatus according to any one of (4) to (6), wherein one or more pieces of the second identification information acquired from the feature extractor are output as the search result.
  • the controller is The content included in the information and handled by the application is the search object,
  • the modality of the content is the modality of the search object,
  • the information processing apparatus includes: The information according to any one of (1) to (11), further including a communication unit that transmits the first identification information and the conversion data for registration to a feature management server having the feature extractor. Processing equipment.
  • Processor Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units; Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration; Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality;
  • An information processing method including performing.
  • Computer Control for storing the registration object included in the registration request information in the storage unit in association with the unique first identification information common to the plurality of feature extraction units; Control for converting the registration object according to the definition of the modality of the registration object, and generating conversion data for registration; Control to output the first identification information and the conversion data for registration to a plurality of feature extractors corresponding to the modality; A program for functioning as a control unit for performing
  • control unit 101 101x feature space management unit 102, 102x modality management unit 103 modality definition unit 105 application 106 information collection unit 107 suggestion unit 110 input unit 120 communication unit DESCRIPTION OF SYMBOLS 130 Output part 140 Storage part 200 Control part 201 Feature management part 202 Feature extraction part 210 Communication part 220 Feature quantity database 240 Feature extraction part 250 Feature quantity database

Abstract

Le problème décrit par la présente invention est de fournir un dispositif de traitement d'informations capable d'améliorer davantage la commodité d'utilisation d'un système dans lequel une pluralité d'espaces de caractéristiques sont gérés ; un procédé de traitement d'informations ; et un programme. À cet effet, l'invention concerne un dispositif de traitement d'informations qui est pourvu d'une unité de commande qui exécute : une commande pour stocker un objet d'enregistrement, inclus dans des informations de demande d'enregistrement, en association avec des premières informations d'identification uniques qui sont communes dans une pluralité d'unités d'extraction de caractéristiques ; une commande pour convertir l'objet d'enregistrement selon une définition de la modalité de l'objet d'enregistrement et générer des données de conversion pour l'enregistrement ; et une commande pour délivrer en sortie les premières informations d'identification et les données de conversion pour un enregistrement à une pluralité d'extracteurs de caractéristiques correspondant à la modalité.
PCT/JP2019/004534 2018-03-16 2019-02-08 Dispositif de traitement d'informations, procédé de traitement d'informations et programme WO2019176398A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2020505682A JP7255585B2 (ja) 2018-03-16 2019-02-08 情報処理装置、情報処理方法、および、プログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018049304 2018-03-16
JP2018-049304 2018-03-16

Publications (1)

Publication Number Publication Date
WO2019176398A1 true WO2019176398A1 (fr) 2019-09-19

Family

ID=67907731

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/004534 WO2019176398A1 (fr) 2018-03-16 2019-02-08 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Country Status (2)

Country Link
JP (1) JP7255585B2 (fr)
WO (1) WO2019176398A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112905829A (zh) * 2021-03-25 2021-06-04 王芳 一种跨模态人工智能信息处理系统及检索方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004348706A (ja) * 2003-04-30 2004-12-09 Canon Inc 情報処理装置及び情報処理方法ならびに記憶媒体、プログラム
JP2006285612A (ja) * 2005-03-31 2006-10-19 Canon Inc 情報処理装置およびその方法
JP2006343850A (ja) * 2005-06-07 2006-12-21 Fuji Xerox Co Ltd 推薦情報提供システム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004348706A (ja) * 2003-04-30 2004-12-09 Canon Inc 情報処理装置及び情報処理方法ならびに記憶媒体、プログラム
JP2006285612A (ja) * 2005-03-31 2006-10-19 Canon Inc 情報処理装置およびその方法
JP2006343850A (ja) * 2005-06-07 2006-12-21 Fuji Xerox Co Ltd 推薦情報提供システム

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112905829A (zh) * 2021-03-25 2021-06-04 王芳 一种跨模态人工智能信息处理系统及检索方法

Also Published As

Publication number Publication date
JPWO2019176398A1 (ja) 2021-04-22
JP7255585B2 (ja) 2023-04-11

Similar Documents

Publication Publication Date Title
CN111753060B (zh) 信息检索方法、装置、设备及计算机可读存储介质
CN111125422B (zh) 一种图像分类方法、装置、电子设备及存储介质
US11030445B2 (en) Sorting and displaying digital notes on a digital whiteboard
US11899681B2 (en) Knowledge graph building method, electronic apparatus and non-transitory computer readable storage medium
CN110301117B (zh) 用于在会话中提供响应的方法和装置
CN110209897B (zh) 智能对话方法、装置、存储介质及设备
US8200695B2 (en) Database for uploading, storing, and retrieving similar documents
US10437868B2 (en) Providing images for search queries
JP6381775B2 (ja) 情報処理システム及び情報処理方法
JP2022002075A (ja) 情報推奨方法及び装置、並びに、電子機器、プログラム及びコンピュータ読み取り可能な記憶媒体
CN107491655B (zh) 基于机器学习的肝脏疾病信息智能咨询系统
CN110516096A (zh) 合成感知数字图像搜索
JP6033697B2 (ja) 画像評価装置
JP3220886B2 (ja) 文書検索方法および装置
CN106874397B (zh) 一种面向物联网设备的自动语义标注方法
US10191921B1 (en) System for expanding image search using attributes and associations
CN109948154B (zh) 一种基于邮箱名的人物获取及关系推荐系统和方法
JP5876396B2 (ja) 情報収集プログラム、情報収集方法および情報処理装置
WO2019176398A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et programme
KR20200075068A (ko) 감성사전 구축 방법 및 시스템
JP2019101944A (ja) 知的財産システム、知的財産支援方法および知的財産支援プログラム
JP6802332B1 (ja) 情報処理方法および情報処理装置
JP2019114308A (ja) 知的財産システム、知的財産支援方法および知的財産支援プログラム
Xu et al. Estimating similarity of rich internet pages using visual information
JP6531302B1 (ja) 知的財産システム、知的財産支援方法および知的財産支援プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19768066

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020505682

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19768066

Country of ref document: EP

Kind code of ref document: A1