JP5679220B2 - Information processing system - Google Patents

Description

  The present invention relates generally to an information processing system, and more particularly to an information processing system that performs evaluation based on the number of reference links to a database.

  With the recent spread of the Internet, the number of viewers and the amount of information on the network have increased, and information retrieval technology has progressed. For example, there are many portal sites that can use powerful search engines. In addition, a technology has been developed to analyze web pages and advertisements that suit the viewer's preference by analyzing the search keywords and access history of the viewer, and marketing based on the keywords used by the viewer has also been performed. It was.

  For example, there is an information providing apparatus that can provide information useful to a user accurately and easily (Patent Document 1). The information providing apparatus includes an access history storage unit that stores access frequency information indicating a frequency of access to each content by a user in association with user identification information for identifying the user, and an access tendency to the content among the users. The similarity between users that calculates the similarity between users expressing the similarity between the users based on the access frequency information stored in the access history storage means, and the similarity between users between the user and each user Content score calculation means for calculating a content score, which is information representing the usefulness of the content for the user, from the access frequency information of the user weighted by the content, and the content score calculation means calculated by the content score calculation means Associating the content score with the user identification information An index storage unit that stores information, a query input unit that receives an input of a query including user identification information transmitted from a communication terminal device, and acquires content identification information of content that conforms to the query received by the query input unit And providing information generating means for generating providing information from the acquired content identification information with reference to the content score stored in the index storage means in association with the user identification information included in the query, An information providing apparatus comprising: provision information output means for outputting the provision information generated by the provision information generation means to the communication terminal apparatus.

  In addition, there is a search server that presents the popularity of search keywords and the reason for the popularity (Patent Document 2). The search server includes a blog article collection unit that collects blog articles, a period determination unit that determines a target period for displaying an appearance frequency of a search keyword according to information received from a client, and an interval within each section determined from the target period. Appearance frequency counting means that counts the appearance frequency of search keywords for updated blog articles and generates time-series information of appearance frequency, peak section detection means that detects a peak section of appearance frequency, and within the peak section The related keyword extracting means for extracting the related keyword of the search keyword from the blog article updated to the above, and the graph information generating means for generating the graph information obtained by graphing the time series information and transmitting it to the client. A related keyword is displayed as a label of the peak section.

  Further, there is an advertisement distribution system in which an advertiser can easily grasp the advertising effect of a banner advertisement and provides a banner advertisement with a higher advertising effect for each user (Patent Document 3). This advertisement distribution system refers to a user profile of a user corresponding to a user terminal from among a plurality of advertisement data stored in advance. Then, referring to the user profile, advertisement data to be distributed to the user terminal is extracted. Next, the extracted advertisement data is distributed to the user terminal via the network, and the advertisement is displayed.

JP 2009-265754 A JP 2008-262301 A Japanese Patent Laid-Open No. 2002-236846

  However, the summation of the reference history and the access history in the conventional system only counts accesses to the entire page in units of the location (URL, etc.) of the web page. For example, for any part of the web page It was not possible to collect attention information down to details such as whether the viewer was interested.

  Further, in the market analysis based on the search keyword and the search query, the text processing such as counting the frequency of use of the keyword and calculating the degree of relevance between the keywords has been mainly performed. In addition, the technology for improving the distribution efficiency of advertisements is limited to the analysis of profile information such as user gender, age group, preferences, and fields of interest to extract and distribute highly relevant advertisements. It is desirable to analyze an object that the viewer specifically focused on (or did not focus on).

  Accordingly, an object of the present invention is to provide an information processing system that analyzes a specific object (the object) focused by a viewer and distributes information, advertisements, and the like related to the focused object. .

  An information processing system according to the present invention provides a plurality of objects for a specific object recognized in response to an image search request from a user based on a mother database (MDB) reconstructed from a database for configuring the object. The search request from the user is evaluated according to the scene, and the advertisement and / or service related to the target object can be sequentially provided to the user who has made the image search request according to the degree of attention.

  In addition, the information processing system according to the present invention makes it possible to provide statistical data generated from the degree of attention to a plurality of objects to a user and / or a third party.

  In addition, when there is an unregistered specific object recognition request in the mother database (MDB) reconstructed from the database for configuring the object, the information processing system according to the present invention has a plurality of users and the If an unknown object discovery is notified to the three parties and the object can be identified based on collective intelligence from the user and information provided by a third party, a detailed database related to the object is stored in the MDB. After registration, guidance is given to a plurality of users and third parties that the unknown object has been identified, and advertisements and / or services related to the target object can be sequentially provided to the user.

  In addition, the information processing system according to the present invention enables the user to change some attributes of the object according to the user's preference by using the MDB of the object recognized as the specific object. Can be provided to third parties as marketing information.

  In addition, the information processing system according to the present invention pre-registers an object that may be put on the market in advance in the MDB so that it can quickly respond to an image search request from a future user.

  In addition, when the specific object recognition request from the user in the scene changes to another object, the information processing system according to the present invention evaluates the degree of attention based on the transition history information, and based on the evaluation Statistical data can be provided to third parties as marketing information.

The information processing system according to the present invention has the following effects.
(1) For an object focused on a specific image (including still images and moving images; the same applies hereinafter) scene, the viewer is useful for the viewer based on the degree of attention (focus frequency and / or rank). An information processing system that provides additional information can be realized.
(2) It is possible to realize an information processing system that additionally displays a display of advertisements and / or services related to a target object based on the degree of attention of an object focused on in a specific image scene by a viewer.
(3) It is possible to provide an information processing system that performs marketing on the basis of a change in the degree of interest of a viewer with respect to individual objects in a specific scene such as an image.

It is a system configuration figure in one embodiment of an information processing system concerning the present invention. It is explanatory drawing explaining the functional block of the server and mother database (MDB) in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the functional block of the terminal device in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the data flow in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the data flow of ADB, MDB, and a marketing information generation part in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the attention object in the scene in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining ADB in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the analysis example (generation example of marketing data) of the attention object in the scene in other embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the operation | movement flow in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the operation | movement flow in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the operation | movement flow in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the operation | movement flow in one Embodiment of the information processing system which concerns on this invention. It is explanatory drawing explaining the operation | movement flow in one Embodiment of the information processing system which concerns on this invention.

  Hereinafter, the form for implementing an information processing system to the present invention is explained in full detail.

FIG. 1 shows a system configuration diagram in an embodiment of an information processing system according to the present invention.
The information processing system 100 includes an information processing server 101, an MDB 102, and terminal devices 105a to 105d used by a user. The server 101 and the MDB 102 are connected by a connection 103. The server 101 and the terminal device group 105 are connected to a network. 104 is connected.

  The server in this specification is a computer system that provides some service in response to a request from a client, means a computer group in which a plurality of server systems are combined, and is driven by a plurality of software. Therefore, in this specification, the significance of the server is positioned as described above.

  A computer as hardware is, as the most basic configuration, an electronic computer including an arithmetic logic unit, a control unit, a storage device, and an input / output device connected by an instruction bus and a data bus. Based on information (bit data) input from the input / output device via the input / output interface, arithmetic operations, logical operations, comparison operations, shift operations, and the like are executed in the arithmetic logic unit. The executed data is stored in the storage device as necessary and output from the input / output device. A series of these processes is controlled by a software program stored in the storage device. Each server machine in the present invention is also hardware having at least the basic functions as a computer as described above, and is controlled by a program group such as an operating system, a device driver, and application software.

  FIG. 2 shows functional blocks of the information processing server 101 and the MDB 102 in an embodiment of the information processing system according to the present invention. The information processing server 101 includes, as software function blocks, an area processing unit 201, a general object recognition unit 202, a specific object recognition unit 203, a network communication control unit 204, a data search processing unit 205, and an MDB search unit 206. , An MDB learning unit 207, an MDB management unit 208, a scene processing unit 210, a marketing information generation unit 211, and an attention point processing unit 212. Although the functional blocks of the information processing server 101 are not necessarily limited to these, these representative functions will be briefly described.

  An area processing unit 201 divides an area in an image and extracts a partial image. The general object recognition unit 202 recognizes an object included in the image with a general name (category). In addition, the object recognition unit 203 collates with information registered in the MDB 102 and identifies an object. The network communication control unit 204 performs image input / output processing, information communication control with a terminal, and the like. The data search processing unit 205 collects information from link destinations, collects inquiries, collects, and searches. The MDB search unit 206 searches tag data such as the name of an object. The MDB learning unit 207 performs addition of detailed information regarding an object, registration of time information, registration, update, addition of incidental information, and the like. The MDB management unit 208 extracts feature points and feature amounts from object configuration data, extracts category information from incidental information and registers it in category data, and expands, divides, updates, integrates, corrects category classifications in category data, Register new categories.

  The scene processing unit 210 performs processing for recognizing the same scene in a still image or a moving image, and the marketing information generation unit 211 extracts a database (DB) structure (attention database) extracted from a degree of attention to a specific object, which will be described later. : Marketing information is generated based on the information of ADB). The point-of-interest processing unit 212 performs specific object recognition based on the MDB 102, processes reference to the target object, and stores the totaled result in the ADB 2101.

  The MDB 102 includes object configuration data 251, incidental information data 252, feature amount data 253, category data 254, and unspecified object data 255. Although the MDB 102 is not necessarily limited to these, these representative functions will be briefly described.

The object configuration data 251 is a database for configuring an object, and forms or manufactures an object such as the structure, shape, dimensions, connection information of parts, layout drawing, movable part, movable range, weight, rigidity, etc. It holds basic information necessary for.
The additional information data 252 holds all information related to the object such as the name of the object, the manufacturer, the part number, the date and time, the material, the composition, and the processing information.
The feature amount data 253 holds feature points and feature amount information of individual objects generated based on the above information.
The category data 254 holds information used when the general object recognition unit classifies an object.
The unspecified majority data 255 stores an object that has not been recognized by the specific object recognition unit. When a specific object is recognized by the evolution of MDB, it is removed.

  FIG. 3 shows a terminal device in an embodiment of the information processing system according to the present invention. The terminal devices 105a to 105d are client terminal devices widely used by users, and include a computer, a personal digital assistant (PDA), a mobile phone, and the like. That is, the terminal devices 105a to 105d represent a state in which many types of electronic information devices are connected to the network. Hereinafter, when it is described as the terminal device 105, it means one of the terminal devices 105a to 105d connected to the network. As a matter of course, the terminal devices 105a to 105d are not necessarily the same model. Any terminal device having an equivalent function (or a minimum function that can be implemented) may be used. Here, typical functional blocks of the terminal device 105 will be described.

  The terminal devices 105 a to 105 d include an input / output unit 301, a network connection unit 302, a CPU 303, a storage unit 304, and a power supply unit 305. The input / output unit 301 includes, for example, an input device such as a keyboard, a mouse, a pen, a jog dial, a joystick, a touch pad (including a built-in display), a microphone for voice recognition, and an output such as a display, a speaker, and a vibrator. Consists of devices. The network connection unit 302 is an interface for connecting to a network, and may be wired or wireless.

  The CPU 303 is a central processing unit, the storage unit 304 is a storage device such as a hard disk or a memory, and the power supply unit 305 is a battery or the like.

  In addition, a sensor for detecting tilt and acceleration, a GPS receiver, and the like can be mounted as necessary.

  FIG. 4 shows a data flow in an embodiment of the information processing system according to the present invention. In the information processing system according to the present invention, the user 401 browses the scene 402 using the terminal device 105, and inquires the information processing server 101 about the object focused on in the scene 402. This inquiry is made by instructing the object of interest in the scene 402 with a pointing device of the terminal device 105 (not shown).

Here, if the scene is two-dimensional, not only a single frame cut out by a still image such as a photo, but if a plurality of photos capture the same situation, the aggregate is the same scene. In the case of a moving image, a group of frames in which one continuous situation is recorded is the same scene. Even in the case of a moving image, when a plurality of moving images record the same scene, a set of the plurality of moving images similarly forms the same scene.
The scene is stored in, for example, a third party server (not shown) and can be viewed by a plurality of users.
For example, the user can make an inquiry about the object of interest to the information processing server 101 via the software according to the present invention incorporated in the terminal device 105.

  On the other hand, although not shown in FIG. 4, the information processing server 101 identifies a detectable object based on the MDB 102 from various still images and moving images existing on the network in advance, and receives information from the user. In response to the inquiry of the object recognition request, the recognition result of the specific object in the scene 402 can be promptly returned to the user. Similarly, in the case of a still image or a moving image uploaded and distributed in real time from the user and the third party 404, detection of a specific object is started in real time based on the MDB 102, and the user and the third party 404 detect the detection. Return results promptly. That is, the information processing server 101 attempts to observe which object in the scene the user has focused on by performing object recognition in response to an inquiry about the target object from the user for each scene. As a result, the third party 404 can provide the user with the transmission information 405 from the advertiser and the service provider.

  The concept of the scene assumed in the present invention covers a wide range. For example, assume a fashion show where many models appear. When shooting this fashion show from different angles with a plurality of cameras, the stage stage constitutes one scene, and the images from each camera constitute one view. The scene in the present invention assumes not only the view of each camera but also the former scene.

  In addition, a scene assumed by the present invention is reconstructed from a plurality of contents created by a user and uploaded to a server or a shared site, called UCG (user-generated content) or CGC (consumer generated content). The For example, a still image or a moving image of sports watching uploaded from a user to a shared site can also constitute the same scene. Alternatively, the same scene is reconstructed from the images of the surveillance camera taken from a plurality of angles.

  In utilizing scenes, not only the content described above, but also third parties 404, advertisers, etc. 405 prepare in advance still images and moving images that include objects that the user wants to see, and how much attention is given to the user. Sometimes it is observed.

The information processing server 101 in FIG. 4 includes a scene processing unit 210 that performs processing for recognizing the same scene in a still image or a moving image, and a point of interest that performs specific object recognition based on the MDB 102 and processes reference to the target object. A processing unit 212, an ADB 2101 for recording the aggregation results processed by the point of interest processing unit 212, and a marketing information generation unit 211 for generating marketing information to be described later based on the information recorded in the ADB 2101. And are included.
Note that “attention rank” means an index of interest by multiple users (for example, how many users have paid attention), and “attention frequency” means the frequency at which a specific user focuses on the same specific object. To do.

  Note that the information processing server 101 can also solve the unconfirmed object by inquiring to the collective intelligence or a third party about the unconfirmed object for which the specific object could not be identified (FIG. 4).

  Further, in order to obtain more accurate marketing information, a third party can provide object configuration data such as CAD for its own product to the information processing server 101 as specific object information (FIG. 4). Further, the information processing server 101 can provide the statistical data or marketing information generated by the marketing information generation unit 211 to the advertiser 405 or the third party 406 (FIG. 4).

  FIG. 5 illustrates the data flow of the ADB, MDB, and marketing information generation unit in an embodiment of the information processing system according to the present invention.

  In FIG. 5, the information processing server 101 manages the data structure of the recognized object in each scene in the ADB 2101 for each scene. For example, in FIG. 5, in the scene 1 (501), specific objects 501a and 501b are recognized. As a result of recognition, for each specific object, (1) category (general name level of the object), (2) specific object information (name and model of the specific product), (3) degree of interest, and (4) about the object of interest Transition information is stored in a predetermined data structure. Here, the transition information can be used to generate statistical data based on transitions within a category, transitions between categories, and transitions along a time axis as described later.

  As is apparent from the above description, the information processing server 101 has a feature of grasping a scene and analyzing the degree of attention of a specific object in the scene.

  FIG. 6 illustrates a specific example of scene and attention level analysis in an embodiment of the information processing system according to the present invention.

  FIG. 6A is a specific example of a scene. As shown in the figure, in the scene 402, the Champs-Elysees street where the Etoile Arc de Triomphe 601 can be seen in the distance is photographed, and countless cars are running on the Champs-Elysees street.

  The scene 402 is shared by a plurality of users, and the user can instruct an object of interest with a pointing device or the like of the terminal device 105 (not shown). Although the user can specify various objects in the scene 402, for the sake of simplicity, the Arc de Triomphe 601, the automobile 602, and the automobile 603 are taken as representative examples of the instructed objects, and what degree of attention is obtained respectively. Whether this has been done will be described with reference to FIG. 6B.

FIG. 6B is a specific example of ADB. First, as shown in the upper table of FIG. 6B, the scene 402 is a view to which 402 is assigned as an ID, and the type is a photo. And it constitutes one view of the scene (c1010) of “Champs Elysees where the Etoile Arc de Triomphe can be seen”. According to the upper table of FIG. 6B, in addition to the view identified by this ID 402, another view 403 (ID 403, type is movie) that constitutes the scene (c 1010) is registered. That is, since the content of ID403 (not shown) also belongs to the scene (c1010), a scene of “Champs Elysees where the Etoile Arc de Triomphe can be seen” is formed as in the case of ID402.
The upper table of FIG. 6B has the following data structure in one embodiment.

  In the “link to the target object” field, a link to one or more target objects focused in the scene (or view) is provided. In FIG. 6B, since the Arc de Triomphe 601, the automobile 602, and the automobile 603 are focused as described above, links to these are provided. The view 403 (contents not shown) also includes the Arc de Triomphe 601 and is linked to the Arc de Triomphe 601.

Next, the structure data collected for each object of interest is shown in the middle table of FIG. 6B. The object 601 focused on in the scene 402 is recognized as “Etoile Arc de Triomphe 601”, the focused object 602 is recognized as “vehicle type A (602)”, and the focused object 603 B (603) ". And the link to MDB about the recognized object and the attention degree of the object are recorded. In the middle table of FIG. 6B, the degree of attention of the Etoile Arc de Triomphe 601 is 3557, the degree of attention of the vehicle type A (602) is 15, and the degree of attention of the vehicle type B (603) is 135.
The degree of attention is calculated not only by the number of times of reference by the user but also appropriately weighted.
The middle table of FIG. 6B has the following data structure in one embodiment.

  In the “link to view” field, a link to a view of view 1 or higher where the target object exists is provided. In FIG. 6B, since the Arc de Triomphe 601 is included in the views 402 and 403 as described above, links to these views are provided.

The lower table in FIG. 6B shows a data structure for managing the time of interest for each scene, view, and object of interest. In the lower first row of FIG. 6B, it is recorded that the object 602 is noticed at 13:23:45 on January 5, 2010 in the view 402 of the scene C1010, and a link to the object 602 is provided. (Middle stage in FIG. 6B), it can be seen that this object is the vehicle type A of the automobile.
The lower table of FIG. 6B has the following data structure in one embodiment.

  In addition, although there is only one item for recording time in the lower part of FIG. 6B, instead of this, together with a user (identified by ID, IP address, etc.) that focuses on a plurality of histories of the time that has been focused on You may make it manage as a list.

  FIG. 7 illustrates an analysis example (marketing data generation example) of an object of interest in a scene in another embodiment of the information processing system according to the present invention. FIG. 7A shows scene statistical information generated from accumulation of the user's attention level for each object detected by the information processing system 100 for each scene. FIG. 7B shows a certain period (for example, 1 hour, 6 hours, 24 hours, 1 week, 1 month, 3 months, 6 months, 1 year, 10 years, etc.), and statistical information generated from a change in the degree of interest for each object in each scene. The generation of these statistical information will be described in detail together with the premise concept.

  Assume that the user A issues a specific object recognition instruction for the object 1 (the car 1) in one view (for example, a photograph) of a scene. In this case, the information processing system 100 identifies the object 1 (automobile 1) and stores information (for example, manufacturer, model name, year, color, model (for example, 2-door, 3-door, hatchback) stored in the MDB. , One box, etc.), displacement, etc.) and / or identifiable options (eg, sunroof, roof rack, fog lights, wheels, etc.) and link to information in the respective MDBs for objects in the scene Along with the degree of interest is recorded in the ADB as being related to the object 1. If the user further recognizes another object in the object 1 (automobile 1), for example, a specific object of the wheel N, the MDB information (for example, the manufacturer name, Record links to model number, size, year, retail price, material, etc.

  Next, when the same user A performs a specific object of another object 2 (automobile 2), specific object recognition is performed in the same manner, and detectable object information unique to the automobile is recorded as the object ADB2. Further, even when the user gives an instruction to specify an object other than a car, it is recorded in the ADB as the attention information of each object.

  Further, it is assumed that another user B recognizes a specific object of a certain car in a certain view of the same scene. In this case, the information processing system 100 identifies the automobile as described above. As a result, if the vehicle is already detected, the degree of attention of the object A in the registered ADB is updated. If it is a newly detected automobile, it is newly registered as an object 3 in the ADB. Further, when the same user instructs the specific object recognition of the wheel N of the vehicle, the attention degree of the wheel N is similarly updated.

  In this way, by repeating the above process, the information processing system 100 can accumulate the degree of interest of each user for each object detected for each scene, and generate statistical information for the scene (FIG. 7A). ).

  In parallel with the above processing, the information processing system 100 is based on a certain period (for example, 1 hour, 6 hours, 24 hours, 1 week, 1 month, 3 months, 6 months, 1 year, 10 years, etc.). Statistical information relating to a change in the degree of interest of the object in a period unit relating to the degree of interest of the object in each scene can be generated (FIG. 7B).

The above generation example is an example, and various other marketing data can be extracted.
For example, the following data can be generated.
1. The degree of attention in the category (for example, automobile) to which the object belongs. For example, the generation of data such as the model name, model, color, and year model of the top N attention level of users among the X cars in the scene.
2. Data generation such as the category, name, attention rank, and location of the object in the scene among the top N objects in all objects in the scene.
3. Data generation such as the category, name, and attention level of the object with the highest attention level for each view in the scene.

  FIG. 8 illustrates an operation flow (particularly, an object recognition and ADB generation flow in a scene) in an embodiment of the information processing system according to the present invention. First, when a steady process step in the information processing system 100 is started in S801, the process proceeds to S802, and data such as an image is input. For example, the data includes a plurality of images such as still images and moving images, and these data are arranged in time series. In step S803, a scene is cut out from the input data. Then, the view is cut out for each cut out scene.

  In step S804, an object is cut out from the cut out view using a segmentation technique or the like. In step S805, a specific object recognition process (see FIG. 12 described later) is performed.

  In S806, if a specific object is recognized (see, for example, FIG. 12), an ADB is generated and a link with the MDB is established. Next, the process proceeds to S807, where it is determined whether there is an image capable of recognizing a specific object in the view. If Yes, the process returns to S805 to repeat the subsequent processes.

  In the case of No in S807, that is, when the scene, view, and all specific objects in the scene are finally recognized, the process proceeds to S808 and the process is terminated. At this time, since the scene and view have a link to the MDB, it is possible to reconstruct them to create a new scene.

  FIG. 9 illustrates an operation flow (particularly, an ADB update and object reference history registration flow) in an embodiment of the information processing system according to the present invention. First, in step S <b> 901, processing of a point of interest designated by the user by a pointing device or the like of the mobile terminal 105 (not shown) is started. Next, the processing proceeds to S902, and category recognition by general object recognition and object identification processing by specific object recognition are performed on an area (object of interest) designated by the user in a view in the scene. In step S903, it is determined whether the object of interest has been recognized. If it cannot be recognized in S903 (No in S903), the process ends. If it can be recognized (Yes in S903), the process proceeds to step S904.

  In step S904, it is determined whether the recognized object has already been recognized in the scene. Specifically, the object object structure is searched and investigated. If it is an already detected object (Yes in S904), the process proceeds to S906. On the other hand, if it is a newly detected object (No in S904), the process proceeds to S905.

  In step S905, registration processing of object information in the ADB is performed. That is, since what is registered is a newly detected object, a link to the object ID, category, and MDB is registered in the target object structure in FIG. 6B, and a scene ID, a link to the view ID, and an image type (Photo or video) is registered.

  Next, proceeding to S906, the degree of attention of the ADB (FIG. 6B) regarding the object of interest is updated. In step S907, the reference history related to the target object is registered as a scene ID, view ID, time, and a link to the object in the ADB (FIG. 6B), and the process related to the target object ends (S908).

  FIG. 10 illustrates an operation flow (an advertisement selection flow based on ADB information) in an embodiment of the information processing system according to the present invention. First, when the process in the case where the user pays attention to an object in S1001 is started, the process proceeds to S1002, and the user pays attention to a specific object. For example, the object is clicked with the mouse or the object is surrounded. In step S1003, it is determined whether the object is already recognized by the information processing system. If it is recognized (Yes in S1003), the process proceeds to S1004 to obtain the ADB of the object. If not recognized (No in S1003), the process proceeds to S1005, and the process of FIG. 12 is executed to recognize the specific object. Next, proceeding to S1006, if a specific object is recognized in the processing of FIG. 12, an ADB is generated and a link with the MDB is established.

  Next, proceeding to S1007, the attention rank of ADB is updated. This value makes it possible to estimate how many people are interested in the object. In S1008, an advertisement and / or service appropriate for the user is selected from the degree of attention and transition information of the ADB. Then, the selected advertisement and / or service is presented to the user (S1009), and the process ends (S1010).

  FIG. 11 shows an operation flow (generation of marketing information based on the degree of attention and information related to the object (for example, scene, transition, location, time, etc.)) in an embodiment of the information processing system according to the present invention. explain. First, when processing is started in S1101, the process proceeds to S1102, and the user pays attention to a specific object. For example, the object is clicked with the mouse or the object is surrounded. In step S1103, it is determined whether the object is already recognized by the information processing system. If it is recognized (Yes in S1103), the object attention degree structure of the object is obtained (S1104). If not recognized (S1104), the process of FIG. 12 is executed to recognize the specific object (S1105). In S1106, if the specific object is recognized by the processing of FIG. 12, an ADB is generated and a link with the MDB is established. In S1107, the attention rank of ADB is updated. This value indicates how many people are interested in the object.

  Next, proceeding to S1108, marketing information is generated from the attention rank of the ADB, transition information, and information related to the object. Then, the selected marketing information is provided to the third party (S1109), and this step is finished (S1110).

  Based on FIG. 12, the flow of the object recognition using MDB201 is demonstrated.

[General object recognition]
The start of the image recognition process (S1201) starts with input of an original image in the terminal device 105 or scene segmentation by the information processing server 101 (S1202). It does not matter whether the original image is a two-dimensional image or a three-dimensional image. In addition, when inputting an original image, a target region of which object in the original image is instructed through a device (not shown) such as a pointing device, or the entire original image is input as a processing target without an instruction of a target point. (For example, scene extraction by the information processing server 101). Next, general object recognition processing is performed in S1204. For the general object recognition processing, for example, a BOF (Bag-Of-Features) method can be adopted. In the general object recognition process, the category of the detected object (general name of the object) is recognized. However, when the point of interest is instructed, the process branches depending on whether the category is recognized or not, and the determination is made in S1205. If category recognition cannot be performed, the process advances to step S1206 to determine handling of an existing category (S1207). If category recognition of an object can be performed regardless of whether or not there is an instruction at the point of interest, identification is performed. The process advances to step S1209 to advance to object recognition processing.

[Specific object recognition]
When the process proceeds to the specific object recognition process in the determination in S1205, first, an individual object image cut-out process is performed in S1209. Then, a specific object recognition process is performed on the cut out individual object image (S1210). In the specific object recognition process, identification of an object is attempted by an evaluation function that calculates a matching degree based on feature amount data extracted from object configuration data registered in the MDB 201.

[Learning process (1)]
On the other hand, if the general object cannot be recognized in the determination of S1205, the process proceeds to S1206, and the information distance between the feature amount of the object of interest and the feature amount of the object belonging to the existing category known by the MDB 201 is obtained. Based on this, a determination is made as to whether to register a new category including the target object (S1207) or to consider expanding an existing category close to the target object (S1208). When a new category is registered (S1207), the process returns to S1204, and when an existing category is expanded (S1208), the process proceeds to S1209.

[Learning after specific object recognition (2)]
In S1211, it is determined whether the specific object has been identified. If the specific object can be identified, the process proceeds to S1213, and it is determined whether the individual object image cut out in S1209 contains more detailed information than the detailed data of the object registered in the MDB 201. If YES is determined in step S1213, the process advances to step S1214, and the detailed data of the object in the MDB 201 is updated by the MDB learning unit to have more detailed information. On the other hand, if it is determined No in S1213, the process proceeds to S1215, and the next determination is made.

  S1215 is a case where it is determined in S1205 that the general object could not be recognized, and the process proceeds to S1208, S1209, and S1210 based on the determination in S1206, and is determined when the specific object is recognized (Yes in S1211). In S1215, if the identified object is an existing category, the definition of the existing category registered in the MDB 201 is expanded, or if the information distance of the objects in the category is dispersed by the extension, Or, when the information distance to the proximity category is less than or equal to the information distance between objects in the category, or when the information object of the existing object is found by registering the specified object The correction is made and the category data is updated (S1216). On the other hand, when the identified object is not an existing category in S1215, the process proceeds to S1207 and is registered as a new category.

  If the specific object cannot be recognized in S1211, the object is registered as an “unconfirmed object” in the MDB 201, and the recognition process is terminated in preparation for future processing (S1217). The recognition process is also terminated when the existing category is extended and updated in S1216 (S1217).

  The learning function of the MDB management 202 in the above description is appropriately reflected in the operation of the information processing system according to the present invention, thereby improving the specific object recognition ability.

[Other Examples]
In the information processing system according to the present invention, the following embodiments can be employed.

(1) With respect to an object that has been recognized as a specific object in the above-described embodiment, the user directly sets some attributes of the object in the scene (for example, color, surface processing, presence / absence of options, etc.) using the MDB for the object. It enables changes such as enlargement, viewpoint movement, and composition so that it can be picked up visually and interactively according to your preference. As a result, in response to mass production, it has not always been possible to launch products that correspond to various user preferences, so that changes according to user preferences, new specification requirement items, etc. can be made more detailed as marketing information. Collection becomes possible, and effective multi-product production can be realized.

(2) An object scheduled to be put on the market in advance is pre-registered in the MDB so that an image search request from the user can be promptly handled. For example, a third party such as a manufacturer registers a part of CAD data of a new product group in advance in the MDB and embeds it in a scene that can be shared by many users. After that, it is also possible to launch products with a high degree of attention into the market. On the other hand, for products that have not been noticed by the user for a long time, they are good judgment materials for considering design changes and sales arrangements.
It should be noted that it is desirable that the advance registration of an object that may be put on the market in advance in the MDB is quickly deregistered (deleted) from the MDB as a non-existing object when the object is not put on the market.

(3) Although the result of recognition of the specific object is stored in the ADB, the metadata (text) as the recognition result is embedded in a predetermined area of the original scene or view image (for example, JPEG image) format. This can also contribute to a conventional image search system.

(4) When there is an unregistered specific object recognition request in the MDB reconstructed from the database for constructing an object, notification of the discovery of an unknown object to a plurality of users and a third party for the object (attention) If the object can be discriminated based on the collective intelligence from the user and the information provided by the third party, the detailed database related to the object is stored in the MDB object configuration data 251 and / or the incidental information data 252. After the registration, the identification of the unknown object is notified to a plurality of user groups and third parties, and it becomes possible to sequentially provide advertisements and / or services related to the target object to the user groups.
The unknown object is stored in the unspecified object data 255 of the MDB for future analysis.
Note that the following object identification methods using collective intelligence are available. For example, it is performed by searching for encyclopedias on the Internet or automatically posting on a Q & A bulletin board. In order for the system to search for an encyclopedia on the net itself, a search query is created using a feature amount generated from the MDB together with a category obtained by general object recognition, and the search is executed. Then, a new feature amount is extracted from the returned contents to try again to identify the object. When automatically posting on the Q & A bulletin board, the original image is uploaded to the bulletin board together with the category obtained by the general object recognition. At this time, automatically edit the standard text prepared in advance, and “Please tell me the model about this ○○” or “Please tell me the website where the design information is published about this Δ △△” Post a query such as Then, from other users (including humans), “It is xx-xxxx” or “The design data of that △△△ is obtained from http://www.aaabbb.com/cad/data.dxf Advice is possible. " The system analyzes and evaluates these advices, accesses a specified URL, and tries to download object design data and the like. If the object is successfully identified based on the newly obtained design data, the obtained new data is added to the MDB, and the database is updated.

100 Information Processing System 101 Information Processing Server 102 Mother Database (MDB)
103 connection (network)
104 network 105a-105d terminal device

Claims (9)

A general object recognition unit and a specific object recognition unit,
General object recognition by the general object recognition unit and a specific object by the specific object recognition unit in response to an image search request from a user based on a mother database (MDB) including at least object configuration data, feature data, and category data Perform object recognition by at least one of the recognition, evaluate search requests from multiple users according to the scene for the recognized object,
Configured to at least one of advertising and services related to the object in accordance with the common interest of a particular user or a plurality of users obtained by the evaluation, sequentially provided to the user of performing the image retrieval request And
At the time of the object recognition, when the object can be identified without being able to recognize a general object, if the identified object belongs to an existing category registered in the MDB, at least the existing category An information processing system for updating the category data by extending a definition .   The information processing system according to claim 1, wherein statistical data generated from a degree of attention to at least one of a plurality of objects and an object common to a plurality of users is provided to at least one of a user and a third party. When there is an unrecognized object recognition request in the MDB, notification of an unknown object discovery (attention) to a plurality of users and third parties for the object,
When an object can be discriminated based on collective intelligence from the user and information provided from at least one of third parties, a detailed database related to the object is registered in the MDB, and then a plurality of users and third parties are registered. To the person that the unknown object has been identified,
The information processing system according to claim 1 or 2, wherein advertisements and / or services related to the target object are sequentially provided to the user. Using the MDB of the recognized object, the user changes some attributes of the object according to the user's preference,
The information processing system according to any one of claims 1 to 3, wherein the user's preference and request are provided to a third party as marketing information. Pre-register objects that may be put on the market in advance in the MDB,
The information processing system according to claim 1, which promptly responds to a future image search request from a user. When the object recognition request from the user in the scene transitions to another object, the degree of attention is evaluated based on the history information of the transition,
The information processing system according to any one of claims 1 to 5, wherein statistical data based on the evaluation is provided to a third party as marketing information. A general object recognition unit and a specific object recognition unit,
General object recognition by the general object recognition unit and a specific object by the specific object recognition unit in response to an image search request from a user based on a mother database (MDB) including at least object configuration data, feature data, and category data Perform object recognition by at least one of the recognition, evaluate search requests from multiple users according to the scene for the recognized object,
Configured to at least one of advertising and services related to the object in accordance with the common interest of a particular user or a plurality of users obtained by the evaluation, sequentially provided to the user of performing the image retrieval request And
At the time of the object recognition, when the object can be identified without being able to recognize a general object, if the identified object belongs to an existing category registered in the MDB, at least the existing category A server that updates the category data by extending the definition . In a server comprising a general object recognition unit and a specific object recognition unit,
General object recognition by the general object recognition unit and a specific object by the specific object recognition unit in response to an image search request from a user based on a mother database (MDB) including at least object configuration data, feature data, and category data Perform object recognition by at least one of the recognition, evaluate search requests from multiple users according to the scene for the recognized object,
Execution of sequentially providing at least one of advertisements and services related to the target object to the user who made the image search request according to the degree of attention common to a specific user or a plurality of users obtained by the evaluation then,
At the time of the object recognition, when the object can be identified without being able to recognize a general object, if the identified object belongs to an existing category registered in the MDB, at least the existing category A computer program for executing updating of the category data by extending a definition .   The computer-readable recording medium which recorded the computer program of Claim 8.