JP7077585B2 - Information processing systems, information processing equipment and programs - Google Patents

Description

The present invention relates to an information processing system, an information processing device and a program.

Various electronic data such as video and audio are collected and stored in a database in various situations. For example, Patent Document 1 below discloses a moving image search device for searching a desired image by using search information corresponding to each of one or more scenes constituting moving image data. In the prior art shown in the same document, search information is associated with moving image data, and the search information is summary information in units of a plurality of spatially divided blocks obtained by spatially dividing the scene. include. The summary information for each spatial division block is numerical information indicating image characteristics calculated based on the moving image data in the spatial division block, and is given as a relative value in the scene. The moving image search device responds to an information management means for reading and managing the search information and a first search request given from the outside, and the information management means obtains summary information in units of spatial division blocks included in the search information. It includes a first scene search means for searching a scene that matches the first search request by using the summary information of the receiving space division block unit.

Japanese Patent No. 4027662

By the way, new ideas for things often come from a combination of fragmentary information. In addition, it is expected that such fragmentary information includes information other than the search results by the intentionally selected search key, thereby encouraging ideas that are not bound by stereotyped ideas. However, with the conventional data search method, it is not possible to extract a combination of fragmentary information from the accumulated data or present search results containing information that does not correspond to the intentionally selected search key. ..

It is an object of the present invention to provide a system for searching and presenting fragmentary information related to such information based on the acquired information.

The present invention according to claim 1 is
A database that stores fragmentary data obtained by decomposing the information based on the feature information extracted by analyzing at least the information including images and sounds.
An information acquisition means that continuously acquires information including at least images and sounds according to specific conditions .
A feature information extraction means that sequentially analyzes information continuously acquired by the information acquisition means to extract feature information, and a feature information extraction means.
A search means for searching the database by sequentially adding the feature information extracted by the feature information extraction means to the search key,
An output means for outputting the search result by the search means and
It is an information processing system characterized by being equipped with.
The present invention according to claim 2 is
The output means processes the fragment data obtained as a search result by the search means, and generates an information image based on the fragment data in a predetermined manner according to the fragment data and the feature information of the fragment data. The information processing system according to claim 1, wherein the information processing system is displayed on a display device.
The present invention according to claim 3 is
The output means is characterized in that each time a search is performed by the search means, the information image based on the fragment data of the search result is generated, and the generated information image is sequentially displayed together with the already displayed information image. The information processing system according to claim 2.
The present invention according to claim 4
The output means sets a starting point position on the screen of the display device, and when newly displaying the information image, the new information image is displayed at the starting point position, and the already displayed information image is displayed. The information processing system according to claim 3, wherein the information processing system is moved to a position farther from the starting point position than the current position.
The present invention according to claim 5 is
Further provided with a receiving means for accepting the selection of the information image displayed on the display device.
The information processing system according to claim 2, wherein the output means outputs the fragment data corresponding to the selected information image by the fragment data before processing.
The present invention according to claim 6 is
The information processing according to claim 5, wherein the search means searches for fragment data associated with the fragment data corresponding to the selected information image, and further outputs the obtained fragment data. It is a system.
The present invention according to claim 7 is
The information processing system according to claim 1 , wherein the search means is used as a search key by giving a larger weight to the feature information extracted later.
The present invention according to claim 8 is
The first aspect of claim 1 , wherein the search means starts searching the database using the feature information as a search key when the extracted feature information satisfies a predetermined start condition. It is an information processing system.
The present invention according to claim 9 is
The search means starts the search of the database on the condition that at least two of the information specifying the place, the information specifying the time, and the information specifying the person are extracted as the feature information. The information processing system according to claim 8 , wherein the information processing system is characterized by the above.
The present invention according to claim 10
Further, a data processing means for decomposing the information acquired by the information acquisition means based on the feature information extracted by the feature information extraction means to generate fragment data and storing the generated fragment data in the database. The information processing system according to claim 1, wherein the information processing system is provided.
The present invention according to claim 11
The information processing system according to claim 10 , wherein the data processing means is characterized in that a plurality of the fragment data generated by decomposing one piece of information are associated with each other and stored in the database.
The present invention according to claim 12
The feature information extracting means extracts, as the feature information, information that identifies a person associated with the acquired information.
The information processing system according to claim 11 , wherein the data processing means classifies the fragment data for each person extracted as the feature information and stores the fragment data in the database.
The present invention according to claim 13 is
At least the information including image and sound is continued according to specific conditions from the database storing the fragment data obtained by decomposing the information based on the feature information extracted by analyzing the information including at least image and sound. The acquisition means for sequentially acquiring the fragment data obtained by sequentially analyzing the information and extracting the feature information , adding the extracted feature information to the search key in sequence , and searching.
An image generation means that processes the fragment data based on the acquired feature information of the fragment data to generate an information image representing a portion related to the feature information.
A display means for displaying the information image generated by the image generation means in a predetermined manner according to the fragment data and the feature information of the fragment data .
It is an information processing apparatus characterized by being provided with.
The present invention according to claim 14
Each time the fragment data of the search result is acquired by the acquisition means, the display means sequentially displays the information image generated by the image generation means based on the fragment data together with the already displayed information image. The information processing apparatus according to claim 13 , wherein the information processing apparatus is displayed.
The present invention according to claim 15
The display means sets a starting point position on the screen, and when newly displaying the information image, the new information image is displayed at the starting point position, and the already displayed information image is displayed at a position higher than the current position. The information processing apparatus according to claim 14 , wherein the information processing apparatus is moved to a position far from the starting point position.
The present invention according to claim 16
13. The display means according to claim 13 , wherein the display means controls the display size of the information image according to the number of matched search keys for the fragment data detected by matching a plurality of search keys. Information processing device.
The present invention according to claim 17 is
The information processing apparatus according to claim 16 , wherein the display means controls the display magnification of a portion of the fragment data displayed on the information image according to the display size of the information image. ..
The present invention according to claim 18 is
The display means is characterized in that it performs a specific display specified for each person with respect to the display of the information image based on the fragment data detected by using the information indicating the relationship with the specific person as a search key. Item 13 is the information processing apparatus.
The present invention according to claim 19
Further provided with a receiving means for accepting the selection of the displayed information image,
The information processing apparatus according to claim 13 , wherein the display means displays contents based on the fragment data based on the fragment data corresponding to the selected information image.
The present invention according to claim 20
The acquisition means acquires fragment data related to the fragment data corresponding to the selected information image, and obtains the fragment data.
The information processing apparatus according to claim 19 , wherein the display means displays contents based on the acquired related fragment data.
The present invention according to claim 21
Computer,
At least the information including image and sound is continued according to specific conditions from the database storing the fragment data obtained by decomposing the information based on the feature information extracted by analyzing the information including at least image and sound. The acquisition means for sequentially acquiring the fragment data obtained by sequentially analyzing the information and extracting the feature information , adding the extracted feature information to the search key in sequence , and searching.
An image generation means that processes the fragment data based on the acquired feature information of the fragment data to generate an information image representing a portion related to the feature information.
As a display control means for displaying the information image generated by the image generation means on a display device in a predetermined manner according to the fragment data and the feature information of the fragment data .
It is a program characterized by making it function.

According to the first aspect of the present invention, a search key including information not intended by the user is included in real time based on continuously acquired information as compared with a configuration in which a search is performed using the search key intended by the user. You can search by .
According to the second aspect of the present invention, the user can be made to recognize the information contributing to the search as compared with the configuration in which the search result is simply output.
According to the third aspect of the present invention, the result of the continuously performed search can be referred to from a bird's-eye view as compared with the configuration in which the previous search result is replaced with the new search result each time the search is performed.
According to the invention of claim 4, it is possible to grasp the results of continuously performed searches in chronological order by comparing with a configuration in which the previous search results are replaced with new search results each time a search is performed. It's easy.
According to the fifth aspect of the present invention, the details of the information can be confirmed by specifying the information of the search result selected by the user as compared with the configuration in which the search result is simply output.
According to the sixth aspect of the present invention, it is possible to acquire information related to the search result of interest by the user as compared with a configuration in which the search result is simply output.
According to the invention of claim 7 , as compared with the configuration in which the search is performed using the search key intended by the user, among the information added with the passage of time, the new information is emphasized and the related information is searched. Can be done.
According to the eighth aspect of the present invention, the search conditions can be narrowed down as compared with the configuration in which the search is performed using the information obtained from the acquired information as the search key.
According to the invention of claim 9 , it is possible to narrow down the search conditions by any of place, time and person, as compared with the configuration in which the information obtained from the acquired information is used as the search key for the search.
According to the tenth aspect of the present invention, as compared with the configuration in which only the search is performed, the information acquired for obtaining the search key can be stored in the database and included in the information to be searched in a later search.
According to the invention of claim 11 , the acquired information is decomposed and the individual data to be searched is compared with the configuration in which the acquired information is simply stored in the database as it is or the configuration in which the acquired information is decomposed and stored in the database. Improvement of the search system because it is possible to remove unnecessary information and trace information that cannot be traced only by the feature information obtained from the information acquired in the search based on the relation that the same original information is decomposed. Can be planned.
According to the invention of claim 12 , a person related to the information of the search result can be identified by comparing with a configuration in which the acquired information is simply stored in the database as it is or a configuration in which the acquired information is decomposed and stored in the database.
According to the thirteenth aspect of the present invention, the information related to the acquired information is searched by the search key including the information not intended by the user, as compared with the configuration in which the search is performed by using the search key intended by the user. At the same time, it is possible to make the user recognize the information that contributed to the search.
According to the invention of claim 14 , it is possible to refer to the result of the continuously performed search from a bird's-eye view as compared with the configuration in which the previous search result is replaced with the new search result each time the search is performed.
According to the invention of claim 15 , it is possible to grasp the results of continuously performed searches in chronological order as compared with a configuration in which the previous search results are replaced with new search results each time a search is performed. It's easy.
According to the invention of claim 16 , it is possible to intuitively grasp the degree of matching for a plurality of search keys as compared with the configuration in which the search result is simply output.
According to the invention of claim 17 , it is possible to facilitate the recognition of the feature portion in the information which is the search result, as compared with the configuration which simply outputs the search result.
According to the invention of claim 18 , a person related to the information of the search result can be identified by comparing with a configuration in which the acquired information is simply stored in the database as it is or a configuration in which the acquired information is decomposed and stored in the database.
According to the invention of claim 19 , the details of the information can be confirmed by specifying the information of the search result selected by the user as compared with the configuration in which the search result is simply output.
According to the invention of claim 20 , it is possible to acquire the information related to the search result of interest by the user as compared with the configuration of simply outputting the search result.
According to the invention of claim 21 , the user intends to obtain information related to the acquired information in the computer in which the program of the present invention is installed, as compared with the configuration in which the search is performed using the search key intended by the user. It is possible to search by a search key that includes information that does not exist, and to make the user recognize the information that contributed to the search.

It is a figure which shows the whole structure of the information processing system by this embodiment. It is a figure which shows the functional structure of a server. It is a figure which shows the functional structure of a personal computer (PC). It is a figure which shows the hardware configuration example of the computer and the personal computer (PC) used as a server. It is a display screen of the display part of the personal computer (PC), and is the figure which shows the configuration example of the related information image display screen which displays the related information image. It is a figure which shows the control example of the related information image display screen by the display control part of a personal computer (PC), FIG. 6A is a figure which shows the state which only the origin image is displayed, and FIG. 6B is the origin image. FIG. 6C is a diagram showing a state in which the related information image is displayed at the position of, FIG. 6C is a diagram showing a state in which the related information image is separated from the starting point image, and FIG. 6D is the following related information at the position of the starting point image. A diagram showing a state in which an image is additionally displayed, FIG. 6 (E) is a diagram showing a state in which the next related information image is separated from the starting image, and FIG. 6 (F) is a diagram showing a state in which the related information image I-30 is additionally displayed. It is a figure which shows the state further away from the origin image SP. It is a figure which shows the example of the display content of the related information image. It is a figure which shows the example of switching the display of a screen by an operation with respect to a related information image. Related information It is a figure which shows another example of switching the display of a screen by an operation with respect to an image. 10 (A) is a diagram showing a display example of an information image based on related fragment data, FIG. 10 (A) shows a first display mode different from the display mode shown in FIG. 9, and FIG. 10 (B) shows a second display mode. It is a figure which shows the aspect. It is a figure which shows the state that the operation menu was displayed on the related information image display screen. It is a figure which shows the application example of the information processing system of this embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
<System configuration>
FIG. 1 is a diagram showing an overall configuration of an information processing system according to the present embodiment. The information processing system 100 includes a camera 111 as information collecting means, a microphone 112, and a sensor 113. Further, the information processing system 100 includes a server 120 as an information management means, a personal computer (PC) 130 as an information output means, and a display device 140.

The camera 111 is an image pickup device that acquires an image as information. The image acquired by the camera 111 is sent to the server 120 as digital data. The image acquired by the camera 111 includes both moving images and still images. In addition to the moving image data in the moving image format, a set of time-series still image data taken at regular time intervals may be acquired. The image acquired by the camera 111 is sent to the server 120 as digital data. As the camera 111, those having various configurations may be used depending on the image pickup target and the usage environment. For example, it may be a portable camera (handy camera) or a fixed camera. Further, a wide-angle camera or an all-around camera that acquires an image of a wide surrounding space, a group of a plurality of cameras that capture a specific object or place at various angles, or the like may be used. Further, paying attention to the data format of the acquired image, it may be a digital camera that acquires an image as a digital image, or an analog camera that acquires an image as an analog image. In the case of an analog camera, the image acquired as an analog image is converted into digital data and output. The specific configuration of the camera 111 in the information processing system 100 is specified according to the environment and scene in which the moving image is collected, the restrictions on the devices that can be used, and the like. Therefore, in a specific environment or scene, only one of a moving image and a still image may be acquired.

The microphone 112 is a recording device that records sound (acoustic information) as information. As the microphone 112, a microphone having various characteristics and configurations may be used depending on the place to be recorded, the type of acoustic information, and the like. For example, when recording the voice of a specific person or the operating sound of a specific object, a microphone 112 having a range characteristic matching the range of the voice or the operating sound and having high directivity is used. Further, when recording the voices and environmental sounds of the participants of the conference, the microphone 112 having low directivity or low directivity is used. The acoustic information recorded by the microphone 112 is sent to the server 120 as digital data. Further, as the configuration of the microphone 112 itself, various configurations such as a fixed microphone, a sound collecting microphone, a pin microphone worn by an individual, a microphone array having strong directivity, and the like can be adopted. The specific configuration of the microphone 112 in the information processing system 100 is specified according to the environment and scene in which recording is performed, restrictions on usable devices, and the like.

The sensor 113 is a detection device that detects various information as physical quantities from an object or the environment. As the sensor 113, various types may be used depending on the type of information to be detected. For example, a temperature sensor, a humidity sensor, an illuminance sensor, a speed sensor, a vibration sensor and the like are used alone or in combination. Then, various information such as environmental information such as room temperature and room brightness, biological information such as blood pressure and pulse, and information for specifying the position and operating state of the object are acquired by these sensors 113. The specific configuration of the sensor 113 in the information processing system 100 is specified according to the target for which information is to be acquired and the type of information to be acquired. The information acquired by the sensor 113 is sent to the server 120 as digital data.

The camera 111, the microphone 112, and the sensor 113 shown in FIG. 1 are merely examples of information collecting means. In the present embodiment, all of the camera 111, the microphone 112, and the sensor 113 may be used, or any one or two means may be used, depending on the scene in which the information is collected, the type of information to be collected, and the like. , Information is collected. Further, the camera 111, the microphone 112 and the sensor 113 continuously acquire information according to specific conditions. The specific conditions are the conditions for starting and ending the acquisition of information. Specifically, input of start instruction and end instruction, time condition and the like are set. When inputting an instruction is a condition, the camera 111, the microphone 112, and the sensor 113 receive the start instruction, start the operation of acquiring information (shooting, recording voice, etc.), receive the end instruction, and output the information. End the acquisition operation. Based on time conditions, the camera 111, microphone 112, and sensor 113 start the operation of acquiring information when the predetermined start time is reached, and when the predetermined end time is reached, the camera 111, the microphone 112, and the sensor 113 start the operation of acquiring information. Ends the operation to acquire information. Further, the operation may be ended when a predetermined time has elapsed from the start of the operation for acquiring information.

The server 120 is a device that manages information acquired by the camera 111, the microphone 112, and the sensor 113 that are information collecting means (hereinafter referred to as acquired information), and is an information processing device that searches for and outputs the accumulated information. be. The server 120 analyzes the acquired information, extracts the feature information, decomposes the acquired information based on the extracted feature information, and generates fragment data. Then, the server 120 stores the generated fragment data in the database and manages it based on the feature information. Further, the server 120 searches the fragment data stored in the database using the feature information extracted from the acquired information as a search key, and outputs the fragment data obtained as the search result. When the server 120 continuously acquires information by the camera 111, the microphone 112, and the sensor 113, the server 120 performs processing such as analysis of acquired information, accumulation of fragment data, search and output of fragment data, and the like. May be done continuously. Although one server 120 is shown in FIG. 1, the server 120 is not limited to a single server machine, and the functions of the server 120 are distributed to a plurality of server machines provided on the network. It may be realized (so-called cloud environment, etc.). Details of each function, feature information, and fragment data of the server 120 will be described later.

The PC 130 is an information processing device that displays an image based on the fragment data searched by the server 120 and accepts an operation on the displayed image. The PC 130 acquires the fragment data and the feature information of the fragment data from the server 120. Then, the PC 130 generates an information image based on the fragment data in a mode specified based on the acquired feature information, and displays the information image on the display unit (display). Further, the PC 130 accepts an operation on the information image displayed on the display unit. Then, the PC 130 reflects the content of the received operation on the display of the display unit, and makes a request to the server 120 according to the received operation. As the PC 130, for example, in addition to a desktop type or notebook type personal computer, a tablet type information terminal or a mobile information terminal such as a smartphone may be used.

The display device 140 is an output device that displays an image based on the fragment data searched by the server 120. The display device 140 acquires fragment data and feature information of fragment data from the server 120, and generates and displays an information image based on the fragment data in a mode specified based on the acquired feature information. As the display device 140, for example, in addition to various display devices such as a liquid crystal display, a projector or the like that displays an image on a screen may be used. Further, the display device 140 may be provided with a means for accepting the user's operation, and the contents of the accepted operation may be reflected in the display, or a request may be made to the server 120 in response to the accepted operation. For example, a touch panel integrated with the display screen may be provided as an operation means to accept intuitive operations such as touching the displayed information image to change the display.

The PC 130 and the display device 140 shown in FIG. 1 are merely examples of information output means. In the present embodiment, various forms of information output means may be configured depending on the situation, equipment, and the like in which the information based on the fragment data searched by the server 120 is to be used. For example, a PC 130 or a display device 140 may be prepared for each user, and each user may be configured to view an information image displayed on his / her own device, or one PC 130 or a display device 140 may be prepared and a plurality of them. The user may be configured to view one display screen. In a configuration in which a plurality of users browse one display screen, the operation means may be used as a touch panel so that each user can directly touch the screen for operation, or as an operation means prepared for each user. Each user may operate using an input device without touching the screen.

In the system configuration shown in FIG. 1, the camera 111, the microphone 112, the sensor 113 and the server 120, the server 120 and the PC 130, and the display device 140 are connected via a network, respectively. The network is not particularly limited as long as it is a communication network used for data communication between each device, and may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like. The communication line used for data communication may be a combination of these, regardless of whether it is wired or wireless. Further, a relay device such as a gateway device or a router may be used to connect each device via a plurality of networks or communication lines. As an example, a configuration is conceivable in which the camera 111, the microphone 112, the sensor 113, and the server 120 are connected by the Internet, and the server 120, the PC 130, and the display device 140 are connected by a LAN.

<Server function configuration>
FIG. 2 is a diagram showing a functional configuration of the server 120. The server 120 includes a data acquisition unit 121, an analysis unit 122, a data processing unit 123, and a database 124. Further, the server 120 includes a search unit 125, a data processing unit 126, and a transmission / reception unit 127.

The data acquisition unit 121 is an information acquisition unit that receives data of various information sent from the camera 111, the microphone 112, and the sensor 113 shown in FIG. The information acquired by the data acquisition unit 121 includes at least images and sounds. When the information is continuously acquired by the camera 111, the microphone 112, and the sensor 113, the data acquisition unit 121 also continuously receives the data sent from the camera 111, the microphone 112, and the sensor 113. Further, in the above example, the camera 111, the microphone 112, and the sensor 113 are operated according to specific conditions in order to continuously acquire information. On the other hand, the camera 111, the microphone 112, and the sensor 113 are configured to constantly acquire information and continuously transmit the data to the server 120, and the data acquisition unit 121 continuously receives the information according to a specific condition. May be.

The analysis unit 122 is a feature information extraction unit that analyzes the acquired information received by the information acquisition unit 121 and extracts the feature information. The feature information is information that characterizes the content itself or a part of the acquired information. As the feature information, information obtained from the contents of the acquired information and information obtained in relation to the acquired information are used. When the data acquisition unit 121 continuously acquires the information, the analysis unit 122 sequentially analyzes the continuously acquired acquisition information and extracts the feature information.

The information obtained from the content of the acquired information as the characteristic information differs depending on the type of the acquired information. Since various forms of information such as images and acoustic information are acquired as acquired information, the characteristic information obtained from the contents of the acquired information is also various, such as information that identifies an image, information that identifies a sound, and keywords that specify a concept. Information in various forms is extracted. For example, if the acquired information is an image, the subject (person, object), the location of the subject on the screen, the characters (text information) reflected on the screen, and the like can be extracted as feature information. This information is obtained by image analysis processing. Further, when the image is a moving image, the temporal position when the moving image is played back (for example, how many minutes have passed after the moving image is played back, the subject appears on the screen, etc.) may be used as the feature information.

Further, if the acquired information is acoustic information, acoustic characteristics (magnitude, height, timbre (waveform)), position and direction of a sound source, changes in these, whether or not the sound is sound, and the like can be extracted as characteristic information. This information is obtained by acoustic analysis processing. Further, when the acoustic information is voice, the utterance content and the speaker information obtained by voice analysis may be used as characteristic information.

When the acquired information is information (sensor data) acquired by various sensors, the acquired value (sensor value), the range including the sensor value (whether or not it is within a predetermined range), The state of change in the sensor value can be extracted as feature information.

The information obtained in relation to the acquired information as the feature information is, for example, data attributes that are acquired information, content attributes, additional information, and other meta information. Specifically, the information that identifies the time when the acquired information was acquired, the information that identifies the place where the acquired information was acquired, the information that identifies the device used to acquire the acquired information, and the image that is the acquired information. Information that identifies the photographer and recording engineer of acoustic information, information on the manager of the acquired information and the person who has the authority to relate to the content of the acquired information, information on the title and theme given to the acquired information, etc. as feature information. Can be extracted.

The above information is merely an example of the feature information, and the information that can be used as the feature information is not limited to the above information. In an actual system, a part of the above information may be used as feature information based on a specific system configuration, operating conditions, etc., or information other than the above information may be extracted and used as feature information. good. The algorithm for extracting the feature information is not particularly limited. The feature information extracted from the acquired information and the algorithm for extracting the feature information can be added or changed according to changes in the system configuration and operating conditions.

The data processing unit 123 is a data processing unit that processes acquired information based on the feature information extracted by the analysis unit 122. The data processing unit 123 decomposes the acquired information based on the feature information and generates fragment data. Usually, a plurality of fragment data are generated from one acquired information. The fragment data is data of partial content including features shown in one or a plurality of feature information obtained by decomposing the content of the acquired information. The partial content is extracted by various conditions and methods according to the type and attribute of the content of the acquired information. To give a specific example, if the acquired information is an image, for example, for each person or object specified as a subject, an image in a certain area in which the subject is captured may be cut out as partial content. Further, when the image is a moving image, for example, scenes in which the same subject or the same place is shown may be collected and used as partial contents. Further, if the acquired information is acoustic information, among the recorded sounds, sounds emitted from the same sound source, uttered voices of the same speaker, and the like may be collected and used as partial contents. Further, the partial content may be cut out based on the time when the sound is emitted, the characteristics of the sound, the change pattern thereof, and the like. If the acquired information is sensor data, partial content may be cut out based on the time when the data was acquired, the range of the sensor value, the change pattern of the sensor value, and the like. The data for each partial content extracted from the acquired information in this way is fragment data.

The conditions and methods for extracting partial content from the acquired information are not particularly limited. A part having a feature shown in one feature information may be extracted as a partial content, or a part having a feature shown in a plurality of feature information may be extracted as a partial content. Further, the conditions and methods for extracting the partial content from the acquired information may be independent for each of the extracted partial contents. Therefore, a part of the content or data may be duplicated in a plurality of partial contents extracted from one acquired information. Further, the feature information used as a condition for extracting the partial content from the acquired information may be manually input by the user in addition to the information extracted from the acquired information by the analysis unit 122.

The data processing unit 123 stores the fragment data generated by decomposing the acquired information and the feature information used for extracting the partial contents to generate the fragment data in the database 124 in association with each other. By doing so, it is possible to specify the fragment data corresponding to the designated feature information by searching the database 124 by designating the feature information. Further, the data processing unit 123 stores a plurality of fragment data generated from one acquired information in the database 124 in association with each other. In this way, when one piece of data is specified in the database 124, it is possible to specify another piece of data generated from the same acquired information as the piece of data by tracing the association between the pieces of data. .. Further, when the data processing unit 123 generates fragment data using information for identifying a person associated with the acquired information as feature information, the data processing unit 123 classifies the fragment data for each person specified by the feature information and stores the fragment data in the database 124. Store. In this way, fragment data based on acquired information acquired in various situations and situations associated with each classified individual person can be managed in association with that person. The person associated with the acquired information is a person related to the acquired information in a manner that can be extracted as characteristic information. For example, the subject of the image, the speaker of the acoustic information, the person who is the target person for acquiring the biological information, the photographer of the image, the person who is the recording engineer of the acoustic information, the manager of the acquired information, and the authority related to the content of the acquired information. It is a person or the like as a person who has.

The database 124 is a storage means for storing fragment data generated by the data processing unit 123. The database 124 associates the fragment data generated by the data processing unit 123 with the feature information and other fragment data, and classifies and holds each person used as the feature information. The database 124 is a part of the configuration of the server 120, but as shown in FIG. 2, it may be realized by a database server (so-called cloud server or the like) provided on the network.

The search unit 125 is a search means for searching fragment data stored in the database 124 using the feature information extracted by the analysis unit 122 as a search key. The search unit 125 acquires the feature information extracted by the analysis unit 122. Then, when the acquired feature information satisfies a predetermined start condition, the search unit 125 starts searching the database 124 using the acquired feature information as a search key. Since various forms of information such as image-specific information, sound-specific information, and keywords are extracted as feature information, the search unit 125 performs a search by combining these various information as a search key.

The start condition can be set according to the situation or situation in which the information is collected and searched by the present embodiment. For example, the starting condition may be that a specific type of feature information has been extracted. As an example, the starting condition may be that any two or three pieces of information that specifies a place, information that specifies a time, and information that identifies a person are extracted as characteristic information. In this way, when the search is started, the search will be performed with the location, time, related person, etc. included in the search key from the beginning, and when searching for fragment data related to a specific scene or situation, Efficient. Further, the starting condition may be that a predetermined number of feature information has been extracted. By doing so, the search is performed in a state where the search target is narrowed down to the fragment data corresponding to the plurality of search keys (feature information) from the beginning.

When the information is continuously acquired by the camera 111, the microphone 112, the sensor 113, and the data acquisition unit 121, the search unit 125 sequentially adds the feature information extracted by the analysis unit 122 to the search key to create a database. Repeat the search. As a result, as the information is continuously acquired by the camera 111, the microphone 112, and the sensor 113, the search unit 125 searches for the fragment data related to the acquired information in real time. In addition, by adding the search key based on the newly extracted feature information to the search key used before that, a search that reflects the search history up to that point (so to speak, a search that matches the context) can be performed. Will be done. Then, not only the fragment data corresponding only to the newly extracted feature information but also the fragment data corresponding to the feature information previously used as the search key can be obtained. In other words, not only the fragment data related only to the recently acquired acquired information but also the fragment data related to the acquired information acquired before that can be obtained. Here, when the newly extracted feature information is sequentially added to the search key, the feature information extracted later may be given a larger weight and used as the search key. As a result, fragment data that is more strongly related to the most recently acquired information can be obtained.

Further, the search unit 125 searches for other fragment data related to the specific fragment data based on the specific fragment data according to the search instruction described later. In this case, the search method by the search unit 125 may differ depending on the search instruction. As an example, the search unit 125 uses the related information attached when the fragment data is generated by the data processing unit 123, and identifies other fragment data generated from the same acquired information as the specific fragment data. Search for. Further, as another example, the search unit 125 performs a search using the feature information corresponding to the specific fragment data as a search key, and obtains other fragment data.

The data processing unit 126 is a processing means for processing the fragment data obtained by the search of the search unit 125 based on the feature information of the fragment data. The data processing unit 126 adds information necessary for generating a related information image (information image) based on the fragment data. The related information image is an image that displays partial content based on fragment data. The partial content constitutes a part of the content of the original acquired information from which the partial content is cut out, and specifically, is a still image, a moving image, acoustic information, sensor data, or the like. Therefore, depending on the type of partial content, it is necessary to generate an image to be displayed as a related information image or to add information to be additionally displayed. Further, depending on the type of partial content and the type of feature information, a related information image may be displayed in a special display mode. For example, when the partial content is a moving image, a sample image (thumbnail) obtained by cutting out a scene of the moving image may be displayed instead of displaying the moving image as it is. Further, when the partial content is acoustic information, it cannot be displayed as it is, so an image or text indicating that it is acoustic information may be displayed. Further, when the partial content is sensor data, the sensor value and its change may be expressed and displayed in a graph or text. Therefore, the data processing unit 126 processes the data necessary for performing such a display.

The transmission / reception unit 127 is a communication means for exchanging data with an external device (PC 130 and display device 140 in the configuration example shown in FIG. 1) which is an information output means. Specifically, the transmission / reception unit 127 transmits the fragment data processed by the data processing unit 126 to the PC 130 and the display device 140. Further, the transmission / reception unit 127 receives a search instruction from the PC 130 or the display device 140. The search instructions will be described later.

<Functional configuration of terminal device>
FIG. 3 is a diagram showing a functional configuration of the PC 130. In the configuration example shown in FIG. 1, the PC 130 and the display device 140 are provided as the information output means for outputting the search result by the server 120, but here, the PC 130 is used as an example of the external device which is the information output means. Explain the function. The PC 130 includes a transmission / reception unit 131, a display control unit 132, a display unit 133, an operation unit 134, and an operation reception unit 135.

The transmission / reception unit 131 is a communication means for exchanging data with the server 120. Specifically, the transmission / reception unit 131 receives fragment data from the server 120. Further, the transmission / reception unit 131 transmits a search instruction to the server 120 according to an operation on the related information image based on the fragment data.

The display control unit 132 is an image generation means for generating a related information image based on the fragment data acquired from the server 120, and is a display control means for displaying the generated related information image on the display unit 133. The display control unit 132 acquires fragment data and feature information from the server 120, and generates a related information image based on the acquired fragment data. Then, the display control unit 132 controls the display of the related information image on the display unit 133. The related information image is an image that presents the information of the partial content by the fragment data to the user of the PC 130. Related information The information of the partial content presented by the image can be set according to the type and content of the partial content. For example, when the partial content is a still image or a document file (including text data), a part of the image (hereinafter referred to as a content image) for displaying the partial content itself may be cut out and used as a related information image. Further, when the partial content is a moving image, a sample image (thumbnail) obtained by cutting out a scene of the moving image and an image of a play button for starting the reproduction of the moving image may be used as a related information image. Further, when the partial content is acoustic information, an image indicating that the partial content is acoustic information or an image of a play button for starting reproduction of the acoustic information may be used as a related information image. Further, when the partial content is an utterance voice, in addition to an image indicating that it is acoustic information and an image of a play button for starting playback of the acoustic information, a text indicating a part of the utterance content obtained by voice analysis. The image of may be used as a related information image.

Each time the display control unit 132 receives the fragment data from the server 120, the display control unit 132 sequentially displays the related information image generated based on the acquired fragment data on the display unit 133 together with the related information image already displayed. Further, the display control unit 132 controls the shape, color, display size of the related information image, the display position and movement of the related information image on the screen of the display unit 133, switching of the display mode, and the like. Further, the display control unit 132 controls the display of the content image according to the instruction received after displaying the related information image. The content image is, so to speak, an image based on fragment data before being processed by the data processing unit 126 of the server 120. The details of the display control regarding the related information image and the content image by the display control unit 132 will be described later.

The display unit 133 is a display means for displaying the related information image or the content image generated by the display control unit 132. The display unit 133 displays the related information image based on the feature information of the fragment data under the control of the display control unit 132. Further, the display unit 133 displays (content display) the content image of the partial content designated by the operation on the related information image under the control of the display control unit 132.

The operation unit 134 is a device for the user to perform an operation on the related information image displayed on the display unit 133. As the operation unit 134, for example, a pointing device such as a mouse, a touch panel, or the like is used. If a touch panel provided integrally with the display screen of the display unit 133 is used as the operation unit 134, the related information image can be operated with an intuitive operation such as directly touching the related information image displayed on the display screen. be able to.

The operation receiving unit 135 is a receiving means for receiving an operation on a related information image performed by using the operation unit 134. The operation receiving unit 135 accepts the selection of the related information image by the operation performed by using the operation unit 134 on the display screen of the display unit 133 on which the related information image is displayed. It also accepts various operations performed using the operation unit 134 on the selected related information image.

Although the functional configuration of the PC 130 has been described here, the display device 140 may have almost the same configuration. Specifically, the display device 140 is associated with a receiving unit (transmission / reception unit 131) that receives fragment data from the server 120, and a display control unit (display control unit 132) that controls the display of related information images and content images. A display unit (display unit 133) for displaying an information image and a content image is provided. The operation on the related information image may be performed using an input device (operation unit 134) as an external device connected to the display device 140. In this case, the connection means of the input device corresponds to the operation receiving unit 135. Further, when a touch panel provided integrally with the display screen of the display unit is used as the input device, the function corresponding to the operation unit 134 is also provided in the display device 140.

<Computer hardware configuration>
FIG. 4 is a diagram showing a hardware configuration example of a computer and a PC 130 used as a server 120. The computer 200 shown in FIG. 4 includes a CPU (Central Processing Unit) 201 which is a calculation means, a main storage device (main memory) 202 which is a storage means, and an external storage device 203. The CPU 201 reads the program stored in the external storage device 203 into the main storage device 202 and executes it. As the main storage device 202, for example, a RAM (Random Access Memory) is used. As the external storage device 203, for example, a magnetic disk device, an SSD (Solid State Drive), or the like is used. Further, the computer 200 includes a display mechanism 204 for displaying and outputting to the display device (display) 210, and an input device 205 for performing an input operation by the user of the computer 200. As the input device 205, for example, a keyboard, a mouse, or the like is used. The computer 200 also includes a network interface 206 for connecting to a network. The configuration of the computer 200 shown in FIG. 4 is only an example, and the computer used in the present embodiment is not limited to the configuration example of FIG. For example, the storage device may be configured to include a non-volatile memory such as a flash memory or a ROM (Read Only Memory).

When the server 120 shown in FIG. 2 is realized by the computer shown in FIG. 4, the data acquisition unit 121 and the transmission / reception unit 127 are realized by, for example, the network interface 206. Each function of the analysis unit 122, the data processing unit 123, the search unit 125, and the data processing unit 126 is realized, for example, by the CPU 201 executing a program. The database 124 is realized by, for example, a main storage device 202 or an external storage device 203. When a database server or the like on the network is used as the database 124, for example, the server 120 and the database 124 are connected via the network interface 206.

When the PC 130 shown in FIG. 3 is realized by the computer shown in FIG. 4, the transmission / reception unit 131 is realized by, for example, the network interface 206. The display control unit 132 is realized by, for example, a program-controlled CPU 201 and a display mechanism 204. The display unit 133 is realized by, for example, the display device 210. The operation unit 134 is realized by, for example, the input device 205. The operation receiving unit 135 is realized by, for example, a program-controlled CPU 201.

Although not particularly shown, when the display device 140 is used as the information output means, the display device 140, together with the display unit, includes a processor and a display mechanism for performing display control, and a network interface for receiving data from the server 120. Be prepared. In a configuration in which an external input device is used to operate a related information image or the like on the screen, the display device 140 includes an interface for connecting the input device. Further, the display device 140 itself may be configured to include an input device such as a touch panel.

<Display control of related information image>
FIG. 5 is a diagram showing a configuration example of a screen for displaying a related information image (hereinafter referred to as a related information image display screen), which is a display screen of the display unit 133 of the PC 130. In the example shown in FIG. 5, one starting point image SP and four related information images I-10 to I-40 are displayed on the related information image display screen. The starting point image SP is an image showing a position serving as a starting point when the related information images I-10 to I-40 are displayed. Further, the starting image SP is used as an operator for operating the display of the related information image display screen. The related information image I-10 is an example of the related information image when the partial content is a document file. The related information image I-20 is an example of the related information image when the partial content is a moving image. The related information image I-30 is an example of the related information image when the partial content is acoustic information. The related information image I-40 is an example of the related information image when the partial content is a still image. The upper limit of the number of related information images displayed on the related information image display screen is determined based on the size (width) of the related information image display screen and the size (size) of the related information image. It is not limited to the four shown in FIG. When the related information image is additionally displayed and the upper limit of the number that can be displayed on the related information image display screen is reached, each time the next related information image is displayed, the oldest related information image is deleted in order. You may control it as you go.

FIG. 6 is a diagram showing a control example of the related information image display screen by the display control unit 132 of the PC 130. 6 (A) is a diagram showing a state in which only the starting image SP is displayed, FIG. 6 (B) is a diagram showing a state in which the related information image I-10 is displayed at the position of the starting image SP, and FIG. 6 (C). ) Is a diagram showing a state in which the related information image I-10 is separated from the starting image SP, and FIG. 6D is a diagram showing a state in which the related information image I-20 is additionally displayed at the position of the starting image SP, FIG. (E) is a diagram showing a state in which the related information image I-20 is separated from the starting image SP, and FIG. 6 (F) is a diagram showing a state in which the related information image I-30 is additionally displayed and further separated from the starting image SP. Is.

As shown in FIG. 6, in the initial state of the related information image display screen, only the starting image SP is displayed (FIG. 6A). When the first fragment data is searched by the server 120, the related information image I-10 based on the fragment data is displayed at the position of the starting image SP (FIG. 6B). Next, the related information image I-10 moves to a position away from the starting image SP (FIG. 6 (C)). In this state, the operation by the operation unit 134 can be accepted for the related information image I-10. Next, the next fragment data is searched by the server 120, and the related information image I-20 based on the fragment data is displayed at the position of the starting image SP (FIG. 6D). Then, the related information image I-20 moves to a position away from the starting image SP (FIG. 6 (E)). More specifically, the related information image I-10 moves to a position farther from the starting image SP than the position shown in FIG. 6 (C), and the related information image I-20 has been moved to a position farther from the related information image I-10. Moves to the position where was displayed. In this state, the operation by the operation unit 134 can be accepted for the related information image I-10 and the related information image I-20. Similarly, when the next fragment data is searched by the server 120, the related information image I-30 based on the fragment data is displayed at the position of the origin image SP and moves to a position away from the origin image SP (FIG. 6). (F)). At this time, the related information image I-10 moves to a position farther from the starting point image SP than the position shown in FIG. 6C, and the related information image I-20 is moved to a position farther from the starting point image SP, and the related information image I-20 is moved to a position farther from the related information image I-10. Moves to the position where was displayed. Then, the related information image I-30 moves to the position where the related information image I-20 has been displayed until then.

In this way, every time the fragment data is searched for in the server 120, the related information image based on the searched fragment data is displayed, and bubbles appear one after another and move so as to float in the space. Here, each time a new related information image is displayed, the already displayed related information image moves to a position farther from the starting point image SP than the previous display position (current position). As a result, the related information image displayed earlier is displayed farther from the starting image SP, and the related information image displayed later is displayed closer to the starting image SP, so that the time series of the related information images is intuitive according to the order. Is grasped by.

FIG. 7 is a diagram showing an example of the display contents of the related information image. FIG. 7 shows an example of a related information image when the partial content is a document file. In the example shown in FIG. 7, a part of the content image when displaying the partial content C-1 itself, which is a document file, is cut out to be the related information image I-10. Here, in the content image of the partial content C-1, the portion cut out as the related information image I-10 is, for example, the portion corresponding to the feature information used for searching the fragment data of the partial content C-1. May be. To give a specific example, it is assumed that the database 124 is searched using a specific word as a search key (keyword), and the partial content C-1 which is a document file containing the word is detected. In this case, a portion of the detected partial content C-1 in which the word is actually described is cut out and displayed on the related information image I-10.

When the partial content is a still image (including an image in which sensor data is represented by a diagram such as a graph), the part corresponding to the feature information in the content image of the still image is displayed as in the case of the above document file. It may be cut out and used as a related information image. When the partial content is moving image or acoustic information, a sample image or an image showing the type of the partial content is used as the related information image instead of using a part of the content image as the related information image.

The display size of the related information image is set to reflect the feature information used for searching the fragment data of the partial content. The search unit 125 of the server 120 may perform a search by combining a plurality of feature information (search keys). Therefore, it may be controlled to display the related information image in a larger display size as the number of matching search keys increases in the search. In this case, the display magnification of the portion of the content image used as the related information image may be controlled according to the display size of the related information image. Specifically, when the related information image is displayed in a large display size, the displayed content image portion may also be enlarged and displayed. Further, the display magnification of the portion of the content image may not be changed, and more information (a wider area in the content image) may be displayed as the related information image.

<Operations for related information images>
When an operation is performed on the related information image displayed on the related information image display screen, the display control unit 132 switches the display of the screen according to the operation. Specifically, depending on the operation of selecting the related information image, the content image of the partial content shown in the selected related information image may be displayed, or related to the partial content shown in the selected related information image. Display information about other partial content.

FIG. 8 is a diagram showing an example of switching the display of the screen by operating the related information image. In the example shown in FIG. 8, it is assumed that the operation of selecting the related information image I-30 is performed on the related information image display screen. In the illustrated example, the touch panel was used as the operation unit 134, and the related information image I-30 was touched as an operation for selecting the related information image. Here, it is assumed that the partial content shown in the related information image I-30 is a moving image. In this case, under the control of the display control unit 132, the screen display is switched from the related information image display screen to the screen for displaying the content image for displaying the partial content. Here, since the partial content is a moving image, the operation panel C-2 of the reproduction software for reproducing the moving image content is displayed. The display screen of the content image has various forms depending on the type of partial content. For example, if the partial content of the selected related information image is a still image or a document file, the screen displays the partial content itself as an image. Further, if the partial content of the selected related information image is acoustic information, the operation panel of the reproduction software for reproducing the acoustic information is displayed as in the case of the moving image.

FIG. 9 is a diagram showing another example of switching the display of the screen by operating the related information image. In the example shown in FIG. 9, it is assumed that the touch panel is used as the operation unit 134 and the related information image I-30 is touched as an operation for selecting the related information image, as in FIG. However, here, as an operation different from the operation shown in FIG. 8, it is assumed that the related information image I-30 is touched (long touch) continuously for a certain period of time or longer. In this case, the display control unit 132 requests the server 120 for other fragment data (hereinafter, related fragment data) related to the fragment data of the partial content shown in the selected related information image. The search unit 125 of the server 120 searches the database 124 in response to this request, and returns the obtained related fragment data to the PC 130. The display control unit 132 of the PC 130 performs display based on the acquired related fragment data.

The display based on the related fragment data may be the same display as the related information image. However, a display is clearly shown that the image is associated with the related information image for which the selection operation has been performed. In the example shown in FIG. 9, five information images I-31 based on the related fragment data are displayed in contact with the selected related information image I-30. As for the information image based on the related fragment data, the display may be switched by the selection operation in the same manner as the related information image. The display mode of the information image based on the related fragment data is not limited to the display mode shown in FIG. 9, as long as the relationship with the selected related information image can be understood.

FIG. 10 is a diagram showing a display example of an information image based on the related fragment data. FIG. 10A shows a first display mode different from the display mode shown in FIG. 9, and FIG. 10B shows a second display mode. In the example shown in FIG. 10A, both images are shown to be related by arranging an information image based on the related fragment data in a ring shape around the related information image. In the example shown in FIG. 10B, it is shown that both images are related by connecting the related information image and the information image based on the related fragment data at an edge.

On the related information image display screen, in addition to accepting an operation on the related information image and performing display control as described above, an operation on the starting image SP may be received and display control may be performed. For example, when the selection operation for the origin image SP is performed, the new display and movement of the related information image is stopped, and when the selection operation for the origin image SP is performed again, the new display and movement of the related information image is restarted. Display control may be performed. Further, when the selection operation for the starting image SP is performed, the operation menu may be displayed so that instructions for various operations can be accepted.

FIG. 11 is a diagram showing how the operation menu is displayed on the related information image display screen. In the example shown in FIG. 11, a selection operation is performed on the starting image SP, and an operation menu M-1 for inputting an operation instruction on the related information image display screen is displayed. When the user selects an operation displayed on the operation menu M-1, the display control unit 132 executes various display controls based on the selected operation on the related information image display screen.

<Application example>
An application example of this embodiment will be described. The information processing system 100 of the present embodiment collects and accumulates information such as images, acoustic information, and sensor data in various situations, and also stores feature information obtained from the collected acquired information as a search key. Search and output fragmentary information. Therefore, as information such as images, acoustic information, and sensor data is acquired, fragmentary information associated with the acquired information by feature information is detected and presented to the user at any time. .. Further, the information search is performed not by the search key intentionally input by the user, but by the search key that combines various feature information extracted from the acquired information. Therefore, the information presented as the search result may include information that the user does not expect as related information.

Hereinafter, an application example of the information processing system 100 of the present embodiment will be described assuming a specific application situation. Here, as an example, a case where the information processing system 100 is applied to a conference will be considered. The applicable meeting is assumed as follows. Held regularly once a week. Participants are fixed, 3 people A, B, C, and the venue is unspecified.

FIG. 12 is a diagram showing an application example of the information processing system 100 of the present embodiment. The scene shown in FIG. 12 is a scene of the above-mentioned conference held at a certain time. Participants A, B, and C surround the table T, and an all-around camera (camera 111) for photographing the surroundings and a microphone 112 are placed on the table T. Further, a display device 140 is provided behind the participant A. Server 120 is not shown. When the conference is started, the camera 111 and the microphone 112 are switched on and information acquisition is started.

The camera 111 and the microphone 112 continuously acquire information during the conference. Therefore, as the conference progresses, the server 120 extracts the feature information from the acquired information one after another and searches for the fragment data. Then, related information images are generated one after another based on the searched fragment data and displayed on the display device 140. For example, it is assumed that one of the participants obtained the statement "the agenda spoken by Mr. B at the meeting last week" by the microphone 112 at the meeting. Then, keywords such as "last week's meeting", "participant B", and "agenda" are extracted as characteristic information from this acquired information. Then, by searching the database 124, fragment data associated with these keywords is detected, and a related information image is generated based on the detected fragment data and displayed on the display device 140.

Further, it is assumed that the participant C presents the material at the meeting and the material is photographed by the camera 111. Then, information such as an image of the material and text described in the material is extracted from this acquired information as feature information. Then, by searching the database 124, fragment data associated with these feature information is detected, and a related information image is generated based on the detected fragment data and displayed on the display device 140.

Fragment data is classified and stored in the database 124 for each person specified by the feature information. Therefore, when there is information for identifying a person as the feature information used as the search key, the person related to the fragment data obtained as the search result can be specified. Therefore, as in the example shown in FIG. 12, when the related information image is displayed in the scene where the information of a plurality of people is acquired, it may be possible to identify who the related information image is related to. For example, the frame of the related information image may be displayed with a different color for each related person. Further, a related information image having a different shape for each related person may be displayed. Furthermore, an image (thumbnail photograph, etc.) that allows the user to visually identify the related person is displayed in connection with the related information image or in the vicinity of the related information image, or information (name) that can identify the related person. Etc.) may be displayed.

In the example shown in FIG. 12, the related information image is displayed on the display device 140 that can be visually recognized by all the participants of the conference, and each participant refers to one related information image display screen. The usage method is not limited to the above example. For example, the related information image may be individually displayed on the display unit of the portable terminal possessed by each participant. As described above, the server 120 searches for fragment data in real time based on the information acquired during the conference as the conference progresses. Then, the related information image based on the obtained fragment data is displayed. Further, the server 120 decomposes the acquired information acquired as the conference progresses based on the extracted feature information, and stores it in the database 124 as fragment data.

By weighting the feature information used as the search key when searching the database 124, it is possible to influence the tendency of the fragment data obtained as the search result. For example, if a search is performed by giving a large weight to characteristic information that identifies an individual, it is expected that the rate of searching fragmentary data related to the identified individual will increase. On the other hand, if the feature information shared by a plurality of users such as place, time, scene, and agenda is given a large weight to the search, it is expected that the rate of searching the fragment data related to the plurality of users will increase. As shown in FIG. 12, when all the participants refer to the information searched in the conference, the latter type of search is performed to display the related information image including the information related to each participant. It is more likely that you will be sick. On the other hand, when displaying the related information image on each participant's portable terminal, the related information image different for each participant is performed by performing a search like the former and individually controlling the display of the related information image. Can be displayed.

In the above example, the application example of the information processing system 100 has been described assuming the situation of a meeting in which a plurality of people participate, but the information processing system 100 may be used in an event in which more people participate. good. Further, one user may browse the related information image while accumulating his / her own information in the database 124 by using his / her own information processing device.

Further, in the application example as shown in FIG. 12, the participants are provided with a sensor 113 for acquiring biological information such as pulse, blood pressure, and respiratory rate for a certain period of time, and the biological information of each participant during the meeting is acquired. It may be acquired as information. Then, by extracting the fluctuation of pulse, the increase of blood pressure, the increase of respiratory rate, etc. as characteristic information and adding it to the fragment data accumulated in the database 124, or by using it for searching the fragment data, the participants may be upset. Changes in emotions such as excitement can be reflected in the search for fragment data.

In the present embodiment, since the fragment data is searched using a search key that combines various feature information extracted from various acquired information, the user does not assume that the information presented as the search result is related information. It was stated that such information may be included. In addition to this, when searching for fragment data, fragment data having a low ratio of matching search keys (not so related to feature information) may be included in the search results at an appropriate ratio. By performing such control, it is possible to present information that is beyond the user's expectation.

Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the above embodiments. For example, in the above embodiment, the server 120 has the data processing unit 126, but the server 120 outputs the search result by the search unit 125 as it is, and the PC 130, which is an information output means, generates a related information image. It may be configured to perform the necessary processing. Further, the server 120 may be configured to control the display of the related information image, and the PC 130 or the display device 140 may be configured to display only the related information image according to the control of the server 120. In addition, various changes and alternative configurations that do not deviate from the scope of the technical idea of the present invention are included in the present invention.

100 ... Information processing system, 111 ... Camera, 112 ... Microphone, 113 ... Sensor, 120 ... Server, 121 ... Data acquisition unit, 122 ... Analysis unit, 123 ... Data processing unit, 124 ... Database, 125 ... Search unit, 126 ... Data processing unit 127 ... Transmission / reception unit, 130 ... Personal computer (PC), 131 ... Transmission / reception unit, 132 ... Display control unit 133 ... Display unit, 134 ... Operation unit, 135 ... Operation reception unit, 140 ... Display device, I-10 to I-40 ... Related information image, SP ... Starting point image

Claims (21)

A database that stores fragmentary data obtained by decomposing the information based on the feature information extracted by analyzing at least the information including images and sounds.
An information acquisition means that continuously acquires information including at least images and sounds according to specific conditions .
A feature information extraction means that sequentially analyzes information continuously acquired by the information acquisition means to extract feature information, and a feature information extraction means.
A search means for searching the database by sequentially adding the feature information extracted by the feature information extraction means to the search key,
An output means for outputting the search result by the search means and
An information processing system characterized by being equipped with. The output means processes the fragment data obtained as a search result by the search means, and generates an information image based on the fragment data in a predetermined manner according to the fragment data and the feature information of the fragment data. The information processing system according to claim 1, wherein the information processing system is displayed on a display device. The output means is characterized in that each time a search is performed by the search means, the information image based on the fragment data of the search result is generated, and the generated information image is sequentially displayed together with the already displayed information image. The information processing system according to claim 2. The output means sets a starting point position on the screen of the display device, and when newly displaying the information image, the new information image is displayed at the starting point position, and the already displayed information image is displayed. The information processing system according to claim 3, wherein the information processing system is moved to a position farther from the starting point position than the current position. Further provided with a receiving means for accepting the selection of the information image displayed on the display device.
The information processing system according to claim 2, wherein the output means outputs the fragment data corresponding to the selected information image by the fragment data before processing. The information processing according to claim 5, wherein the search means searches for fragment data associated with the fragment data corresponding to the selected information image, and further outputs the obtained fragment data. system. The information processing system according to claim 1 , wherein the search means is used as a search key by giving a larger weight to the feature information extracted later. The first aspect of claim 1 , wherein the search means starts searching the database using the feature information as a search key when the extracted feature information satisfies a predetermined start condition. Information processing system. The search means starts the search of the database on the condition that at least two of the information specifying the place, the information specifying the time, and the information specifying the person are extracted as the feature information. 8. The information processing system according to claim 8 . Further, a data processing means for decomposing the information acquired by the information acquisition means based on the feature information extracted by the feature information extraction means to generate fragment data and storing the generated fragment data in the database. The information processing system according to claim 1, wherein the information processing system is provided. The information processing system according to claim 10 , wherein the data processing means associates a plurality of the fragment data generated by decomposing one piece of information with each other and stores the fragment data in the database. The feature information extracting means extracts, as the feature information, information that identifies a person associated with the acquired information.
The information processing system according to claim 11 , wherein the data processing means classifies the fragment data for each person extracted as the feature information and stores the fragment data in the database. At least the information including image and sound is continued according to specific conditions from the database storing the fragment data obtained by decomposing the information based on the feature information extracted by analyzing the information including at least image and sound. The acquisition means for sequentially acquiring the fragment data obtained by sequentially analyzing the information and extracting the feature information , adding the extracted feature information to the search key in sequence , and searching.
An image generation means that processes the fragment data based on the acquired feature information of the fragment data to generate an information image representing a portion related to the feature information.
A display means for displaying the information image generated by the image generation means in a predetermined manner according to the fragment data and the feature information of the fragment data .
An information processing device characterized by being equipped with. Each time the fragment data of the search result is acquired by the acquisition means, the display means sequentially displays the information image generated by the image generation means based on the fragment data together with the already displayed information image. The information processing apparatus according to claim 13 , wherein the information processing apparatus is displayed. The display means sets a starting point position on the screen, and when newly displaying the information image, the new information image is displayed at the starting point position, and the already displayed information image is displayed at a position higher than the current position. The information processing apparatus according to claim 14 , wherein the information processing apparatus is moved to a position far from the starting point position. 13. The display means according to claim 13 , wherein the display means controls the display size of the information image according to the number of matched search keys for the fragment data detected by matching a plurality of search keys. Information processing equipment. The information processing apparatus according to claim 16 , wherein the display means controls the display magnification of a portion of the fragment data displayed on the information image according to the display size of the information image. The display means is characterized in that it performs a specific display specified for each person with respect to the display of the information image based on the fragment data detected by using the information indicating the relationship with the specific person as a search key. Item 13. The information processing apparatus according to item 13. Further provided with a receiving means for accepting the selection of the displayed information image,
The information processing apparatus according to claim 13 , wherein the display means displays contents based on the fragment data based on the fragment data corresponding to the selected information image. The acquisition means acquires fragment data related to the fragment data corresponding to the selected information image, and obtains the fragment data.
The information processing apparatus according to claim 19 , wherein the display means displays contents based on the acquired related fragment data. Computer,
At least the information including image and sound is continued according to specific conditions from the database storing the fragment data obtained by decomposing the information based on the feature information extracted by analyzing the information including at least image and sound. The acquisition means for sequentially acquiring the fragment data obtained by sequentially analyzing the information and extracting the feature information , adding the extracted feature information to the search key in sequence , and searching.
An image generation means that processes the fragment data based on the acquired feature information of the fragment data to generate an information image representing a portion related to the feature information.
As a display control means for displaying the information image generated by the image generation means on a display device in a predetermined manner according to the fragment data and the feature information of the fragment data .
A program characterized by making it work.

Images