JP5638549B2 - Useful information presentation system and control method of useful information presentation system - Google Patents

Description

  The present invention relates to a useful information presentation system and a control method for the useful information presentation system.

  Conventionally, in a service that extracts useful information from a real-time video and presents it, a user terminal such as a smartphone recognizes one or more images or sounds constituting the real-time video (hereinafter, the object to be recognized is called a subject), It collates with useful information to be extracted (for example, Non-Patent Document 1). Conventionally, a method of extracting useful information from a figure having a specific format such as a two-dimensional barcode, QR code (registered trademark), AR marker, LLT marker, digital watermark, etc., a signboard or a menu written in a foreign language, etc. There has been a method of translating and displaying in the native language (for example, Non-Patent Document 2). There is also a method of acquiring information on the SNS of a subject from a face photograph and specifying or estimating the position of the subject or user terminal from a landmark image (for example, Non-Patent Document 3).

  Further, a system has been proposed in which a user terminal autonomously recognizes a subject that the user is not gazing at and alerts a dangerous object that approaches the subject (for example, Non-Patent Document 4). In addition, a system has been proposed in which a terminal device that consumes power, such as a wireless tag, is embedded in an environment, and information related to a user's surrounding environment is widely presented and guided (for example, Non-Patent Document 5).

Bruji, “BarcodeScanner”, [online], [Search January 17, 2012], Internet <http://www.bruji.com/cocoa/barcode.html> Apple, “Word Lens”, [online], [Search January 17, 2012], Internet <http://itunes.apple.com/app/word-lens/id383463868?ign-mpt=uo%3D5> remenbAR, “remenbAR”, [online], [searched on January 17, 2012], Internet <http://remembar.me/en.html> Mitsuhiro Okayasu, “Development and commercialization of smart white cane”, [online], [searched on January 17, 2012], Internet <http://www.rehab.go.jp/ri/event/at_cafe2010/report /cafe2_okyasu_slide.pdf> Tokyo, “The 8th Tokyo IC Tag Demonstration Experiment Committee Document”, [online], [searched on January 17, 2012], Internet <http://www.tokyo-ubinavi.jp/pdf/tokyo_ictag08 .pdf>

  However, since all of the conventional techniques perform the collation processing by the user terminal, the type and quantity of useful information extracted are limited by the processing capability of the user terminal.

  Conventionally, in order to use the service, the user has had to pay attention to the location of the predetermined subject in advance. Alternatively, the location of a predetermined subject is limited to simple collation that does not require the user to pay attention to it in advance. Moreover, when a terminal device that consumes power, such as a wireless tag, is embedded in the surrounding environment, there are problems in terms of cost and environmental load.

  Conventionally, the information presentation timing is the timing at which the presented information itself is generated, or the timing at which the user transitions to a predefined state (for example, the timing at which a predetermined time / position is reached). The mode of use has been limited. For example, when information is presented at the timing of the state transition of the user, the presented information is limited to information that can be managed in association with a schedule or map information.

  Therefore, the object of the present invention made in view of the above problems is not dependent on the processing capability of the user terminal, and the user terminal takes a picture without making the user aware of the location of the predetermined subject in advance. It is an object of the present invention to provide a useful information presentation system and a control method for the useful information presentation system that can present a wide range of useful information extracted from the real-time video, and can prevent oversight or oversight of useful information.

In order to solve the above problems, a useful information presentation system according to the present invention is:
A useful information presentation system comprising a user terminal and a server,
The user terminal captures a real-time video and transmits the real-time video to the server;
The server collates the real-time video and collation information to extract useful information, and sends an extraction completion notification to the user terminal,
The user terminal outputs the extraction completion notification, transmits an acquisition request for specific contents related to the useful information,
When the server receives the acquisition request, the server transmits the specific content to the user terminal.
When the server receives the acquisition request, the server collates the verification information after receiving the acquisition request with the latest real-time video acquired from the user terminal after receiving the acquisition request. The specific content is updated, and the updated specific content is transmitted to the user terminal.

Moreover, the control method of the useful information presentation system according to the present invention includes:
A control method of a useful information presentation system comprising a user terminal and a server,
The user terminal capturing real-time video and transmitting the real-time video to the server;
The server collates the real-time video with collation information to extract useful information, and sends an extraction completion notification to the user terminal;
The user terminal outputting the extraction completion notification and transmitting an acquisition request for specific content related to the useful information;
When the server receives the acquisition request, transmitting the specific content to the user terminal;
A useful information presentation system control method comprising:
When the server receives the acquisition request, the step of transmitting the specific content includes the verification information after receiving the acquisition request, and the latest acquired from the user terminal after receiving the acquisition request. The specific content is updated by collating with a real-time video, and the updated specific content is transmitted to the user terminal.

  According to the useful information presentation system and the control method of the useful information presentation system in the present invention, the user terminal takes a picture without depending on the processing capability of the user terminal and making the user aware of the location of the predetermined subject in advance. It is possible to present a wide range of useful information extracted from the real-time video, and to prevent oversight or oversight of useful information.

It is a block diagram which shows the general | schematic structure of the whole useful information presentation system which concerns on Embodiment 1. FIG. 3 is a block diagram of a user terminal according to Embodiment 1. FIG. 1 is a block diagram of a video distribution device according to Embodiment 1. FIG. 1 is a block diagram of a video collation device according to Embodiment 1. FIG. 1 is a block diagram of an information presentation control apparatus according to Embodiment 1. FIG. 4 is a flowchart showing the operation of the useful information presentation system according to Embodiment 1. 10 is a flowchart showing the operation of a modified example of the useful information presentation system according to Embodiment 1. 3 is a conceptual diagram illustrating an operation example of the useful information presentation system according to Embodiment 1. FIG. It is a block diagram which shows the general | schematic structure of the whole useful information presentation system which concerns on Embodiment 2. FIG. 10 is a flowchart showing the operation of the useful information presentation system according to the second embodiment.

  Embodiments of the present invention will be described below.

(Embodiment 1)
FIG. 1 is a block diagram showing an overall schematic configuration of a useful information presentation system according to Embodiment 1 of the present invention. The useful information presentation system according to Embodiment 1 of the present invention includes a user terminal 1, a video distribution device 2, a video collation device (31 to 33), a configuration management device 4, an information presentation control device 5, A knowledge management device 6. The user terminal 1 communicates with the video distribution device 2 or the information presentation control device 5 via a communication network.

  The user terminal 1 captures a real-time video and transmits the real-time video to the video distribution device 2. In addition, the user terminal 1 transmits context information including user attribute information and / or situation information to the video distribution device 2. Here, the user attribute information is information relating to static attributes of the user or the user terminal 1, for example, the user's age, gender, birthplace, address, occupation, hobby, disability type, user terminal 1. The identifier of the control unit 15 of the user terminal 1 to be described later. The status information is information regarding a dynamically changing status, such as time, current location, moving speed, noise, illuminance, terminal processing capability, user terminal status, driving of a car, in a train, and the like.

  The video distribution device 2 distributes the real-time video and context information received from the user terminal 1 to the video verification devices (31 to 33). The video distribution device 2 transmits context information as control information to the information presentation control device 5.

  Each video collation device (31 to 33) collates the real-time video received from the video distribution device 2 with the collation information, and extracts useful information by parallel processing. Each video collation device (31-33) transmits information related to the collation result including the useful information (hereinafter referred to as collation result) to the information presentation control device 5.

  Here, the collation information is information for collating a predetermined subject that may appear in a real-time video. For each verification information, an urgency level, an importance level, or a relevance level corresponding to user attribute information and / or situation information is defined. For example, a high degree of urgency is defined for a person with a disability regarding collation information related to a signboard indicating the location of danger such as a construction site. Further, a high degree of importance or relevance is defined for the elderly with respect to the verification information related to the pictosign indicating the location of the traffic barrier-free facility.

  Each of the video collating devices (31 to 33) recognizes one or more images (features of the subject of each frame or successive frames) constituting the real-time video. Each of the video collating devices (31 to 33) recognizes speech (a scale of a subject, a word including a name, a phoneme feature such as a speaker). And each video collation apparatus (31-33) collates a recognition result and the information for collation, and extracts useful information.

  The configuration management device 4 stores verification information. The verification information is updated and deleted as appropriate. For example, when a road sign or the like, which is verification information, is changed, the configuration management device 4 changes and stores the verification information. Each video collation device (31 to 33) communicates with the configuration management device 4 and updates the collation information held by itself.

  The information presentation control device 5 selects useful information included in the collation results received from the video collation devices (31 to 33) based on the control information (that is, context information) received from the video distribution device 2, and the user terminal 1 to send. Moreover, the information presentation control apparatus 5 refers to the knowledge management apparatus 6 and acquires specific contents corresponding to useful information. The specific content means useful information itself, detailed information of useful information, or information related to useful information. The information presentation control device 5 presents (transmits) the specific content to the user terminal 1 when there is a request for acquisition of specific content related to useful information from the user terminal 1.

  Next, the detailed configuration of each part of the useful information presentation system according to Embodiment 1 of the present invention will be described with reference to FIGS. FIG. 2 is a block diagram of the user terminal 1 according to the first embodiment. The user terminal 1 includes a session control unit 11, an operation unit 12, a context information generation unit 13, a real-time video generation unit 14, a control unit 15, and an output unit 16.

  The session control unit 11 performs control related to the start and end of a session with the video distribution device 2. Means for realizing session control are, for example, call connection using SIP, PUSH using WebSocket, pseudo-PUSH using XMLHttpRequest or Comet, PULL using HTTP, or the like. Through the session, the user terminal 1 performs real-time communication with the video distribution device 2. The operation unit 12 is preferably configured by a touch panel, a keyboard, and the like, receives an operation by a user, and passes information related to the operation to the control unit 15.

  The context information generation unit 13 generates context information including user attribute information and / or situation information. The user attribute information is information registered in advance by a user or an operator, and the context information generation unit 13 acquires the information from a memory (not shown). The context information generation unit 13 includes various sensors (not shown) and dynamically generates status information.

  The real-time video generation unit 14 preferably includes a camera and a microphone (not shown), shoots a subject with the camera, and acquires sound with the microphone to generate a real-time video. The real-time video generation unit 14 continuously transmits the generated real-time video to the video distribution device 2.

  The control unit 15 performs various controls related to the user terminal 1. Specifically, the control unit 15 starts a session by the session control unit 11 based on a user operation on the operation unit 12. Further, the control unit 15 receives an extraction completion notification of useful information from the information presentation control device 5 and causes the output unit 16 to output the extraction completion notification. Further, the control unit 15 transmits an acquisition request for specific content related to useful information to the information presentation control device 5 based on the operation of the user on the operation unit 12. In addition, when receiving specific content from the information presentation control device 5, the control unit 15 causes the output unit 16 to output the specific content.

  The output unit 16 includes a display, a speaker, a vibrator, and the like, and outputs (reproduces) the extraction completion notification and specific contents of useful information by video, audio, vibration, and the like based on the control of the control unit 15. For example, the output unit 16 outputs an extraction completion notification in a chime, and outputs specific contents by voice announcement.

  FIG. 3 is a block diagram of the video distribution device 2 according to the first embodiment. The video distribution device 2 includes a session control unit 21 and a media distribution unit 22.

  The session control unit 21 performs control related to the start and end of a session with the user terminal 1. Through the session, the video distribution device 2 performs real-time communication with the user terminal 1.

  The media distribution unit 22 distributes the real-time video received from the user terminal 1 to the video collation devices (31 to 33). Moreover, the media distribution part 22 transmits context information to a video collation apparatus (31-33). Further, the media distribution unit 22 transmits the context information received from the user terminal 1 to the information presentation control device 5 as control information. Further, when receiving the control information from the information presentation control device 5, the media distribution unit 22 distributes the real-time video at that time to the video collation devices (31 to 33).

  Further, the media distribution unit 22 selects a video collation device (31 to 33) to which distribution is appropriately performed based on at least one of the context information, the control information received from the information presentation control device 5, and the media format and type of the real-time video. Increase or decrease. For example, in a situation where a user is moving at high speed, unless there is an acquisition request, only a video collation device (for example, video collation device 31) that collates graphics in the signboard is used, and the signboard requires excessive processing. A video collation device that recognizes the written characters (for example, the video collation device 32) is not used. As described above, the media distribution unit 22 increases or decreases the processing amount and processing speed of the entire system by increasing or decreasing the video collating devices (31 to 33) to which the distribution is performed, thereby increasing the efficiency of the entire system.

  FIG. 4 is a block diagram of the video collation apparatus 31 according to the first embodiment. The video collation device 31 includes a media recognition unit 311, a reference management unit 312, and a media collation unit 313. Since the video collation devices 32 and 33 have the same configuration as the video collation device 31, a description thereof will be omitted. Preferably, each video collating device (31 to 33) shares the type of media to be collated. For example, the video collation device 31 performs collation processing related to graphics, the video collation device 32 performs collation processing related to characters, and the video collation device 33 performs collation processing related to speech, etc. Also good.

  The media recognizing unit 311 recognizes the characteristics of images and audio that constitute the real-time video received from the video distribution device 2. Then, the media recognizing unit 311 passes information related to the recognized feature to the media collating unit 313.

  The reference management unit 312 stores useful information to be extracted as verification information. Further, the reference management unit 312 communicates with the configuration management device 4 and updates the collation information stored therein.

  The media collation unit 313 collates the information related to the feature recognized by the media recognition unit 311 with the collation information stored in the reference management unit 312. The media verification unit 313 extracts information that matches the verification information based on the verification, that is, useful information. Then, the media verification unit 313 generates a verification result. The collation result is extracted from the identifier of the user terminal 1 that generated the real-time video, the identifier of the control unit 15 of the user terminal 1, the identifier of the video collation device (31 to 33), and the identifier of the matching information for matching (that is, extracted). Useful information identifier), position information in the video from which the information is extracted (for example, the pixel position of the signboard / signpost in the image), and position information of the subject corresponding to the information (for example, the user terminal 1 as the origin) 3 axis position coordinates and 3 axis rotation angle). Then, the media collation unit 313 transmits the collation result to the information presentation control device 5.

  FIG. 5 shows a block diagram of the information presentation control apparatus 5 according to the first embodiment. The information presentation control device 5 includes a filtering unit 51, a control unit 52, a media conversion unit 53, and a refresh unit 54.

  The filtering unit 51 includes one or more pieces of useful information included in the collation result received from the video collation device (31 to 33), each piece of information determined by the identifier of the useful information and the control information received from the video distribution device 2. Filtering (selection) based on at least one of the urgency level, freshness level, importance level, and relevance level. For example, the filtering unit 51 determines that useful information previously presented to the user terminal 1 is information with low relevance and excludes it, thereby avoiding redundant presentation of the same information. In addition, for example, the filtering unit 51 may have an attribute that is not suitable for going up and down stairs for a user who arrives in front of the station (for example, a person with a physical disability, an injured person, a pregnant woman, or an elderly person). ) Selects information on the location of escalators and elevators as highly important and relevant.

  The control unit 52 performs various controls related to the information presentation control device 5. Specifically, the control part 52 produces | generates the specific content of the extraction result which concerns on the identifier of one or more selected useful information. The control unit 52 refers to the knowledge management device 6 when generating the specific content. The knowledge management device 6 stores information related to useful information. For example, the knowledge management device 6 stores in advance information about events and facilities around the user in association with the position and time. For example, when the user terminal 1 transmits a real-time video of a signboard at the station entrance, the control unit 52 of the information presentation control device 5 uses information on the location of the station (for example, an escalator) based on the position information of the subject included in the verification result. And elevator operation information) from the knowledge management device 6. In this way, the control unit 52 generates not only that the station / entrance signboard is in the field of view, but also generates specific contents including the location and state of the premises equipment and presents them to the user terminal 1. .

  When the generation of the specific content is completed, the control unit 52 transmits an extraction completion notification to the user terminal 1. In addition, when receiving an acquisition request from the user terminal 1, the control unit 52 presents the generated specific content to the user terminal 1. Further, the control unit 52 updates the specific content of the useful information by the refresh unit 54 as appropriate.

  The media conversion unit 53 converts the data to be transmitted to the user terminal 1 into an optimal media format according to the user attribute information based on the control information received from the video distribution device 2 under the control of the control unit 52. For example, if the user attribute information includes information that is not suitable for confirmation of the screen display (for example, if the user attribute information includes “visually impaired”), the media conversion unit 53 announces a character string to be read out. Convert to format. Similarly, when the user attribute information includes “foreigner”, the media conversion unit 53 converts data to be transmitted to the user terminal 1 into data translated into the native language of the foreigner.

  The refresh unit 54 adaptively updates the specific contents of the extracted useful information based on at least one of the urgency level, freshness, importance level, and association level of the useful information under the control of the control unit 52. To do. The refresh unit 54 transmits control information based on the acquisition request to the video distribution device 2 when updating specific contents. For example, the refresh unit 54 determines that the freshness of the specific content is low and updates the specific content when a period of time equal to or longer than a predetermined period elapses after the extraction completion notification is transmitted and the acquisition request is received. In addition, when the content of the subject such as digital signage changes every moment, the refresh unit 54 determines that the specific content at the time of transmitting the extraction completion notification is low in freshness, and has received an acquisition request. Update to the current one.

  For example, it is assumed that the useful information is a vending machine and the specific content is “a vending machine is installed in the right front”. The information presentation control device 5 transmits the useful information extraction completion notification to the user terminal 1, and after a predetermined time has elapsed, the position of the user terminal 1 changes and the vending machine is installed on the right rear side. Assume that the positional relationship is as follows. In such a case, the refresh unit 54 updates the specific content of the useful information because the predetermined time has passed. In this way, the information presentation control device 5 can present the updated specific content “A vending machine is installed at the right rear” to the user terminal 1.

  Next, the operation of the useful information presentation system according to the present invention will be described with reference to the flowchart shown in FIG.

  First, the session control unit 11 of the user terminal 1 performs control related to the start of a session with the session control unit 21 of the video distribution device 2 based on the operation of the operation unit 12 of the user (step S1). Through the session, the user terminal 1 performs real-time communication with the video distribution device 2.

  Subsequently, the context information generation unit 13 of the user terminal 1 generates context information including user attribute information and / or situation information. Then, the context information generation unit 13 transmits the context information to the video distribution device 2 (step S2).

  Subsequently, the real-time video generation unit 14 of the user terminal 1 generates a real-time video. Then, the real-time video generation unit 14 transmits the generated real-time video to the video distribution device 2 (Step S3). In FIG. 6, only the first transmission of the real-time video is shown for simplicity, but the real-time video generation unit 14 continuously transmits the real-time video to the video distribution device 2.

  Subsequently, the media distribution unit 22 of the video distribution device 2 distributes the real-time video received from the user terminal 1 to the video collation devices (31 to 33) (step S4). The media distribution unit 22 preferably transmits the context information to the video collating devices (31 to 33). In FIG. 6, only the video collation device 31 is shown for the sake of simplicity, but the media distribution unit 22 also distributes the real-time video to the video collation device 32 and the video collation device 33, and preferably context information. Send. Subsequently, the media distribution unit 22 transmits the context information received from the user terminal 1 as control information to the information presentation control device 5 (step S5).

  Subsequently, the video verification device 31 performs video verification of the received real-time video (step S6). Specifically, the media recognizing unit 311 of the video collating device 31 recognizes the characteristics of images and audio that constitute the real-time video received from the video distribution device 2. Then, the media recognizing unit 311 passes information related to the recognized feature to the media collating unit 313. The media collation unit 313 then collates the information related to the feature recognized by the media recognition unit 311 with the collation information stored in the reference management unit 312. The media verification unit 313 extracts useful information based on the verification and generates a verification result. Subsequently, the media verification unit 313 transmits the generated verification result to the information presentation control device 5 (step S7).

  Subsequently, the information presentation control device 5 performs processing related to the collation result (step S8). Specifically, the filtering unit 51 of the information presentation control device 5 includes one or more useful information included in the collation result received from the video collation device (31 to 33), the identifier of the useful information, and the video distribution device 2. Filtering (selection) based on at least one of the urgency, freshness, importance and relevance of each piece of information determined by the control information received from. And the control part 52 produces | generates the specific content of the extraction result which concerns on the identifier of one or more selected useful information. The control unit 52 refers to the knowledge management device 6 that accumulates information related to useful information when generating the specific content.

  Subsequently, the media conversion unit 53 of the information presentation control device 5 controls the data transmitted to the user terminal 1 based on the control information received from the video distribution device 2 under the control of the control unit 52, that is, the extraction completion notification and the specific content. Is converted into an optimum media format corresponding to the user attribute information (step S9). Subsequently, the control unit 52 transmits an extraction completion notification to the user terminal 1 (step S10).

  Next, when receiving the useful information extraction completion notification from the information presentation control device 5, the control unit 15 of the user terminal 1 causes the output unit 16 to output the extraction completion notification (step S11). For example, the output unit 16 outputs an extraction completion notification using a chime.

  Subsequently, the control unit 15 of the user terminal 1 transmits an acquisition request for specific content related to useful information to the information presentation control device 5 based on the operation of the user related to the operation unit 12 (step S12).

  Subsequently, when there is a request for acquisition of specific content from the user terminal 1, the control unit 52 of the information presentation control device 5 presents (transmits) the specific content to the user terminal 1 (step S13).

  Subsequently, the output unit 16 of the user terminal 1 outputs (reproduces) specific contents by video, audio, vibration, or the like (step S14). For example, the output unit 16 outputs the specific content by voice announcement. Then, the process ends.

  Next, the operation of the modification of the useful information presentation system according to the present invention will be described with reference to the flowchart shown in FIG. The same operations as those in the flowchart shown in FIG. 6 are denoted by the same reference numerals, and description thereof is omitted.

  In FIG. 7, after the user terminal 1 outputs the extraction completion notification (step S <b> 11), the control unit 15 of the user terminal 1 is useful for the information presentation control device 5 based on the operation of the user related to the operation unit 12. An acquisition request for specific contents related to the information is transmitted (step S21). In addition, the real-time video generation unit 14 of the user terminal 1 transmits the generated real-time video to the video distribution device 2 (Step S22).

  Subsequently, the control unit 52 of the information presentation control device 5 sets the specific content of the useful information to the refresh unit 54 based on at least one of the urgency level, freshness level, importance level, and relevance level of the information. Update the contents. The refresh unit 54 transmits control information based on the acquisition request to the video distribution device 2 (step S23).

  Subsequently, the media distribution unit 22 of the video distribution device 2 distributes the real-time video received from the user terminal 1 to the video collation devices (31 to 33) (step S24). In FIG. 7, only the video collation device 31 is illustrated for simplicity, but the media distribution unit 22 distributes the real-time video also to the video collation device 32 and the video collation device 33.

  Subsequently, the video matching device 31 performs video matching of the received real-time video (step S25). Specifically, the media recognizing unit 311 of the video collating device 31 recognizes the characteristics of images and audio that constitute the real-time video received from the video distribution device 2. Then, the media recognizing unit 311 passes information related to the recognized feature to the media collating unit 313. The media collation unit 313 then collates the information related to the feature recognized by the media recognition unit 311 with the collation information stored in the reference management unit 312. The media verification unit 313 extracts useful information based on the verification and generates a verification result.

  Subsequently, the media verification unit 313 transmits the generated verification result to the information presentation control device 5 (step S26).

  Subsequently, the control unit 52 of the information presentation control device 5 updates the specific content based on the collation result. The control unit 52 preferably refers to the knowledge management device 6 when generating the specific content. Then, the control unit 52 presents (transmits) the updated specific content to the user terminal 1 (step S27).

  Subsequently, the output unit 16 of the user terminal 1 outputs (reproduces) specific contents by video, audio, vibration, or the like (step S28). Then, the process ends.

  The control unit 52 of the information presentation control device 5 may store the useful information in the knowledge management device 6 when the useful information is extracted. Then, the control unit 52 may present the accumulated useful information to a plurality of user terminals. Preferably, the useful information is stored in association with the context information related to the extraction source real-time video. Then, the control unit 52 may be able to present (transmit) the accumulated information to a plurality of user terminals. Preferably, the control unit 52 may be able to present (transmit) to a plurality of user terminals having the context information.

  FIG. 8 is a conceptual diagram showing an operation example of the useful information presentation system according to Embodiment 1 of the present invention. FIG. 8A illustrates a foreign traveler and a visually impaired person as users. In this case, the context information transmitted from the user terminal 1 to the video distribution device 2 includes user attribute information “foreigner” and “person with visual impairment”, respectively. For example, the user terminal 1 possessed by a foreign traveler transmits a construction site video to the video distribution device 2 as a real-time video. If it does so, the image | video collation apparatus 31 will acquire signboard information from the image | video of a construction site. Based on the user attribute information “foreigner”, the information presentation control device 5 translates the data to be transmitted (“There is a detour on the right”) and sends useful information “Detour: Turn Right” to the user terminal 1. Present.

  Similarly, when the real-time video is received from the user terminal 1 of the visually impaired person, the information presentation control device 5 converts the data to be transmitted into audio data based on the user attribute information “visually impaired person”. , Useful information “There is a detour on the right” is presented to the user terminal 1.

  FIG. 8B exemplifies “elderly people”, “pregnant women”, “injured people”, and “limbs disabled” as users. In this case, the context information transmitted from the user terminal 1 to the video distribution device 2 includes user attribute information “elderly”, “pregnant”, “injured”, and “limb disabled”. For example, the user terminal 1 of an elderly person transmits a video in front of the station to the video distribution device 2 as a real-time video. Then, the video collation apparatus 31 acquires signboard information from the video in front of the station. The information presentation control device 5 transmits useful information to the user terminal 1 that “the usual escalator is under construction. There is an elevator in the 11 o'clock direction”.

  As described above, according to the present invention, the user terminal 1 transmits real-time information to the video distribution device 2 and extracts useful information from the real-time video using the video collation devices (31 to 33). It is possible to present a wide range of useful information extracted from real-time video captured by the user terminal 1 without making the user aware of the location of the predetermined subject in advance without depending on the processing capability of the user.

  Further, according to the present invention, the information presentation control device 5 first transmits only useful information extraction completion notification, and presents the specific content when receiving the acquisition request of specific content from the user terminal 1, It is possible to prevent oversight or oversight of useful information, and it is possible to steadily present guidance to a user in a distracted state who is not conscious of the location of a predetermined subject.

  According to the present invention, when useful information is extracted, the useful information is stored in the knowledge management device 6 and the stored useful information is presented to a plurality of user terminals. Can be shared with other users, and the discrepancy between the information stored in the knowledge management device 6 and the actual situation in the real world can be reduced.

  Further, according to the present invention, the configuration management device 4 accumulates the collation information, and the video collation devices (31 to 33) collate using the collation information. Therefore, the schedule table that can be managed by the user terminal 1 The system can be operated without changing the configuration of the user terminal 1 as well as a wide range of collations not limited to map information.

  Further, according to the present invention, since a session is established between the user terminal 1 and the video distribution device 2 and the user terminal 1 continuously transmits real-time video, a predetermined subject related to useful information is located around Even if the user does not know the facts that exist and the user is not conscious of their location, it is possible to guide useful information by simply reflecting the subject in the field of view of the real-time video.

  Also, according to the present invention, by generating useful information with reference to the knowledge management device 6, not only information on the subject itself but also other related information related to the subject can be guided.

  Further, according to the present invention, the filtering unit 51 of the information presentation control device 5 filters useful information based on the useful information identifier and the urgency level of each information determined by the control information. Really useful information can be presented.

  In addition, according to the present invention, the media conversion unit 53 of the information presentation control device 5 converts the data to be transmitted to the user terminal 1 into an optimal media format according to the user attribute information. Useful information can be presented in a suitable information format.

  Further, according to the present invention, the refresh unit 54 of the information presentation control device 5 updates the specific content of the useful information based on at least one of the urgency level, freshness level, importance level, and relevance level of the information. For example, even when a specific content acquisition request is received after a certain time has elapsed, truly useful information can be presented.

  Note that the context information generation unit 13 of the user terminal 1 is operated by the user (for example, setting of the drive mode / manner mode), and sensors (for example, a camera, a microphone, a GPS, an illuminance sensor, a magnetic sensor) provided in the user terminal 1 State estimation based on input from sensors, acceleration sensors), information managed by the OS of the user terminal 1 (for example, the type of application being activated, the remaining battery level), or from the outside via a communication network (for example, a web service) Context information may be generated based on the input.

  In addition, although the example which produces | generates and transmits the real-time image | video comprised from a video and an audio | voice has shown the real-time video generation part 14 of the user terminal 1, it is not restricted to this, The multimedia comprised only by one of an image | video or an audio | voice. Information may be generated and transmitted.

  The transmission of the real-time video from the user terminal 1 to the video distribution device 2 may be file transmission / reception by HTTP or streaming or streaming by RTP. The transmission of context information and the transmission of real-time video may be performed synchronously and continuously, or asynchronously and intermittently. For example, the transmission of the context information may be performed every time the sensor value changes more than a specified value after activation of the user terminal 1, and the real-time video may be transmitted periodically and intermittently after the session starts. Moreover, you may superimpose and transmit context information as metadata of real-time video. Similarly, the control unit 15 of the user terminal 1 may superimpose and transmit the acquisition request as real-time video metadata. Further, the user terminal 1 may include a plurality of cameras, and a plurality of real-time videos may be transmitted to the video distribution device 2.

  Similarly, the transmission (distribution) of the real-time video from the video distribution device 2 to the video verification device 31 may be a file format or a streaming format. Furthermore, media received from the user terminal 1 in a streaming format may be converted into a file format and distributed. Streaming protocols (RTSP / MMS / RTMP, etc.) and file formats (for example, GIF / JPEG / PNG / BITMAP for image files, WAV / AAC / MP3 for audio files, MPEG2 / MPEG4 / MP4 / AVI for video files, etc.) May be distributed after conversion.

  Note that the video distribution device 2 uses the type or number of video collation devices (31 to 33) that perform collation processing in parallel after receiving an acquisition request when the collation processing is performed in parallel before receiving the acquisition request. The video may be distributed so as to be more than that. By doing in this way, the kind or number of processes used for processing among the video collating devices (31 to 33) can be reduced regularly, and the load on the entire system at the steady state can be suppressed. On the other hand, after receiving an acquisition request from the user terminal 1, the number of types or the number to be used is increased from that in the normal state, and verification processing is performed with a larger number of units, so that the processing efficiency of the entire system is improved and the processing delay time is reduced. It can be shortened.

  The video collation devices (31 to 33) receive control information based on the acquisition request from the information presentation control device 5 via the video distribution device 2, and increase or decrease the collation information used based on the control information. Also good. Specifically, the video collation devices (31 to 33) are used for collation when a set of collation information when collating with real-time video before receiving an acquisition request is collated with real-time video after receiving the acquisition request. It may be a subset of information. That is, the collation information used before receiving the acquisition request is a part of the collation information used after receiving the acquisition request. By doing so, it is possible to reduce the load of calculation processing in a steady state related to the collation before the acquisition request, and to reduce the load of the entire system. In addition, collation processing related to collation after an acquisition request can be performed in more detail by increasing the collection of collation information from the normal state, improving the overall processing efficiency and shortening the processing delay time. be able to. That is, this system can adaptively increase / decrease the type or number of verification information and video verification devices (31-33) based on control information and the like, thereby improving the processing efficiency of the entire system.

  The video collation devices (31 to 33), based on the context information received from the video distribution device 2, select collation information to be used for collation according to the degree of relevance and importance of the context information. It may be limited to information. For example, when the video collation device (31-33) collates road signs, only the sign posted to the driver is included in the collation information during driving, and the sign displayed to the pedestrian is not included. Good. By doing in this way, the load of the collation process of a video collation apparatus (31-33) can be reduced.

  Note that the media recognition unit 311 of the video collating device 31 may perform media recognition limited to a part of an image or sound (for example, a partial area of an image or a partial frequency of sound) constituting a real-time video, A part of the image may be processed (for example, a signboard / signature portion shown in an image is cut out and a three-axis rotation operation is performed to reproduce an image in a state of facing the signboard / signature). Further, the media recognition unit 311 may perform preprocessing for target limitation / processing in cooperation with the media collation unit 313. For example, a figure having a specific format, such as a QR code (registered trademark) or an AR marker, can be collated even if it is photographed with a certain degree of rotation angle of the three axes, and the result of the comparison includes the three axes of the subject. A rotation angle may be included. Here, in recognition of characters, it is generally necessary to take a picture in a face-to-face state. Therefore, when recognizing characters written on a signboard / signpost with an AR marker attached, the above-mentioned pre-processing for character recognition is performed by grasping the rotation angles of the three axes of the subject from the AR marker comparison result. Can be processed (reproduction of the directly facing state).

  It should be noted that a predetermined subject, that is, useful information to be extracted may be replaced as the verification information, and the verification processing of the video verification devices (31 to 33) may be made efficient. For example, when the logo of a company such as a bank is changed, the new logo is registered as a subject. By doing in this way, the efficiency of the collation process of a video collation apparatus (31-33) can be achieved.

  In addition, you may define based on the combination of a user attribute and status information regarding the urgency, the importance, or the relevance of the information for collation. For example, the urgency level, importance level, and relevance level of the verification information may be defined based on a combination of the user attribute “user group looking for an elevator” and the situation information. For example, the user attribute information “elderly”, “pregnant”, “injured”, and “limb disabled” in FIG. 8B are designated as the “user group looking for an elevator” and the urgency of the verification information -You may define importance and relevance.

  For example, in FIG. 8B, each user has user attribute information “elderly”, “pregnant”, “injured”, and “limb disabled”. These user attribute information is common as “user group looking for elevator”. In such a case, the information presentation control device 5 may present useful information such as “the usual escalator is under construction. There is an elevator in the 11 o'clock direction” to the plurality of user terminals.

  The control unit 52 of the information presentation control device 5 notifies the extraction completion when presenting useful information with a high degree of urgency to the user terminal 1 (for example, when alerting a dangerous object such as an approaching vehicle). The specific contents may be transmitted to the user terminal 1 at the same time. In this case, the control unit 15 of the user terminal 1 outputs the specific contents immediately by the output unit 16.

  In addition, the control part 52 of the information presentation control apparatus 5 may transmit the extraction completion notification different from usual, when showing useful information with high urgency to the user terminal 1. For example, the extraction completion notification is a chime when it is not an emergency, but a loud siren during an emergency. In this way, it is possible to prompt the user to perform a quick request operation. Furthermore, when guiding information with high importance or relevance, the control unit 52 may transmit different extraction completion notifications to the user terminal 1 to prompt the user to perform reliable request operations. For example, a dedicated extraction completion notification may be transmitted for each user type (for example, a user group searching for an elevator and a user group searching for a slope). That is, the control unit 52 of the information presentation control device 5 may include a plurality of types of extraction completion notifications, and may adaptively use the plurality of types of extraction completion notifications based on the urgency level, importance level, relevance level, and the like of useful information. . Further, the control unit 15 of the user terminal 1 may automatically transmit an acquisition request when receiving an extraction completion notification with a high degree of urgency.

  Note that the refresh unit 54 of the information presentation control device 5 may receive a request for explicitly updating the specific content from the user terminal 1 and update the specific content based on the request. Preferably, the user terminal 1 transmits a specific content update request simultaneously with the specific content acquisition request. Furthermore, the control unit 15 of the user terminal 1 may transmit a request for updating explicit specific contents even if the extraction completion notification is not received.

  In addition, although the useful information presentation system which concerns on Embodiment 1 illustrated the structure provided with three video collation apparatuses (31-33) in FIG. 1, it is not restricted to this, Two or less may be sufficient, or There may be four or more.

  In the first embodiment, the useful information presentation system includes the video distribution device 2, the video collation devices (31 to 33), the configuration management device 4, the information presentation control device 5, and the knowledge management device 6 as individual hardware. Although the example which comprised is shown, it is not restricted to this, You may comprise by one server which mounted all the functions of each apparatus, and you may comprise by two or more servers.

  Each functional block (session control unit 11, operation unit 12, context information generation unit 13, real-time video generation unit 14, control unit 15, and output unit 16) in the user terminal 1 is distributed by a plurality of terminals. You may prepare. For example, a configuration in which the user carries a wearable camera having the real-time video generation unit 14 and a smartphone having the output unit 16 can be considered. In addition, for example, a configuration in which a user having an attribute unsuitable for a touch panel operation of a smartphone such as a visually impaired person carries a remote control (for example, an electronic white with a communication function) provided with an operation unit 12 for remotely operating the smartphone. I can think.

  In addition, in this useful information presentation system, a predetermined subject such as a figure is placed at a certain height above the ground (for example, a range of 150 cm to 250 cm) so that the subject can easily enter the visual field of view, and the elevation angle at the time of shooting The ground height at which the subject is placed may be included in the field of view. Alternatively, the user terminal 1 may prompt the user to adjust the elevation angle by outputting a warning with sound or vibration.

(Embodiment 2)
The second embodiment of the present invention will be described below. FIG. 9 is a block diagram showing the overall configuration of the useful information presentation system according to Embodiment 2 of the present invention. The same components as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted. The useful information presentation system according to the second embodiment is different from the configuration according to the first embodiment in that a user management terminal 7 is provided.

  The user terminal 1 according to the second embodiment transmits context information to the user management terminal 7. In addition, the user terminal 1 transmits an acquisition request for specific contents of useful information to the user management terminal 7.

  The user management terminal 7 transmits (transfers) the context information received from the user terminal 1 to the video distribution device 2 and the information presentation control device 5. In addition, when the user management terminal 7 receives an acquisition request for specific contents from the user terminal 1, the user management terminal 7 transmits (transfers) the acquisition request to the information presentation control device 5.

  Next, the operation of the useful information presentation system according to Embodiment 2 of the present invention will be described with reference to the flowchart shown in FIG. The same reference numerals are given to the same operations as those in the flowchart (FIG. 7) of the corresponding operations in the first embodiment, and the description will be omitted.

  When the session is started in step S1, the context information generation unit 13 of the user terminal 1 transmits the generated context information to the user management terminal 7 (step S2a). The user management terminal 7 transfers the received context information to the video distribution device 2 (step S2b).

  Further, the control unit 15 of the user terminal 1 transmits a request for obtaining useful information to the user management terminal 7 based on the operation of the user related to the operation unit 12 (step S12a). Subsequently, the user management terminal 7 transfers the received acquisition request to the information presentation control device 5 (step S12b).

  As described above, according to the useful information presentation system according to the second exemplary embodiment, the user management terminal 7 receives the context information and the acquisition request from the user terminal 1, so that the destination transmitted by the user terminal 1 is the user. The management terminal 7 can be unified and centrally managed.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, the functions included in each means, each step, etc. can be rearranged so that there is no logical contradiction, and a plurality of means, steps, etc. can be combined or divided into one. .

DESCRIPTION OF SYMBOLS 1 User terminal 2 Video distribution apparatus 31-33 Video collation apparatus 4 Configuration management apparatus 5 Information presentation control apparatus 6 Knowledge management apparatus 11 Session control part 12 Operation part 13 Context information generation part 14 Real-time video generation part 15 Control part 16 Output part 21 Session control unit 22 Media distribution unit 311 Media recognition unit 312 Reference management unit 313 Media collation unit 51 Filtering unit 52 Control unit 53 Media conversion unit 54 Refresh unit 7 User management terminal

Claims (8)

A useful information presentation system comprising a user terminal and a server,
The user terminal captures a real-time video and transmits the real-time video to the server;
The server collates the real-time video and collation information to extract useful information, and sends an extraction completion notification to the user terminal,
The user terminal outputs the extraction completion notification, transmits an acquisition request for specific contents related to the useful information,
When the server receives the acquisition request, the server transmits the specific content to the user terminal.
When the server receives the acquisition request, the server collates the verification information after receiving the acquisition request with the latest real-time video acquired from the user terminal after receiving the acquisition request. The useful information presentation system characterized by updating a specific content and transmitting the updated specific content to the user terminal.   The useful information presentation system according to claim 1, wherein the server updates the specific content of the useful information based on at least one of urgency, freshness, importance, and relevance. . The collation information when collating with the real-time video before receiving the acquisition request is a part of the collation information when collating with the latest real-time video after receiving the acquisition request. The useful information presentation system according to claim 1, wherein the system is useful information presentation. The server is a plurality of devices, and the plurality of devices perform collation between the real-time video and the collation information by parallel processing,
The number of the devices that process the collation in parallel after receiving the acquisition request is larger than the number of the devices that process the collation in parallel before receiving the acquisition request. Useful information presentation system given in any 1 paragraph. A control method of a useful information presentation system comprising a user terminal and a server,
The user terminal capturing real-time video and transmitting the real-time video to the server;
The server collates the real-time video with collation information to extract useful information, and sends an extraction completion notification to the user terminal;
The user terminal outputting the extraction completion notification and transmitting an acquisition request for specific content related to the useful information;
When the server receives the acquisition request, transmitting the specific content to the user terminal;
A method for controlling a useful information presentation system including:
When the server receives the acquisition request, the step of transmitting the specific content includes the verification information after receiving the acquisition request, and the latest acquired from the user terminal after receiving the acquisition request. A control method of a useful information presentation system, wherein the specific content is updated by collating with a real-time video, and the updated specific content is transmitted to the user terminal.   6. The useful information presentation system according to claim 5, wherein the server updates specific contents of useful information based on at least one of urgency, freshness, importance, and relevance. Control method. The collation information when collating with the real-time video before receiving the acquisition request is a part of the collation information when collating with the latest real-time video after receiving the acquisition request. The control method of the useful information presentation system according to claim 5 or 6, characterized in. The server is a plurality of devices, and the plurality of devices perform collation between the real-time video and the collation information by parallel processing,
8. The number of devices that perform parallel processing of collation after receiving the acquisition request is greater than the number of devices that perform parallel processing of collation before receiving the acquisition request. A control method of the useful information presentation system according to claim 1.