JP2016036106A - Information processing apparatus, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display the relevant information of a captured image stably, in an augmented reality display system using a device performing instable positional variation.SOLUTION: An information processing apparatus has superimposed display means performing recognition of an object from a captured image being displayed on a display device, extraction of the relevant information of the object, and display of the relevant information on the display device while superimposing on the captured image, and monitoring means for monitoring whether or not the object, recognized in the recognition, is recognized continuously. When the object is not recognized by the monitoring means, the superimposed display means continues or stops superimposed display, based on the progress thereof.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  Conventionally, in a technology called augmented reality (or virtual reality) in which an object is recognized from a live-action video, and electronic information related to the object is converted into an electronic video and superimposed on the live-action video, the readability and visual recognition of the information that has been converted into electronic video A method for improving the performance has been proposed. Patent Document 1 discloses a method for changing the display mode of an electronic video according to shooting conditions such as weather and shooting location. Further, Patent Document 2 discloses related information by simultaneously displaying information on a visible object and information on an invisible object related to the object by using a positional relationship between the shooting field of view of the live-action video and the object. It shows how to browse all of them.

JP 2011-233005 A JP 2011-164701 A

In a method of recognizing an object from a live-action video and superimposing and displaying related electronic video, in order to display the electronic video stably, keep the object stable in the field of view so that the recognition of the object is not interrupted. It is necessary to keep. On the other hand, as an apparatus for executing augmented reality display, a device in which a camera and a display are integrated, such as a tablet or a glasses-type display, is generally used. In such a device, the photographing field of view easily fluctuates in accordance with the movement of the user's hand or face, so it is difficult to stabilize the object within the photographing field of view.
For this reason, for example, when an object moves out of the field of view and cannot be recognized due to a user's unexpected operation, there is a problem that display of an electronic video is interrupted and the user cannot view information stably. In addition, there is a problem that information cannot be browsed efficiently if displayed information is displayed again when an object returned to the field of view is re-recognized. In the above-described prior art, these problems have not been solved.

  The present invention has been made in view of such problems, and an object of the present invention is to stably superimpose and display related information of a live-action video in an augmented reality display system using a device that performs unstable position fluctuation.

  Therefore, the present invention is an information processing apparatus, a recognition process for recognizing an object from a captured image being displayed on a display device, an extraction process for extracting related information related to the object, and the related information, Superimposing display means for performing superimposing display processing including display processing for superimposing on a captured image and displaying on the display device; and monitoring means for monitoring whether or not the object recognized in the recognition processing is continuously recognized. And when the object is no longer recognized by the monitoring unit, the superimposed display means continues or stops the superimposed display process based on the progress of the superimposed display process. .

  According to the present invention, it is possible to stably superimpose and display related information of a captured image in an augmented reality display system using a device that performs unstable position fluctuation.

It is a figure which shows an information processing system. It is explanatory drawing of an object. It is a figure which shows the hardware constitutions of a tablet terminal. It is a figure which shows the software structure of a tablet terminal. It is a figure which shows an example of a data structure of object information. It is a figure which shows an example of a data structure of status management information. It is a flowchart which shows a superimposition display process. It is a flowchart which shows a superimposition display process. It is a flowchart which shows a superimposition display process. It is a flowchart which shows the monitoring process. It is a figure which shows an example of the data structure of instruction | indication management information. It is a flowchart which shows the superimposition display process which concerns on 2nd Embodiment. It is a flowchart which shows the superimposition display process which concerns on 2nd Embodiment. It is a figure which shows the example of a display of an inquiry screen. It is a flowchart which shows the monitoring process which concerns on 2nd Embodiment. It is a figure which shows the hardware constitutions of the tablet terminal which concerns on 3rd Embodiment. It is a flowchart which shows the monitoring process which concerns on 3rd Embodiment. It is a flowchart which shows the monitoring process which concerns on 4th Embodiment. It is a flowchart which shows the monitoring process which concerns on 4th Embodiment. It is a figure which shows the hardware constitutions of the tablet terminal which concerns on 5th Embodiment. It is a flowchart which shows the monitoring process which concerns on 5th Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating an information processing system according to the first embodiment. The tablet terminal 101 as an example of an information processing apparatus executes an augmented reality (AR) display process. That is, the tablet terminal 101 includes a photographing device and a display device, and superimposes and displays an electronic image obtained by visualizing electronic information on a video (moving image) of the real world object 104 photographed by the photographing device. . Here, real world objects refer to all real objects such as maps, paper documents and buildings.

  The object database (DB) 102 is a database in which object information associated with a real-world object on a one-to-one basis is registered. By using the object DB 102, it is possible to search / acquire object information using information that can uniquely identify an object in the real world as a search condition. The related information database (DB) 103 is a database in which related information of object information registered in the object DB 102 is registered. The related information is assumed to be a moving image. The related information may be text information or a still image. That is, the related information is information in an arbitrary format. A plurality of pieces of related information may be associated with one object. In the aforementioned object information, data for accessing related information is registered.

  In this system, WWW (World Wide Web) is used as the related information database, and the URL (Uniform Resource Locator) of the information source of the related information is registered in the object information. The tablet terminal 101, the object DB 102, and the related information DB 103 are connected to the network 105, and are in a state where they can communicate with each other.

  FIG. 2A is a diagram illustrating an example of a printed material used as the real world object 104 in the information processing system according to the present embodiment. A two-dimensional barcode 201 is printed on the printed material 200. The two-dimensional barcode 201 is obtained by encoding an object identifier that uniquely identifies the printed material 200. Assume that the object identifier is registered as object information in the object DB 102. In order to simplify the explanation, this information processing system uses a method of specifying a real-world object using a two-dimensional barcode, but the present invention can be implemented using other methods. . For example, the information processing system may specify the object by a method of identifying the object by extracting the image feature amount from the imaging data. Further, for example, the information processing system may specify an object based on information other than imaging data, such as RFID (Radio Frequency IDentification).

  FIG. 2B is a diagram illustrating a display example of the display device of the tablet terminal 101. The display device 210 displays a printed material 200 photographed by the photographing device. Further, electronic images 211 and 212 as related information associated with the two-dimensional barcode specified from the printed matter 200 are displayed in a superimposed manner. In the example illustrated in FIG. 2B, the tablet terminal 101 displays pop-up electronic images 211 and 222 related to the chart of the printed matter 200.

  Next, an outline of augmented reality display processing on the tablet terminal 101 by the information processing system will be described. First, when the user photographs the printed material 200 with the tablet terminal 101, the tablet terminal 101 recognizes the two-dimensional barcode 201 of the printed material 200 and detects an object identifier from the two-dimensional barcode 201. Next, the tablet terminal 101 connects to the object DB 102, searches for object information using the detected object identifier as a search condition, and acquires this. Subsequently, the tablet terminal 101 extracts a URL for accessing the related information DB 103 from the acquired object information. Next, the tablet terminal 101 accesses the extracted URL, acquires related information, and generates an electronic video to be displayed on the tablet terminal 101. Finally, the tablet terminal 101 superimposes and displays the generated electronic video on the captured image of the printed material 200. The augmented reality display is performed by the above procedure.

  FIG. 3 is a diagram illustrating a hardware configuration of the tablet terminal 101. The CPU 301 executes an OS (Operating System) and various application programs, and controls each unit of the tablet terminal 101. The ROM 302 stores fixed data among programs executed by the CPU 301 and calculation parameters. A RAM 303 provides a work area for the CPU 301 and a temporary storage area for data. The ROM 302 and the RAM 303 are connected to the CPU 301 via the bus 304. Note that the functions and processing of the tablet terminal 101 to be described later are realized by the CPU 301 reading a program stored in the ROM 302 or the like and executing this program.

  An input device 305, a display device 210 such as a CRT and a liquid crystal display, a hard disk device, an external storage device 307 such as an MO and a CD-ROM, and a photographing device 308 are connected to a bus 304 via an interface 309. The bus 304 is connected to the network 105 via the communication device 310.

  FIG. 4 is a diagram illustrating a software configuration of the tablet terminal 101. The tablet terminal 101 includes a reception unit 401, a shooting control unit 402, a superimposed display processing unit 410, and a monitoring unit 407. The accepting unit 401 accepts information input by the user via the input device 305. The shooting control unit 402 controls the shooting device 308 to generate a shot image with a real world object as a subject. Here, it is assumed that the captured image is a moving image.

  The superimposed display processing unit 410 performs a superimposed display process as an augmented reality (AR) display process. The superimposed display processing unit 410 includes a display processing unit 403, an object recognition unit 404, an object management unit 405, and a related information management unit 406, and the superimposed display processing is realized by the cooperation of these units. . That is, the display processing unit 403 displays the obtained captured image on the display device 210, and further displays the related information of the object obtained from the captured image superimposed on the captured image (display processing). The object recognition unit 404 analyzes the moving image and detects a two-dimensional barcode as an object (recognition process). The object management unit 405 analyzes a two-dimensional barcode as an object and specifies an object identifier. The object management unit 405 also manages the recording of objects detected by the object recognition unit 404 and the exchange of data with the object DB 102. The related information management unit 406 extracts a moving image as related information of the real world object recognized by the object recognition unit 404 from the related information DB 103 (extraction process). When the related information is information for generating a moving image, the related information management unit 406 generates a moving image from the related information and sends it to the display processing unit 403.

  When the superimposed display processing by the superimposed display processing unit 410 starts, the monitoring unit 407 monitors whether or not the object recognized by the object recognition unit 404 is continuously recognized. When the object is no longer recognized, the monitoring unit 407 determines whether to continue or stop the superimposed display process based on the progress of the superimposed display process, and determines whether the superimposed display process unit 410 determines whether the superimposed display process is to be stopped. Give instructions.

  Next, data used by the tablet terminal 101 according to the present embodiment will be described. FIG. 5 is a diagram illustrating an example of a data configuration of the object information 501 registered in the object DB 102. The object identifier 502 is an identifier that uniquely identifies an object in the real world. The related information identifier 503 is information for accessing the related information of the real-world object. As described above, in the information processing system according to the present embodiment, the related information identifier 503 is the URL of the information source of the related information. Is registered. If there are a plurality of related information, a plurality of related information identifiers 503 are listed in association with one object identifier 502.

FIG. 6 is a diagram illustrating an example of a data configuration of the situation management information 601 used by the object management unit 405. The status management information 601 is information for managing the progress status of the superimposed display processing for the object. In the object identifier 602, the object identifier specified by the object management unit 405 is recorded. In the recognition state 603, a state relating to object recognition processing by the tablet terminal 101 is recorded. In the present embodiment, one of the following two states is recorded as the recognition state.
Recognition state “1”: Detection Recognition state “2”: Recognition

In the processing state 604, a state relating to object processing for the recognized object is recorded. In the present embodiment, any one of the following seven states is recorded as the processing state.
Processing state “1”: Object recognized Processing state “2”: Object identifier being acquired Processing state “3”: Object identifier already acquired Processing state “4”: Relevant information being acquired Processing state “5”: Relevant information already acquired State “6”: Displaying Processing state “7”: Displayed Note that the recognition state and the processing state are examples of information indicating the progress status of the superimposed display processing unit 410. The display position information 605 records what part of the related information the display device 210 of the tablet terminal 101 has displayed. In the present embodiment, since it is a moving image, the last displayed moving image frame number is recorded.

  7 to 9 are flowcharts showing the superimposing display process by the tablet terminal 101. In step S <b> 701, the reception unit 401 waits for input of a start instruction to the input device 305. When accepting the start instruction (Yes in S701), the accepting unit 401 advances the process to S702. In step S702, the imaging control unit 402 starts imaging. In step S <b> 703, the display processing unit 403 starts displaying the captured image obtained in step S <b> 702 on the display device 210.

  In step S704, the object recognition unit 404 analyzes the captured image and detects a two-dimensional barcode as an object. In step S <b> 705, the object recognition unit 404 confirms whether the detected object is a redetected object. Here, the redetected object is the same object as the object that has been detected in the past. Specifically, the object recognition unit 404 records objects detected in the past, and compares the object obtained this time with the recorded object. Then, the object recognizing unit 404 determines that the detected object is a redetected object when the degree of similarity between the two is equal to or greater than the threshold.

  If the detected object is a redetected object (Yes in S705), the object recognizing unit 404 advances the process to S901 shown in FIG. If the detected object is not a redetected object (No in S705), the object recognizing unit 404 advances the process to S706. In step S706, the object management unit 405 generates status management information 601 and records it in the RAM 303 or the like. At this time, the object identifier 602, the recognition state 603, the processing state 604, and the display position information 605 of the status management information 601 are not set. In step S <b> 707, the object management unit 405 sets “1 (detected)” in the recognition state 603 of the situation management information 601.

  In step S708, the object management unit 405 identifies the object identifier by decoding the two-dimensional barcode as the object. In step S <b> 709, the object management unit 405 sets “1 (object recognized)” to the processing state 604 of the situation management information 601. In step S <b> 710, the object management unit 405 sets the specified object identifier as the object identifier 602 of the situation management information 601. In step S <b> 711, the object management unit 405 updates the recognition state 603 of the situation management information 601 to “2 (recognition)”.

  In step S <b> 712, the object management unit 405 connects to the object DB 102 via the communication device 310 and checks whether the identified object identifier is registered in the object DB 102. If the identified object identifier is registered (Yes in S712), the object management unit 405 advances the process to S713. If the identified object identifier is not registered (No in S712), the object management unit 405 advances the process to S714. In step S713, the related information management unit 406 performs processing for displaying related information in a superimposed manner, that is, related information management processing. Detailed processing in S713 will be described with reference to FIG. On the other hand, in S714, the object management unit 405 deletes the status management information 601 recorded in S706, and ends the superimposed display process.

  FIG. 8 is a flowchart showing detailed processing in the related information display processing. In step S <b> 801, the object management unit 405 updates the processing state 604 of the situation management information 601 to “2 (acquiring object information)”. In step S <b> 802, the object management unit 405 receives object information 501 including the specified object identifier from the object DB 102, and records this in the RAM 303 or the like. In step S <b> 803, the object management unit 405 changes the processing state of the status management information 601 to “3 (object information acquired)”.

  In step S <b> 804, the related information management unit 406 extracts the URL recorded as the related information identifier from the status management information 601. In step S <b> 805, the related information management unit 406 updates the processing state of the status management information 601 to “4 (acquiring related information)”. Next, in S806, the related information management unit 406 receives the related information from the related information DB 103 by using the URL, and records this in the RAM 303 or the like. In step S <b> 807, the related information management unit 406 updates the processing state of the status management information 601 to “5 (related information has been acquired)”.

  In step S <b> 808, the display processing unit 403 displays a moving image as related information in a superimposed manner on a captured image that is being displayed by the display device 210. Thereby, augmented reality display is realized. When the related information is information for generating an electronic video, the display processing unit 403 generates an electronic video from the related information and displays the electronic video superimposed on the captured image. In step S809, the display processing unit 403 updates the processing state to “6 (displaying)”. Next, the display processing unit 403 confirms whether or not the reproduction of the moving image as the related information has ended. When the reproduction of the moving image is completed (Yes in S809), the display processing unit 403 advances the process to S812. In step S812, the display processing unit 403 updates the processing state to “7 (displayed)”. This completes the superimposed display process.

  Next, processing when the detected object is a redetected object will be described with reference to FIG. In step S901, the object management unit 405 identifies the object identifier of the object detected in step S704. In step S902, the object management unit 405 compares the object identifier 602 of the situation management information 601 with the identified object identifier. If the two match (Yes in S902), the object management unit 405 advances the process to S903. If the two do not match (No in S902), the object management unit 405 proceeds to S706, and proceeds with the process assuming that a new object has been recognized.

  In step S903, the object management unit 405 confirms the processing state. If the processing state is “7 (displayed)” (Yes in S903), the object management unit 405 advances the process to S904. If the processing state is other than “7 (displayed)” (No in S903), the object management unit 405 does not execute the processing in S706 to S713 on the already detected object. The superimposed display process is terminated without performing the process. On the other hand, in S904, the object management unit 405 deletes the status management information 601. The superimposed display process is thus completed. As described above, when the same object is detected again, the tablet terminal 101 does not execute the superimposed display process from the beginning, but continues the process being executed.

  FIG. 10 is a flowchart showing monitoring processing by the tablet terminal 101. The monitoring process is a process for monitoring the progress of the superimposed display process. In step S1001, when the object is detected in step S704, the monitoring unit 407 starts monitoring whether the object is continuously detected (recognized). If the object is no longer detected (Yes in S1001), the monitoring unit 407 advances the process to S1002. In step S1002, the monitoring unit 407 confirms whether the status management information 601 is recorded. If the status management information 601 is recorded (Yes in S1002), the monitoring unit 407 advances the process to S1003. If the status management information 601 is not recorded (No in S1002), the monitoring unit 407 ends the monitoring process.

  In step S <b> 1003, the monitoring unit 407 confirms the recognition state 603 of the status management information 601. If the recognition state 603 is “1 (detection)” (Yes in S1003), the process proceeds to S1006. When the recognition state 603 is other than “1” (No in S1003), the monitoring unit 407 advances the process to S1004. In S1004, the monitoring unit 407 confirms the processing state 604. When the process state 604 is “7 (displayed)” (Yes in S1004), the monitoring unit 407 advances the process to S1006. In addition, when the processing state 604 is other than “7” (No in S1004), the monitoring unit 407 advances the processing to S1005.

  In step S <b> 1005, the monitoring unit 407 ends the monitoring process without instructing the superimposed display processing unit 410 to stop the superimposed display process. That is, in this case, the superimposed display processing unit 410 continues to perform the superimposed display process. For example, when “2 (object information is being acquired)” is set in the processing state 604 during the execution of S1005 and the processing of S802 is being executed, the object management unit 405 continues the processing of S802. If the processing of S802 and S803 is completed and “3 (object information acquired)” is set in the processing state 604, the object management unit 405 advances the processing to S804. Further, for example, when “5 (displayed)” is set in the processing state 604 at the time of executing S1005 and the related information is being superimposed on the display device 210, the object management unit 405 performs the superimposed display processing. continue.

  As another example, in the case where related information is displayed in a superimposed manner in S1005, that is, in the case of a superimposed display while the object detection is suspended, the display processing unit 403 changes the display form of the superimposed display to continue object detection. You may make it different from the display form of superposition display in the inside. Specifically, the display processing unit 403 may change the color of the related information pop-up display. As another example, the display processing unit 403 may additionally display an icon indicating that the object cannot be recognized. As a result, the user can easily notice that the photographing field of view has changed.

  On the other hand, in S1006, the monitoring unit 407 deletes the status management information 601. In step S <b> 1007, the monitoring unit 407 instructs the superimposed display processing unit 410 to stop the superimposed display process being executed. Furthermore, the monitoring unit 407 instructs the superimposed display processing unit 410 to restart the superimposed display process. In response to this, the superimposed display processing unit 410 stops the superimposed display process being executed, and restarts a new superimposed display process from S701. As described above, the monitoring unit 407 instructs to stop the superimposed display process when the reproduction display of the related information has been completed.

  Thus, in the information processing system according to the present embodiment, related information can be continuously displayed even when the detected object is out of the field of view. That is, the information processing system can stably superimpose and display related information of a captured image.

(Second Embodiment)
The information processing system according to the second embodiment performs the augmented reality display continuously when the detection of the object is interrupted only for the object selected by the user. Hereinafter, the difference between the information processing system according to the second embodiment and the information processing system according to the first embodiment will be described. FIG. 11 is a diagram illustrating an example of a data configuration of the instruction management information 1101. The instruction management information 1101 is information used by the superimposed display processing unit 410. The instruction management information 1101 includes an object identifier 1102 and a related information display instruction 1103. The object identifier 1102 records the identifier of the object selected by the user. In the related information display instruction 1103, information indicating whether or not an instruction to continue related information of the object is input from the user is recorded. Here, the continuation instruction is information for instructing the continuation display of related information corresponding to an object when the object is no longer detected. If a continuation instruction is input, “correct” is recorded in the related information display instruction 1103. If a continuation instruction is not input, “not” is recorded in the related information display instruction 1103. The

  12 and 13 are flowcharts showing the superimposed display process. In addition, the same number is attached | subjected to the process same as each process of the superimposition display process which concerns on 1st Embodiment. The tablet terminal 101 according to the second embodiment advances the process to S1201 after the process of S704. In step S1201, the display processing unit 403 displays an inquiry screen. FIG. 14 is a diagram showing a display example of the inquiry screen 1401. The inquiry screen 1401 is a screen for inquiring the user whether or not to continue the superimposed display of related information when an object is no longer detected. Thereafter, in S1202, the object management unit 405 confirms whether or not an input of a continuation instruction by the user has been received. When the object management unit 405 receives an input of a continuation instruction (Yes in S1202), the process proceeds to S1203.

  In step S1203, the object management unit 405 generates instruction management information 1101 indicating “correct”. Specifically, the object management unit 405 identifies the object identifier of the object detected in S407, and records this in the object identifier 1102. Further, “correct” is recorded in the related information display instruction 1103. Thereafter, the object management unit 405 records the generated instruction management information 1101 in the RAM 303 or the like, and then advances the process to S705. The object management unit 405 determines that a continuation instruction has not been accepted when a certain period of time has elapsed without accepting a display instruction for a certain period of time (No in S1202), and advances the process to S1204. In step S1204, the object management unit 405 generates instruction management information 1101 indicating “NO”. Thereafter, the object management unit 405 records the generated instruction management information 1101 in the RAM 303 or the like, and then ends the superimposed display process.

  FIG. 13 is a flowchart illustrating a process when the object detected in S704 is a rediscovered object. In addition, the same number is attached | subjected to the process same as the process which concerns on 1st Embodiment. In other words, if the detected object is a re-detected object in S704 (Yes in S704), the object recognition unit 404 advances the process to S1301. In step S1301, the object management unit 405 identifies an object identifier. In step S <b> 1302, the object management unit 405 confirms the related information display instruction 1103 in the instruction management information 1101. If “correct” is recorded in the related information display instruction 1103 (Yes in S1302), the object management unit 405 advances the process to S901. On the other hand, when “No” is recorded in the related information display instruction 1103 (No in S1302), the object management unit 405 ends the superimposed display process.

  FIG. 15 is a flowchart showing the monitoring process. In addition, the same number is attached | subjected to the process same as each process of the monitoring process which concerns on 1st Embodiment. In the monitoring process according to the second embodiment, if no object is detected in S1001 (Yes in S1001), the process proceeds to S1501. In step S1501, the monitoring unit 407 confirms the related information display instruction 1103 in the instruction management information 1101. If “correct” is recorded in the related information display instruction 1103 (Yes in S1501), the monitoring unit 407 advances the process to S1002. On the other hand, when “No” is recorded in the related information display instruction 1103 (No in S1501), the monitoring unit 407 ends the monitoring process. The remaining configuration and processing of the information processing system according to the second embodiment are the same as the configuration and processing of the information processing system according to other embodiments.

  As described above, the information processing system according to the second embodiment can select whether to continuously display related information when an object is no longer recognized, according to the user's intention. . That is, the information processing system can determine whether the change in the recognition state of the object is due to an unexpected change in the shooting field of view or the intentional change of the shooting field of view of the user by following a user instruction. It becomes possible. As a result, the information processing system can execute an appropriate process such that the display state of the related information is maintained in the former case and the display of the related information is ended in the latter case. Furthermore, the information processing system can improve the browsing property of information by suppressing unnecessary related information from being displayed.

  As a modification example of the information processing system according to the second embodiment, the timing of inquiring the user whether or not to display related information is not limited to the embodiment. For example, the information processing system may make an inquiry (display of an inquiry screen) to the user after receiving the related information or after the display of the related information is started.

(Third embodiment)
The information processing system according to the third embodiment stops the superimposition display process when a certain period of time elapses in a state in which the object displaying the related information is out of the field of view. Hereinafter, the difference between the information processing system according to the third embodiment and the information processing system according to the first embodiment will be described. FIG. 16 is a diagram illustrating a hardware configuration of the tablet terminal 101 according to the third embodiment. The tablet terminal 101 according to the third embodiment further includes a timer 1601 that performs time measurement.

  FIG. 17 is a flowchart showing the monitoring process according to the third embodiment. If the processing state 604 is other than “7” in S1004 (No in S1004), the monitoring unit 407 advances the processing to S1701. In step S <b> 1701, the monitoring unit 407 sets a waiting time T <b> 1 in the timer 1601. Here, T1 is a preset value and is recorded in the ROM 302 or the like. Furthermore, the monitoring unit 407 instructs the timer 1601 to start the time measurement for T1. The timer 1601 starts time measurement for T1 in accordance with the start instruction. Thereafter, the monitoring unit 407 advances the process to S1005, and further advances the process to S1702 without instructing the superimposed display processing unit 410 to stop the superimposed display process.

  In S1702, the monitoring unit 407 confirms whether T1 has elapsed. If T1 has elapsed (Yes in S1702), the monitoring unit 407 advances the process to S1705. If T1 has not elapsed (No in S1702), the monitoring unit 407 advances the process to S1703. In step S1703, the monitoring unit 407 confirms whether or not the object detected in step S705 has been detected again. If the object is detected again (Yes in S1703), the monitoring unit 407 advances the process to S1704. If the object has not been detected again (No in S1703), the monitoring unit 407 advances the process to S1702. Note that if a new object is detected, the monitoring unit 407 does not re-detect the object, and the process advances to S1702. In step S1704, the monitoring unit 407 instructs the timer 1601 to end the time measurement of T1, and ends the monitoring process.

  In step S1705, the monitoring unit 407 instructs the timer 1601 to end the time measurement of T1. In step S <b> 1706, the monitoring unit 407 instructs the superimposed display processing unit 410 to stop the superimposed display process being executed. In response to this, the superimposed display processing unit 410 stops the superimposed display process being executed. Next, in S1707, the monitoring unit 407 confirms the processing state. If the processing state is “5 (relevant information has been acquired)” or “6 (during display)” (Yes in S1707), the monitoring unit 407 advances the processing to S1708. If the processing state is other than “5” and “6” (No in S1707), the monitoring unit 407 advances the processing to S1709. In step S1708, the monitoring unit 407 deletes related information recorded in the RAM 303 or the like. In step S1709, the monitoring unit 407 deletes the status management information 601 recorded in the RAM 303 or the like. This is the end of the monitoring process. The remaining configuration and processing of the information processing system according to the third embodiment are the same as the configuration and processing of the information processing system according to other embodiments.

  As described above, the information processing system according to the third embodiment continues the superimposed display process that is being executed until a predetermined time has elapsed, and ends the superimposed display process when the predetermined time has elapsed. For example, when an object moves out of the shooting field of view due to an unexpected movement of the user carrying the tablet terminal 101, it is estimated that the object immediately returns to the shooting field of view. On the other hand, when a certain amount of time has passed while the object is out of the shooting field of view, it is estimated that the user intentionally removed the object from the shooting field of view. In response to this, the information processing system according to the third embodiment continues the superimposed display process in consideration of the time after the object is no longer detected. Thereby, the information processing system can perform an appropriate display process depending on whether or not the reason why the object disappeared from the captured image is due to the user's intention.

  A modification example of the information processing system according to the fourth embodiment will be described. When the tablet terminal 101 detects the object again before the elapse of T1 and continues the superimposed display process being executed, the display mode of the related information superimposed display is being continued without interruption. It may be different from the display form of the superimposed display.

(Fourth embodiment)
Similar to the information processing system according to the third embodiment, the information processing system according to the fourth embodiment controls the superimposed display processing in consideration of the elapsed time after the object is no longer detected. In the information processing system according to the fourth embodiment, the tablet terminal 101 resumes the superimposed display when the object is detected again after the superimposed display is stopped. Furthermore, when restarting the superimposed display, the tablet terminal 101 continuously reproduces and displays the related information from the stop position at the previous display time. Hereinafter, the difference between the information processing system according to the fourth embodiment and the information processing system according to the third embodiment will be described.

  18 and 19 are flowcharts showing the monitoring process according to the fourth embodiment. As illustrated in FIG. 18, after the processing of S1705, the monitoring unit 407 advances the processing to S1801. In step S1801, the monitoring unit 407 confirms the processing state. If the processing state is “6 (displaying)” (Yes in S1801), the monitoring unit 407 advances the process to S1901 (FIG. 19). If the processing state is other than “6” (No in S1801), the monitoring unit 407 advances the processing to S1706. Then, after the processing of S1706, the monitoring unit 407 advances the processing to S1802. In step S1802, the monitoring unit 407 confirms the processing state. If the processing state is “5 (relevant information has been acquired)” (Yes in S1802), the monitoring unit 407 advances the process to S1708. If the processing state is other than “5” (No in S1802), the monitoring unit 407 advances the process to S1709.

  In S1901 shown in FIG. 19, the monitoring unit 407 records the reproduction position displayed when the related information display is stopped, that is, the stop position, in the display position information 605 of the status management information 601 (recording process). Next, in S1902, the monitoring unit 407 sets a new waiting time T2 in the timer 1601. Here, T2 is a preset value and is recorded in the ROM 302 or the like. Note that T1 and T2 may be equal or different. Furthermore, the monitoring unit 407 instructs the timer 1601 to start measuring time T2. The timer 1601 starts measuring time T2 in accordance with the start instruction. In step S <b> 1903, the monitoring unit 407 instructs the superimposed display processing unit 410 to stop superimposed display of related information. Then, the superimposed display processing unit 410 stops the superimposed display according to the instruction.

  Next, in S1904, the monitoring unit 407 confirms whether T2 has elapsed. If T2 has elapsed (Yes in S1904), the monitoring unit 407 advances the process to S1909. If T2 has not elapsed (No in S1904), the monitoring unit 407 advances the process to S1905. In step S1905, the monitoring unit 407 confirms whether or not the object detected in step S705 has been detected again. If the object is detected again (Yes in S1905), the monitoring unit 407 advances the process to S1906. If the object is not detected again (No in S1905), the monitoring unit 407 advances the process to S1904. In step S1906, the monitoring unit 407 instructs the timer 1601 to stop the time measurement at T2. The timer 1601 stops time measurement according to the instruction. Next, in S1907, the monitoring unit 407 reads the display position information 605 of the status management information 601, and determines the stop position as the reproduction start position. Next, the monitoring unit 407 restarts the display information superimposition display (S808) from the determined reproduction start position. This is the end of the monitoring process.

  On the other hand, in S1909, the monitoring unit 407 instructs the timer 1601 to stop the time measurement of T2. The timer 1601 stops time measurement according to the instruction. In step S1910, the monitoring unit 407 deletes related information recorded in the RAM 303 or the like. Next, in S1911, the monitoring unit 407 deletes the status management information recorded in the RAM 303 or the like. This is the end of the monitoring process. Note that other configurations and processes of the information processing system according to the fourth embodiment are the same as the configurations and processes of the information processing systems according to other embodiments.

  As described above, in the information processing system according to the present embodiment, the tablet terminal 101 can skip a portion that has already been displayed and start playback and display of the remaining portion from the previous stop position. As a result, the user does not need to see the same information repeatedly, and information can be obtained efficiently.

  As a first modification of the information processing system according to the fourth embodiment, whether the tablet terminal 101 starts reproduction display from the previous stop position or performs reproduction display again from the top of the related information. It is good also as switching according to the instruction | indication from a user.

  As a second modification, the related information may be a still image or text. Assume that the related information is still images that are sequentially displayed in a scroll format. In this case, the monitoring unit 407 records information for identifying the displayed area in the display position information 605 of the status management information 601 in S1901. If the related information is text that is sequentially displayed in a scroll format, the monitoring unit 407 records the number of displayed lines in the display position information 605.

(Fifth embodiment)
In the information processing system according to the fifth embodiment, the tablet terminal 101 controls the superimposed display process based on the movement of the tablet terminal 101 main body. Hereinafter, the difference between the information processing system according to the fifth embodiment and the information processing system according to another embodiment will be described. FIG. 20 is a diagram illustrating a hardware configuration of the tablet terminal 101. The tablet terminal 101 according to the fifth embodiment further includes an acceleration sensor 2001 that detects the movement and posture of the tablet terminal 101.

  FIG. 21 is a flowchart illustrating the monitoring process according to the fifth embodiment. In step S <b> 2101, the monitoring unit 407 instructs the acceleration sensor 2001 to start measurement. The acceleration sensor 2001 starts measuring the amount of movement in accordance with the start instruction. Thereafter, the monitoring unit 407 advances the process to S1001. In S1001, the monitoring unit 407 advances the process to S2102 when no object is detected (Yes in S1001). In S2102, the monitoring unit 407 acquires the amount of movement as a measurement result by the acceleration sensor 2001 (acquisition processing). Next, in S2103, the monitoring unit 407 compares the measurement value obtained as a measurement result with the threshold values S1 and S2. Here, the thresholds S1 and S2 are predetermined values and are recorded in the ROM 302 or the like. Note that S2 is a smaller value than S1.

  When the measured value is smaller than the threshold value S1 and larger than the threshold value S2 (Yes in S2103), the monitoring unit 407 advances the process to S1002. On the other hand, when the measured value is not less than the threshold value S1 or not more than the threshold value S2 (No in S2103), the monitoring unit 407 advances the process to S2104 and continues the superimposed display process. When a steep movement that is equal to or greater than the threshold value S1 is measured, the object deviating from the shooting field of view is caused by an unintended movement of the tablet terminal 101 by the user and not by the user's intention. Guessed. In addition, when the movement is small and the threshold value S2 or less, the object is not detected from the captured image because there is a change in the external environment such as when the shielding object moves between the tablet terminal 101 and the object. It is assumed that this is the cause and not the user's intention. Therefore, in these cases, the tablet terminal 101 continues the superimposed display process. The remaining configuration and processing of the information processing system according to the fifth embodiment are the same as the configuration and processing of the information processing system according to other embodiments.

  As described above, the information processing system according to the fifth embodiment estimates whether the reason for the disappearance of the object from the captured image is due to the user's intention, and performs an appropriate display process according to the estimation result. It can be carried out.

<Other embodiments>
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) that records a program code (computer program) of software that implements the functions of the above-described embodiments is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium.
Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Moreover, the program code of software may be supplied to the system or apparatus via a network.

  An operating system (OS) operating on the computer performs part or all of the actual processing based on the instruction of the program code read by the computer. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

Furthermore, it is assumed that the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU of the function expansion board or function expansion unit performs part or all of the actual processing, and the processing of the above-described embodiment is realized by the processing. Needless to say.
When the present invention is applied to the storage medium, the storage medium stores program codes corresponding to the flowcharts described above.

  As described above, according to each of the above-described embodiments, an object is to stably superimpose and display related information of a captured image in an augmented reality display system using a device that performs unstable position fluctuation. be able to.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims.・ Change is possible.

101 Tablet terminal 102 Object DB
103 Related information DB

Claims (14)

A recognition process for recognizing an object from a captured image being displayed on a display device, an extraction process for extracting related information related to the object, and a display for displaying the related information superimposed on the captured image and displayed on the display device Superimposing display means for performing superimposing display processing including processing;
Monitoring means for monitoring whether or not the object recognized in the recognition process is continuously recognized,
The information processing apparatus according to claim 1, wherein the superimposed display unit continues or stops the superimposed display process based on a progress of the superimposed display process when the object is no longer recognized by the monitoring unit.   2. The information processing according to claim 1, wherein when the object is no longer recognized, the superimposed display unit continues the superimposed display process when the object is being displayed on the display device. apparatus. The related information is a moving image,
The superimposing display unit stops the superimposing display process when the reproduction display of the moving image as the related information is finished when the object is no longer recognized. The information processing apparatus described.   The information processing apparatus according to any one of claims 1 to 3, wherein the superimposed display unit stops the superimposed display process when a predetermined time has elapsed since the object is no longer recognized. The related information is a moving image,
It further has a recording means for recording the stop position of the related information displayed when the superimposed display is stopped,
5. The superimposing display unit reproduces and displays the related information from the stop position when an object that is no longer recognized is recognized again after the superimposing display process is stopped. Information processing device. The related information is a still image or text information,
The superimposed display means sequentially displays the related information part by part in the display process,
It further has a recording means for recording the stop position of the related information displayed when the superimposed display is stopped,
The information according to claim 4, wherein the superimposed display unit displays the related information in a superimposed manner from the stop position when recognizing an object that is no longer recognized after the superimposed display process is stopped. Processing equipment. Further comprising an acquisition means for acquiring a movement amount of the information processing apparatus;
The superimposition display unit continues the superimposition display process when the amount of movement is larger than a first threshold when the object cannot be recognized. The information processing apparatus described in 1.   The superimposed display means continues the superimposed display process when the amount of movement is smaller than a second threshold value smaller than the first threshold value when the object cannot be recognized. The information processing apparatus according to claim 7. Further comprising an acquisition means for acquiring a movement amount of the information processing apparatus;
The superimposition display means continues the superimposition display processing when the amount of movement is smaller than a threshold when the object cannot be recognized. Information processing device. And further has a receiving means for receiving an instruction to continue the superimposed display process from the user,
The information processing apparatus according to claim 1, wherein the monitoring unit monitors the superimposed display process when the receiving unit receives the continuation instruction. A recognition process for recognizing an object from a captured image being displayed on a display device, an extraction process for extracting related information related to the object, and a display for displaying the related information superimposed on the captured image and displayed on the display device Superimposing display means for performing superimposing display processing including processing;
Monitoring means for monitoring whether or not the object recognized in the recognition process is continuously recognized,
The superimposing display means continues or stops the superimposing display process based on the progress of the superimposing display process when the object is no longer recognized by the monitoring means. An information processing method executed by an information processing apparatus,
A recognition process for recognizing an object from a captured image being displayed on a display device, an extraction process for extracting related information related to the object, and a display for displaying the related information superimposed on the captured image and displayed on the display device A superimposed display step for performing superimposed display processing including processing;
A monitoring step of monitoring whether or not the object recognized in the recognition process is continuously recognized,
In the superimposed display step, when the object is no longer recognized in the monitoring step, the superimposed display process is continued or stopped based on the progress of the superimposed display process. An information processing method executed by an information processing system,
A recognition process for recognizing an object from a captured image being displayed on a display device, an extraction process for extracting related information related to the object, and a display for displaying the related information superimposed on the captured image and displayed on the display device A superimposed display step for performing superimposed display processing including processing;
A monitoring step of monitoring whether or not the object recognized in the recognition process is continuously recognized,
In the superimposed display step, when the object is no longer recognized in the monitoring step, the superimposed display process is continued or stopped based on the progress of the superimposed display process. Computer
A recognition process for recognizing an object from a captured image being displayed on a display device, an extraction process for extracting related information related to the object, and a display for displaying the related information superimposed on the captured image and displayed on the display device Superimposing display means for performing superimposing display processing including processing;
Function as monitoring means for monitoring whether or not the object recognized in the recognition process is continuously recognized;
The superimposition display unit continues or stops the superimposition display process based on the progress of the superimposition display process when the object is no longer recognized by the monitoring unit.

Images