JP6197405B2 - Information processing apparatus, information processing system, and information processing program - Google Patents

Description

  The present invention relates to an information processing apparatus, an information processing system, and an information processing program.

  2. Description of the Related Art Today, a technique called augmented reality (AR) is known in which information generated by a computer device or the like is superimposed on information given to perception from an actual environment to give supplementary information. Regarding this augmented reality, for example, Patent Document 1 (Japanese Patent Laid-Open No. 2007-228315) discloses a basic technique relating to composition of a stationary object and an image as a construction construction and inspection support device for a structure in a building. It is disclosed.

  When providing augmented reality, for example, a head mounted display (HMD) is used. In this case, the virtual space image generated in accordance with the position and orientation of the moving image capturing apparatus is superimposed on the real space image captured by the moving image capturing apparatus and displayed on the HMD. The virtual space image is, for example, a virtual object or character information drawn by computer graphics.

  Such an information processing apparatus is applied to surgical support for displaying a state of a body in a superimposed manner on a patient's body surface. In addition, the information processing apparatus is applied to a navigation system that superimposes and displays a name or guidance of a famous building or the like included in an image of a real space obtained by capturing an image of a city area as a virtual space image. In addition, as disclosed in Patent Document 2 (Japanese Patent Application Laid-Open No. 2004-145657), the information processing apparatus is also applied to a virtual museum system that displays a virtual space such as a building at the time superimposed on the real space. Has been.

  Further, the information processing apparatus recognizes an object moving in the real space, which is disclosed in Patent Document 3 (Japanese Patent Laid-Open No. 2009-289035), and displays a virtual image on the moving object in real time. Applied to image display system. This image display system, for example, synthesizes the image of the building at that time against the foundation stones existing in the visited ruins. Further, the image display system synthesizes the costumes of the time with the images of the surrounding tourists. This makes it possible to virtually reproduce the buildings at the time and the appearance of people at the time.

  However, the conventional information processing apparatus applied to Patent Literature 1 to Patent Literature 3 and the like in this way is a virtual image (additional information) with respect to a monitoring object (what is in the field of view) that the user actually sees. ). For this reason, there is a problem that it is difficult to superimpose additional information on an object that cannot be seen from the current position due to an obstacle such as a ceiling, a wall, or another building, and also a distance. It was.

  SUMMARY An advantage of some aspects of the invention is that it provides an information processing apparatus, an information processing system, and an information processing program that display additional information for a monitoring target that cannot be seen from the current position.

In order to solve the above-described problems and achieve the object, the present invention provides target image generation means for generating a monitoring target image of a monitoring target corresponding to the position and direction of the image display device, the position of the image display device, and Obstacle image generation means for generating an obstacle image of an obstacle that obstructs the visual recognition of the monitoring object corresponding to the direction, and the monitoring target on the obstacle image so as to correspond to the position and direction of the image display device The image is superimposed so that the image can be visually recognized, and additional information corresponding to the monitoring object is superimposed to generate a superimposed image . The relative distance between the image display device and the monitoring object from the position of the image display device is equal to or less than a predetermined value. when it is detected that a distance, and the superimposed image generating means for generating only the additional information to stop the generation of the superimposed image, the superimposed image, or a transmission means for transmitting only the image display device the additional information A.

  According to the present invention, it is possible to display additional information for a monitoring object that is not visible from the current position.

FIG. 1 is a diagram illustrating a state in which an obstruction such as a monitored object that is a store desired by a user and other stores existing in front of the monitored object are viewed from diagonally above. FIG. 2 is a diagram illustrating a state where the user is looking at the obstacle from a viewpoint in which the corner portion of the obstacle is the front. FIG. 3 is a diagram illustrating a display example of a superimposed image of the information processing system according to the first embodiment. FIG. 4 is a system configuration diagram of the information processing system according to the first embodiment. FIG. 5 is a time chart for explaining processing of each unit of the information processing system according to the embodiment. FIG. 6 is a schematic diagram for explaining processing of each unit of the information processing system according to the first embodiment. FIG. 7 is a flowchart illustrating the operation of the image display apparatus provided in the information processing system according to the first embodiment. FIG. 8 is a flowchart illustrating the operation of the server device provided in the information processing system according to the first embodiment. FIG. 9 is a flowchart illustrating the operation of the monitoring device provided in the information processing system according to the first embodiment. FIG. 10 is a system configuration diagram of an information processing system according to the second embodiment. FIG. 11 is a diagram illustrating a display example of a superimposed image of the information processing system according to the second embodiment. FIG. 12 is a flowchart illustrating the operation of the image display apparatus provided in the information processing system according to the second embodiment. FIG. 13 is a flowchart illustrating the operation of the server apparatus provided in the information processing system according to the second embodiment. FIG. 14 is a flowchart illustrating the operation of the monitoring target selection unit provided in the information processing system according to the second embodiment. FIG. 15 is a flowchart illustrating the operation of the monitoring device provided in the information processing system according to the second embodiment. FIG. 16 is a system configuration diagram of an information processing system according to the third embodiment. FIG. 17 is a diagram for explaining the relationship between the relative distance and the relative angle with respect to the monitoring object. FIG. 18 is a flowchart illustrating the operation of the image display apparatus provided in the information processing system according to the third embodiment. FIG. 19 is a flowchart illustrating the operation of the server device provided in the information processing system according to the third embodiment. FIG. 20 is a flowchart illustrating the operation of the monitoring device provided in the information processing system according to the third embodiment.

  Hereinafter, embodiments of an information processing apparatus, an information processing system, and an information processing program will be described in detail with reference to the accompanying drawings.

(First embodiment)
The information processing system according to the first embodiment adds and displays additional information to the image of the monitoring target 1 that exists in the real space but cannot be seen from the current position due to the influence of an obstacle or the like. Specifically, for example, in a state where the user is on the first floor of a department store building where a plurality of stores are mixed, there are a plurality of stores of interest, but the floor, walls, and other stores become obstacles, Further, it is assumed that no store can be visually recognized from the current position due to a long distance. In this case, since the user is in a position where the store cannot be visually recognized, it is difficult to grasp the current state of the store such as the degree of congestion of the store and the stock of the product.

  In such a situation, the information processing system according to the first embodiment displays an image of a desired store that is made visible by virtually removing an obstacle existing between the current position and the desired store. Display on a portable wearable display. Further, the information processing system displays, for example, additional information such as information indicating the distance and direction from the current time, information indicating the current state of the store, information indicating recommended items, and the like superimposed on the image of the store.

  That is, the information processing system of the first embodiment connects a server device, a portable wearable display device, and a determination device to each other via a network. The determination device monitors the monitoring object 1 that is difficult to view from the current position of the user, determines the position and the degree of congestion of the monitoring object 1 in real time, and provides the additional information as additional information. In addition, the server device virtually removes an obstacle between the monitoring object 1 and superimposes additional information on the model of the monitoring object 1 registered in the database, and transmits it to the portable wearable display device. To do.

  In the case of the information processing system according to the first embodiment, the image of the monitoring target 1 and the additional information may be displayed with an accuracy that allows a certain degree of direction to be understood. For this reason, the display accuracy required by a general augmented reality (AR) technology is not so necessary. For this reason, it is also possible to eliminate the need for an image capturing unit that captures a scene currently being viewed by the user, which is necessary for detecting the AR marker or the feature point of the building.

  Hereinafter, the information processing system according to the first embodiment will be described in detail. FIG. 1 shows a state in which, for example, a monitoring target 1 that is a store desired by the user and an obstacle 2 such as another store that exists in front of the monitoring target 1 are viewed from an oblique viewpoint A. Yes. FIG. 2 shows a situation where the user is viewing the obstacle 2 from the viewpoint B with the corner of the obstacle 2 as the front. As can be seen from FIG. 1, in this case, the monitored object 1 and the obstacle 2 are substantially the same height. Further, a cylindrical monitoring target 1 is present behind the obstacle 2. Moreover, in this example, the monitoring target object 1 which is a store desired by the user is an additional information addition target.

  In the case of the viewpoint B in FIG. 2, the obstacle 2 having the same height exists between the user and the monitoring target 1, and the monitoring target 1 cannot be seen from the user. As a situation in which the monitoring target 1 cannot be seen, for example, in a shopping center, there is a case where a desired store that becomes the monitoring target 1 in this case is present in front of another store and an obstacle such as a wall. In addition, there is a case where a desired store, which is the monitoring object 1 in this case, exists on a floor different from the floor corresponding to the current position with an obstacle called a floor interposed therebetween. In addition, we are riding a narrow alley surrounded by a stone wall higher than my line of sight, and there is a cross road several meters ahead, but a car or pedestrian running in a direction perpendicular to the direction of travel (= in this case) The monitored object 1) may not be visible due to obstacles such as stone walls.

  In such a case, the information processing system of the embodiment sees or removes the obstacle 2 between the monitoring object 1 and the monitoring object ahead of the obstacle 2 as shown in FIG. 1 is displayed. In addition, the information processing system displays additional information such as characters and images corresponding to the monitoring target object 1 superimposed on the monitoring target object 1. Thereby, while being able to visually recognize the monitoring target object 1 which exists in the position where visual recognition is difficult from a user's present position, the present condition (additional information) of the monitoring target object 1 can be grasped | ascertained. As described above, the information processing system according to the embodiment enables display of information on the monitoring target 1 that is difficult to view from the current position.

  FIG. 4 shows a system configuration diagram of the information processing system according to the first embodiment. As shown in FIG. 4, the information processing system is configured by connecting an image display device 11 and an information providing device 12 to each other via a network (NW) 13 such as a wireless communication line.

  The information providing device 12 includes a server device 14, a superimposed image database 15, and a monitoring device 16. The server device 14 includes a superimposed image extraction unit 17, a display image transmission unit 18, a superimposed image adjustment unit 19, an image composition unit 20, and an additional information reception unit 21. The superimposed image extraction unit 17 is an example of a target image generation unit. The superimposed image adjusting unit 19 is an example of an obstacle image generating unit and a superimposed image generating unit. The display image transmission unit 18 is an example of a transmission unit. The superimposed image database 15 includes a simple 3D (Three Dimensions) model data storage unit 22. The monitoring device 16 includes a storage unit such as a CPU, a hard disk drive, a ROM, and a RAM, as well as an additional information storage unit 24, a monitoring position holding unit 25, an additional information transmission unit 26, an additional information generation unit 27, and an object monitoring unit 28. , A relative position calculator 29 and a positioning data receiver 30.

  The image display device 11 displays an image with additional information added. The image display device 11 includes a position / direction acquisition unit 31, a positioning data transmission unit 32, a display image reception unit 33, an image output unit 34, and the like.

  Note that each unit of the image display device 11, the server device 14, the superimposed image database 15, and the monitoring device 16 may be realized by software by an information processing program or the like to be described later, or may be realized in whole or in part by hardware. May be.

  It is desirable that the image display device 11 is of a size that can be carried by the user, and preferably wearable such as an HMD (Head Mounted Display) or a retina operation display. In addition, a portable terminal device having a display unit and a communication function may be used, such as a so-called tablet terminal device or a portable game machine.

  The position / direction acquisition unit 31 includes, for example, a GPS sensor and a six-axis gyro sensor, and generates positioning data indicating the current position and direction of the image display device 11 (= user's current position and line-of-sight direction). To do. GPS is an abbreviation for “Global Positioning System”. The positioning data transmission unit 32 transmits the positioning data acquired by the position / direction acquisition unit 31 to the monitoring device 16 in real time via the network 13. The display image receiving unit 33 receives a superimposed image including additional information generated by the server device 14.

  The image output unit 34 is a display device such as a liquid crystal display device that displays the superimposed image received by the display image receiving unit 33 in real time. The display of the image may basically be permutation synthesis or transmission synthesis. However, in the case of transmission composition, it is assumed that output is performed with a transmittance of 0%. The replacement synthesis is a synthesis method in which a part of an actual image is removed and replaced with the synthesized image at the same position. The transparent composition is a composition method in which a part of an actual image is left as it is and the images after composition are superimposed. In the case of transmission composition, this is a display method in which a part of an actual image can be seen through at a constant ratio (hereinafter referred to as “transmittance”). Here, the transmittance of 0% is synonymous with “substitution synthesis”. An image display apparatus using optical means is premised on transmission composition.

  The positioning data receiving unit 30 of the monitoring device 16 receives the positioning data transmitted from the image display device 11. Although the object monitoring unit 28 is difficult to visually recognize from the current position of the user, the object monitoring unit 28 images the monitoring object 1 for which additional information is to be displayed and converts it into digital data. As the object monitoring unit 28, a number of surveillance camera devices installed for crime prevention can be used.

  The monitoring position holding unit 25 holds the relative position, angle, and absolute position information of the object monitoring unit 28 with respect to the monitoring object 1. In addition, although it is an example, each information of a relative position, an angle, and an absolute position is a fixed value, respectively. The relative position calculation unit 29 determines the relative position from the monitoring object 1 based on the positioning data transmitted from the image display device 11 and the relative position, angle, and absolute position information held by the monitoring position holding unit 25. Then, it is calculated whether the image display device 11 is facing the direction of the monitoring object 1.

  The additional information generation unit 27 is based on real-time imaging information and positioning data, preset other information such as recommendation information, real-time congestion degree, direction and distance of the monitoring target 1, and additional information on the monitoring target 1 recommendation information. Etc. are generated as an image. The additional information storage unit 24 stores additional information used for generating the additional information image. The additional information transmission unit 26 transmits the generated additional information and the calculated relative position information to the server device 14.

  The simple 3D model data storage unit 22 of the superimposed image database 15 stores a 3D model of the monitoring target 1 for which additional information is to be provided, other characters, image information, and the like. The 3D model or the like of the monitoring object 1 can be taken out and edited at any time.

  The superimposed image extraction unit 17 of the server device 14 acquires a 3D model of the monitoring target 1 from the superimposed image database 15. The additional information receiving unit 21 receives and accumulates additional information and relative position information transmitted from the monitoring device 16. The superimposed image adjustment unit 19 adjusts the angle of the 3D model (superimposed image) extracted from the accumulated image database 15 based on the relative position information received from the monitoring device 16 and converts it into a two-dimensional image. .

  The image composition unit 20 has a white-painted image for virtually erasing the obstacle 2 between the monitoring object 1 and the superimposition image adjusted by the superimposition image adjustment unit 19, and the monitoring The additional information generated by the device 16 is combined. The display image transmission unit 18 transmits information indicating whether the image display device 11 is facing the monitoring target 1 from the positioning data of the image display device 11 and a superimposed image generated by the image composition unit 20. have. With such a configuration, the information processing system according to the first embodiment provides the user with information on the monitoring target 1 that cannot be seen due to the influence of the obstacle 2.

  FIG. 5 shows a time chart for explaining processing of each part of the information processing system according to the embodiment. FIG. 6 is a schematic diagram for explaining processing of each unit of the information processing system according to the embodiment. First, in step S1 of the time chart of FIG. 5, the position / direction acquisition unit 31 of the image display device 11 measures the current position and direction of the image display device 11 in real time to generate positioning data. 5 and 6, the positioning data transmission unit 32 transmits the positioning data to the monitoring device 16.

  The positioning data receiving unit 30 of the monitoring device 16 receives positioning data in step S3. In step S <b> 4, the monitoring device 16 analyzes the current state of the captured image of the monitoring target 1 captured by the target monitoring unit 28. In step S5, the monitoring device 16 calculates the relative position between the image display device 11 and the monitoring target 1 from the received positioning data by the relative position calculation unit 29. Moreover, the monitoring apparatus 16 produces | generates the image of the additional information, such as the real-time condition and recommendation information of the monitoring target object 1 imaged by the target object monitoring part 28 in step S6. As additional information, as an example, as shown in FIG. 6, the distance from the image display device 11 to the monitoring object 1 (remaining ○ m), information indicating the current state of the monitoring object 1 (congestion degree □%), And the recommendation information (during time sale) of the monitoring object 1 is generated. 5 and 6, such additional information is transmitted to the server device 14 together with the relative position information calculated by the relative position calculation unit 29.

  The additional information receiving unit 21 of the server device 14 receives the additional information and the relative position information transmitted from the monitoring device 16 in step S8. 5 and 6, the superimposed image extraction unit 17 acquires simple 3D model data of the monitoring object 1 registered in advance in the superimposed image database 15.

  In step S10, the superimposed image adjustment unit 19 of the server device 14 viewed the 3D model of the monitoring target 1 in the virtual world from the direction of the viewpoint of the user indicated by the relative position information received together with the additional information from the monitoring device 16. The appearance of the time is calculated and converted to a 2D plane image. Then, the superimposed image adjustment unit 19 adjusts the display position of the 2D image of the monitoring target 1 within the display area of the image display device 11. The superimposed image adjustment unit 19 is based on the premise that the obstacle 2 between the image display device 11 and the monitoring target 1 can be seen through and the size of the simple 3D model of the monitoring target 1 is constant. Adjust the display position.

  Next, in step S11, the image composition unit 20 of the server device 14 has a white-painted image as a background of the entire screen that is previously held as a default, an image of the monitoring object 1 converted into a 2D planar image, and a monitoring device. The additional information received from 16 is combined so as to be image data of one screen, and superimposed image data is generated. The display image transmission unit 18 of the server device 14 transmits the superimposed image data to the image display device 11 via the network 13 such as a wireless communication network in step S12 of FIG. 5 and FIG.

  The display image receiving unit 33 of the image display device 11 receives the superimposed image data in step S13. In step S14, the image output unit 34 such as a liquid crystal display device displays an image corresponding to the received superimposed image data on the display screen. Accordingly, for example, as shown in FIG. 3 or FIG. 6, the obstacle 2 becomes the background of the white-painted image, and the monitoring object 1 is clearly displayed in a state where the obstacle 2 is transparent, and the monitoring object 1 An image on which additional information corresponding to is displayed is displayed on the image display device 11.

  FIG. 7 is a flowchart showing the operation of the image display device 11 described above. An information processing program that enables each process shown in the flowchart of FIG. 7 is stored in a storage unit such as a ROM in the image display device 11. The CPU of the image display device 11 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format. DVD is an abbreviation for “Digital Versatile Disk”.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 7, when the main power supply of the image display apparatus 11 is turned on (step S21: Yes), the image display apparatus 11 acquires the current position and direction (step S22). Next, the image display device 11 transmits the acquired positioning data to the monitoring device 16 (step S23). Thereafter, when receiving the display image information (superimposed image) from the server device 14 (step S24: Yes), the image display device 11 outputs the received image from the output unit (step S25).

  On the other hand, the image display device 11 does not receive the display image information (superimposed image) (step S24: No), and when the monitoring object flag is turned on (step S27: Yes), the image is not updated. , Continue to display the previous image as it is. The monitoring object flag is a flag that is turned on when the monitoring device 16 determines that the image display device 11 is facing the monitoring object 1.

  On the other hand, when the monitoring device 16 determines that the image display device 11 is not facing the direction of the monitoring object 1, and the monitoring object flag is turned off (step S27: No), a superimposed image is displayed. There is no need. For this reason, the image display apparatus 11 displays the scene that the user actually sees as a transparent image as it is (step S28). The image display device 11 repeatedly executes the processing of each step described above until it is determined in step S26 that the main power of the image display device 11 has been turned off (step S26: No).

  FIG. 8 is a flowchart showing the operation of the server device 14 described above. An information processing program that enables each process shown in the flowchart of FIG. 8 is stored in a storage unit such as a ROM in the server device 14. The CPU of the server device 14 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 8, when the server device 14 receives the additional information image created by the monitoring device 16 (step S31: Yes), the server device 14 acquires simple 3D model data of the monitoring object 1 from the superimposed image database 15 (step S31). S32). The server device 14 calculates the appearance when the simple 3D model of the monitoring target 1 is viewed in the virtual world from the viewpoint direction of the relative position information calculated by the monitoring device 16 (step S33). It adjusts where to arrange (step S34). Thereafter, the created 2D image, additional information, and white background image are combined (step S35), and the superimposed image data generated as a result is transmitted to the image display device 11 (step S36).

  On the other hand, when only the above-described flag information is received without receiving the additional information created by the monitoring device 16 (step S31: No), the received flag information (monitoring object flag) is displayed. The image is transmitted to the image display device (step S38).

  FIG. 9 is a flowchart showing the operation of the monitoring device 16 described above. An information processing program that enables each process shown in the flowchart of FIG. 9 is stored in a storage unit such as a ROM in the monitoring device 16. The CPU of the monitoring device 16 executes the following processes according to this information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 9, when the main power supply is turned on (step S41: Yes), the monitoring device 16 starts imaging the monitoring target 1 (step S42). When receiving the positioning data from the image display device 11 (step S43: Yes), the monitoring device 16 calculates the relative position from the monitoring target 1 (step S44). Here, when it is determined that the image display device 11 does not face the direction of the monitoring object 1 (step S45: No), the monitoring device 16 turns off the monitoring object flag (step S54) and goes to the server device 14. Flag information is transmitted (step S55).

  On the other hand, when it is determined that the image display device 11 is facing the direction of the monitoring object 1 (step S45: Yes), the monitoring device 16 turns on the monitoring object flag (step S46) and takes an image. Information is digitized (step S47).

  Next, the monitoring device 16 analyzes the imaging data of the monitoring target 1 (step S48), extracts necessary real-time information, and compares it with the previous extraction result (step S49). If the previously extracted result and the situation (relative position or real-time situation) have changed significantly (step S50: Yes), new additional information is generated as an image (step S51) and transmitted to the server device 14 (step S52). In addition, when it does not differ so much from the result extracted last time (step S50: No), only the monitoring object flag information (ON) is transmitted to the server device (step S55). The monitoring device 16 repeatedly executes the processing of each step described above until it is determined in step S53 that the main power supply of the monitoring device 16 has been turned off.

  As is clear from the above description, the information processing system according to the first embodiment exists as a real space, but for an image of the monitoring object 1 that cannot be seen from the current position due to the influence of the obstacle 2 or the like. Thus, additional information that can be intuitively understood can be added and displayed.

  In addition, since the information processing system according to the first embodiment can superimpose additional information on a target that is not actually visible (in an obstacle or in a distant place), the accuracy of the superimposition place such as a general AR technique can be superimposed. Is not necessary, and it is sufficient to perform superposition with an accuracy in which a certain direction is known. For this reason, it is possible to eliminate the need for an imaging unit that captures a scene that is currently being viewed by the user, which is necessary for detecting an AR marker or a feature point of a building.

(Second Embodiment)
Next, an information processing system according to the second embodiment will be described. The information processing system according to the second embodiment is configured by adding the following units to the configuration of the information processing system according to the first embodiment described above. In the following description of the information processing system according to the second embodiment, only differences from the information processing system according to the first embodiment described above will be described, and redundant description will be omitted.

  FIG. 10 illustrates a system configuration diagram of an information processing system according to the second embodiment. As illustrated in FIG. 10, the image display device 11 of the information processing system according to the second embodiment includes an imaging data transmission unit 41, an actual scene imaging unit 42, in addition to the position / direction acquisition unit 31 to the image output unit 34. An information extraction target selection unit 43 and an enlargement / reduction adjustment unit 44 are provided.

  The real landscape image capturing unit 42 captures a scene that the user is currently viewing. The imaging data transmission unit 41 transmits the imaging data captured by the real landscape imaging unit 42 to the server device 14. The information extraction target selection unit 43 switches the monitoring target 1 from which additional information is to be extracted. The enlargement / reduction adjustment unit 44 adjusts the display size of the superimposed data.

  The information providing device 12 includes the server device 14, the superimposed image database 15, the first monitoring device 16 (corresponding to the monitoring device 16 in the first embodiment), the monitoring target selection device 50, and the second and third devices. Monitoring devices 52 and 53 are provided. The second and third monitoring devices 52 and 53 have the same configuration as that of the first monitoring device 16. The first to third monitoring devices 16, 52, and 53 monitor different monitoring objects. The monitoring target selection device 50 selects the output of any of the monitoring devices 16, 52, 53 that are monitoring the monitoring target selected by the user. Each monitoring device 16, 52, 53 is provided with a monitoring target determination unit 49 that determines whether or not the monitoring device has been selected.

  In the superimposed image database 15, a simple 3D model data storage unit 22 in which simple 3D model data of each monitoring object is stored, and a superimposed image selection unit 48 that selects simple 3D model data corresponding to the monitoring device selected by the user. Is provided.

  The server device 14 includes a data reception unit 45 and a superimposed image adjustment unit 19 to which a function for adjusting the scale of the superimposed image is added. Further, the server device 14 determines the size of the obstacle from the actual landscape image, and generates a white-painted image of the white-painted range calculation unit 46 that calculates the white-painted image range of the background and the calculated size and location. A white paint image generation unit 47 is provided.

  Each unit of the image display device 11, the server device 14, the superimposed image database 15, the monitoring devices 16, 52, 53, and the monitoring target selection unit 50 may be realized in software by an information processing program described later. , All or a part may be realized by hardware.

  In the case of the information processing system according to the second embodiment, the configuration in which these units are added allows the user to select a monitoring target (a target for which additional information is to be obtained), and the superimposed image The scale ratio can also be adjustable. Further, by acquiring an image of a real landscape captured by the image display device 11 carried by the user, it is possible to whiten the obstacle 2 within a certain range and show the monitoring target. In the case of the first embodiment described above, the entire screen of the image display device 11 is painted white, so that the surrounding landscape cannot be seen (sight close to VR). In the case of the information processing system according to the second embodiment, a part of the scenery that is actually seen is painted out (AR) to display the monitoring object.

  As described with reference to FIGS. 2 and 3, when the monitoring target 1 is viewed from the viewpoint B with the corner of the obstacle 2 in front, the entire screen of the image display device 11 has a white background. On the white background, the monitoring object 1 and additional information existing ahead of the obstacle 2 are displayed.

  On the other hand, in the case of the information processing system according to the second embodiment, the image display apparatus 11 also captures a real landscape, so that the range of obstacles cannot be seen in order to show the monitored object. You can consider what to do. For this reason, the server device 14 calculates a range to be a white-painted image, and generates a superimposed image of the white-painted image according to the calculation result. Thereby, as shown in FIG. 11, only a part of the real scenery image of the obstacle 2 can be erased, and the monitoring object 1 and the additional information can be displayed at the place where the real scenery image is erased.

  FIG. 12 is a flowchart showing the operation of the image display device 11 described above. An information processing program that enables each process shown in the flowchart of FIG. 12 is stored in a storage unit such as a ROM in the image display device 11. The CPU of the image display device 11 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format. DVD is an abbreviation for “Digital Versatile Disk”.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 12, steps S61 to S65 are processes newly added to the flowchart of FIG. In the case of the image display device 11 provided in the information processing system according to the second embodiment, after starting real-time imaging of a landscape (step S61), selecting an information acquisition target to obtain additional information (step S62), Scale information (enlarged / reduced data) having a size for displaying the monitoring target set in advance is acquired (step S63). Thereafter, the positioning data and the information acquisition target information are transmitted to the monitoring device (step S64). Further, real-time actual scenery image data and scale information on the image display device 11 side are transmitted to the server device 14 (step S65).

  FIG. 13 is a flowchart showing the operation of the server device 14 described above. An information processing program that enables each process shown in the flowchart of FIG. 13 is stored in a storage unit such as a ROM in the server device 14. The CPU of the server device 14 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 13, steps S71 to S75 are processes newly added to the flowchart of FIG. In the case of the server device 14 provided in the information processing system according to the second embodiment, the size corresponding to the scale before performing the 2D conversion of the simple 3D model based on the scale information acquired from the image display device 11. Then, the model is converted (step S71 and step S72).

  In the case of the server device 14 provided in the information processing system according to the second embodiment, real-time image data on the real-time image display device 11 side received from the previous image, or any monitoring device 16, When the additional information acquired from 52 and 53 has changed significantly (step S73: Yes), the white area of the background is recalculated (step S74), and the white area image of the recalculated area is generated again. (Step S75).

  FIG. 14 is a flowchart illustrating the operation of the monitoring target selection unit 50. An information processing program that enables each process shown in the flowchart of FIG. 14 is stored in a storage unit such as a ROM in the monitoring target selection unit 50. The CPU of the monitoring target selection unit 50 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  The monitoring object selection unit 50 provided in the information processing system according to the second embodiment is a mechanism added by monitoring a plurality of monitoring objects. When the monitoring target selection unit 50 receives target selection data notifying the change of the monitoring selection target from the image display device 11 (step S81: Yes), the monitoring target selection unit 50 switches the monitoring device that supplies the positioning data from the image display device 11 (step S81). S82). And the monitoring object selection part 50 supplies the positioning data from the image display apparatus 11 to the monitoring apparatus used as a switching destination (step S83).

  FIG. 15 is a flowchart showing the operation of the monitoring devices 16, 52 and 53 described above. An information processing program that enables each process shown in the flowchart of FIG. 15 is stored in a storage unit such as a ROM in each of the monitoring devices 16, 52, and 53. The CPUs of the monitoring devices 16, 52, and 53 execute the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 15, steps S91 and S92 are processing newly added to the flowchart of FIG. In the case of the monitoring devices 16, 52, and 53 provided in the information processing system according to the second embodiment, whether or not the own device is selected as the monitoring target by the monitoring target determination unit 49 shown in FIG. Is determined. When the monitoring target determination unit 49 determines that the own device is selected as the monitoring target (step S91: Yes), the simplified 3D model of the monitoring target monitored by the own device is displayed on the superimposed image database 15 side. Data selection is instructed (step S92). Thereby, since the simple 3D model data output from the superimposed image database 15 is uniquely determined, the server device 14 simply requests acquisition of the simple 3D model data, and the currently selected monitoring device is Simple 3D model data of the monitored object being monitored can be acquired.

  As is apparent from the above description, the information processing system according to the second embodiment is provided with a plurality of monitoring devices 16, 52, and 53 that monitor different monitoring objects, so that a desired monitoring device can be freely set. By switching, additional information of the monitoring target desired by the user can be obtained. Further, the scale of the superimposed image is adjusted on the image display device 11 side, and the server device 14 generates a scaled superimposed image adjusted on the image display device 11 side and transmits it to the image display device 11. Thereby, the scale ratio of the superimposed image displayed on the image display device 11 can be adjusted.

  In addition, the actual scenery image of the obstacle 2 is partially made a white-painted image, not the entire display screen of the image display device 11, and the superimposed image is displayed on the white-painted image. As a result, the real scenery image and the superimposed image can be merged, and the same effect as in the first embodiment can be obtained.

(Third embodiment)
Next, an information processing system according to the third embodiment will be described. In the information processing system according to the third embodiment, in addition to the configuration of the information processing system according to the second embodiment described above, the image display device 11 includes the following units. That is, the image display device 11 includes an obstacle relative distance detection unit 61 that detects a relative distance to the obstacle 2 that is closest to the scene that the user actually sees. In addition, when the value of the obstacle relative distance detection unit 61 becomes a certain distance or less, the image display device 11 automatically switches from the additional information superimposed image being displayed to a normal actual landscape image, and the user An output image switching unit 62 for preventing inconvenience of touching the obstacle 2 is provided.

  Each monitoring device 16, 52, 53 has an object detection range storage unit 65 in which numerical information indicating a range in which a monitoring object can be seen by the user in the real world is stored in advance for each monitoring object. Each of the monitoring devices 16, 52, and 53 determines that the superimposed image transmitted from the server device 14 to the image display device 11 side is only the additional information when it is determined that the user (image display device) has entered the range. It has the target object detection part 66 which controls so that it may become.

  The server device 14 includes an update frequency adjustment unit 64 that changes the update frequency of the superimposed image to be displayed in accordance with the distance to the monitoring target that is an addition target of the additional information.

  Each unit of the image display device 11, the server device 14, the superimposed image database 15, the monitoring devices 16, 52, 53, and the monitoring target selection unit 50 may be realized in software by an information processing program described later. , All or a part may be realized by hardware.

  FIG. 17 is a diagram for explaining the difference in the appearance of the monitoring target 1 according to the distance between the monitoring target 1 and the user (image display device 11). In FIG. 17, when the current position of the user moves from position D to position C, since the position C and position D are far from the monitoring object 1, the relative angle between the monitoring object 1 and the user is not much. It does not change greatly. That is, the angle difference θC−θD between the relative angle θC of the user positioned at the position C with the monitoring object 1 and the relative angle θD of the user positioned at the position D with the monitoring object 1 is small. For this reason, the appearance of the monitoring object 1 does not change so much between the position C and the position D.

  However, in FIG. 17, when the current position of the user moves from position B to position A, the position A and position B are close to the monitoring object 1, so the relative angle between the monitoring object 1 and the user is also growing. That is, the angle difference θA−θB between the relative angle θA of the user located at the position A with the monitoring object 1 and the relative angle θB of the user located at the position B with the monitoring object 1 becomes large. For this reason, the appearance of the monitoring object 1 also varies greatly between the position A and the position B.

  The information processing system according to the third embodiment adjusts the update frequency of the superimposed image according to the relative angle or the relative distance between the monitoring object 1 and the user. The information processing system according to the third embodiment switches the image to be displayed on the image display device 11 from the superimposed image to the actual landscape image when the distance between the monitoring target 1 and the user is equal to or less than a predetermined distance. The inconvenience of the user coming into contact with the monitoring object 1 is prevented.

  FIG. 18 is a flowchart showing the operation of the image display device 11 described above. An information processing program that enables each process shown in the flowchart of FIG. 18 is stored in a storage unit such as a ROM in the image display device 11. The CPU of the image display device 11 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format. DVD is an abbreviation for “Digital Versatile Disk”.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 18, steps S101 to S107 are processes newly added to the flowchart of FIG. The image display device 11 detects the relative distance from the obstacle 2 in front of the eyes together with the position and direction (positioning data) of the image display device 11 (step S101). If the relative distance to the obstacle 2 is equal to or less than a certain value (step S102: Yes), the obstacle flag is turned on (step S105), and the obstacle flag is transmitted to the currently selected monitoring device (step S106). . Then, the output image switching unit 62 switches the superimposed image that has been displayed so far to the actual landscape image (transparent image) captured by the actual landscape imaging unit 42 (step S107). Thereby, it can be made to recognize exactly that the user is approaching the obstacle 2. FIG.

  On the other hand, when the relative distance to the obstacle 2 is equal to or less than a certain value (step S102: No), the image display device 11 turns off the obstacle flag (step S103), and the obstacle flag information together with the positioning data and the target selection data. Is transmitted to the currently selected monitoring device (step S104).

  FIG. 19 is a flowchart showing the operation of the server device 14 described above. An information processing program that enables each process shown in the flowchart of FIG. 19 is stored in a storage unit such as a ROM in the server device 14. The CPU of the server device 14 executes the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 19, steps S111 to S113 are processes newly added to the flowchart of FIG. In the case of the information processing system according to the third embodiment, an object detection flag indicating whether or not the user exists in a range where the user can directly view the monitoring object 1 is transmitted from the currently selected monitoring apparatus. Specifically, when the user exists in the range where the monitoring target 1 can be directly seen, the target detection flag turned ON is transmitted, and when the user does not exist within the range where the monitoring target 1 can be seen directly, it is turned OFF. The detected object detection flag is transmitted.

  When the object detection flag is ON (step S111: Yes), it means that the user exists in a range where the monitoring object 1 can be directly seen. In this case, it is only necessary to show the monitoring object 1 itself without forcibly displaying the simple 3D model. Therefore, the server device 14 transmits only the additional information generated by the currently selected monitoring device to the image display device 11 side (step S113).

  Further, the server device 14 compares the positioning result (relative distance and relative angle) of the image display device 11 with the relative distance and relative angle when the superimposed image was transmitted to the image display device 11 side last time. If the comparison result is equal to or smaller than the predetermined threshold (step S112: Yes), the superimposed image displayed on the image display device 11 side does not need to be changed, and the process returns to step S31. If the comparison result exceeds a predetermined threshold value (step S112: No), a superimposed image is generated by superimposing additional information on the actual landscape image in which the range calculated as described above is painted white, and this is displayed as an image. Transmit to device 11. As a result, a superimposed image corresponding to the relative distance and relative angle with respect to the monitoring object 1 is displayed on the image display device 11.

  FIG. 20 is a flowchart showing the operation of the monitoring devices 16, 52, 53 described above. An information processing program that enables each process shown in the flowchart of FIG. 20 is stored in a storage unit such as a ROM in each of the monitoring devices 16, 52, and 53. The CPUs of the monitoring devices 16, 52, and 53 execute the following processes according to the information processing program.

  The information processing program may be configured to be provided by being recorded in a computer-readable recording medium such as a CD-ROM or a flexible disk (FD) in a file in an installable or executable format. . The information processing program may be configured to be recorded in a computer-readable recording medium such as a CD-R or a DVD in an installable or executable format.

  Furthermore, the information processing program may be stored on a computer device connected to the network 13 such as the Internet and provided by being downloaded via the network 13. Further, the information processing program may be provided or distributed via the network 13 such as the Internet.

  In the flowchart of FIG. 20, steps S121 to S125 are processes newly added to the flowchart of FIG. The monitoring devices 16, 52, 53 provided in the information processing system according to the third embodiment display an ON obstacle flag indicating that the relative distance between the image display device 11 and the obstacle 2 is less than a certain value. When received (step S121: Yes), this means the following. That is, in this case, on the image display apparatus 11 side, the display image is switched from the superimposed image to the real landscape image (see step S107 in FIG. 18). For this reason, since it is not necessary to generate a superimposed image on the server device 14 side, an obstacle flag is transmitted to the server device 14 side (step S122). Thereby, when the relative distance between the image display device 11 and the obstacle 2 becomes a certain distance or less, the server device 14 prevents the inconvenience of continuing to generate a superimposed image that is not displayed on the image display device 11 side. be able to.

  In addition, when the monitoring devices 16, 52, 53 detect from the calculation result of the monitoring device that the image display device 11 is present in the range where the monitoring target 1 can be seen (step S123: Yes), the target The detection range flag is switched to ON (step S124). Then, the monitoring devices 16, 52, and 53 transmit the information on the object detection range flag that is turned ON to the server device 14 together with the additional information (step S52). When the monitoring device 16, 52, 53 detects from the calculation result of the monitoring device that the image display device 11 does not exist in the range where the monitoring target object 1 is visible (step S123: No), the target device The detection range flag is switched to OFF (step S125). And the monitoring apparatuses 16, 52, and 53 transmit information to the server apparatus 14 together with the additional information about the object detection range flag that is set to OFF (step S52).

  As is apparent from the above description, the information processing system according to the third embodiment is configured such that when the user approaches the obstacle 2, the image displayed on the image display device 11 is changed from the superimposed image to the actual scenery image. Switch to. As a result, it is possible to prevent the user from approaching the obstacle 2 more than necessary and coming into contact with the obstacle 2, thereby improving the user's safety. In addition, it is possible to smoothly switch between the virtual world and the real world. In addition, since the update frequency of the superimposed image can be adjusted, it is possible to improve the performance of communication data efficiency.

  The above-described embodiments are provided as examples, and are not intended to limit the scope of the present invention. Each of the novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. Each embodiment and modifications of each embodiment are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 1 Monitoring object 2 Obstacle 11 Image display apparatus 12 Information provision apparatus 13 Network 14 Server apparatus 15 Superimposed image database 16 Monitoring apparatus (1st monitoring apparatus)
DESCRIPTION OF SYMBOLS 17 Superimposed image extraction part 18 Display image transmission part 19 Superimposed image adjustment part 20 Image composition part 21 Additional information receiving part 22 Simple 3D model data storage part 24 Additional information storage part 25 Monitoring position holding part 26 Additional information transmission part 27 Additional information Generation unit 28 Object monitoring device 29 Relative position calculation unit 30 Positioning data reception unit 31 Position / direction acquisition unit 32 Positioning data transmission unit 33 Display image reception unit 34 Image output unit 41 Imaging data transmission unit 42 Real scene imaging unit 43 Information Extraction target selection unit 44 Enlargement / reduction adjustment unit 45 Data reception unit 46 White area calculation unit 47 White image generation unit 48 Superimposed image selection unit 49 Monitoring target determination unit 50 Monitoring target selection unit 52 Second monitoring device 53 Third Monitoring device 61 Obstacle relative distance detection unit 62 Output image switching unit 64 Update frequency adjustment unit 65 Object detection range Storage unit 66 object detection unit

JP 2007-228315 A JP 2004-145657 A JP 2009-289035 A

Claims (7)

Target image generating means for generating a monitoring target image of the monitoring target corresponding to the position and direction of the image display device;
Obstacle image generation means for generating an obstacle image corresponding to the position and direction of the image display device, the obstacle being an obstacle to visual recognition of the monitored object;
The superimposed image is superimposed on the obstacle image so as to be visible on the obstacle image so as to correspond to the position and direction of the image display device, and is superimposed with additional information corresponding to the monitored object. And when the relative distance between the image display device and the monitoring object is a predetermined distance or less from the position of the image display device, the generation of the superimposed image is stopped and the additional information is Superimposing image generation means for generating only ,
An information processing apparatus comprising: a transmission unit configured to transmit only the superimposed image or the additional information to the image display apparatus. The target image generation means generates the monitoring target image of a monitoring target specified through the image display device among a plurality of monitoring targets,
The information processing apparatus according to claim 1, wherein the superimposed image generating unit generates the superimposed image in which additional information corresponding to a monitoring target specified via the image display device is superimposed. Enlargement / reduction adjustment means for adjusting the size of displaying the monitoring target image;
The information processing apparatus according to claim 1, further comprising: a scale adjustment unit that adjusts a scale of the obstacle image. The superimposed image generation unit generates a display area having a size suitable for displaying the monitoring target image on the obstacle in the real landscape image captured on the image display device side, and the monitoring area includes the monitoring area. The information processing apparatus according to any one of claims 1 to 3, wherein the target image is superimposed. The superimposed image generating means, among of claims 1 to 4, characterized in that to generate the superimposed image at a frequency corresponding to the relative distance of the image display device and the object to be monitored, in any one The information processing apparatus described. When information indicating the position and direction is transmitted, the received superimposed image is displayed , and the relative distance to the monitoring object is less than a certain distance by the real landscape imaging means for capturing the real landscape image In addition, an image display device that displays the superimposed image that has been displayed so far by switching to the real landscape image ;
Target image generation means for generating a monitoring target image of the monitoring target corresponding to information indicating the position and direction received from the image display device using the monitoring target image of the monitoring target, and the position of the image display device Obstacle image generation means for generating an obstacle image of an obstacle that is an obstacle to the visual recognition of the monitoring object, and the obstacle image so as to correspond to the position and direction of the image display device A superimposed image generating unit that superimposes the monitoring target image so as to be visible and superimposes additional information corresponding to the monitoring target to generate the superimposed image, and displays the superimposed image on the image display device. An information processing system comprising: an information processing device including a transmission unit that transmits the information to the computer. Computer
Target image generating means for generating a monitoring target image of the monitoring target corresponding to the position and direction of the image display device;
Obstacle image generation means for generating an obstacle image corresponding to the position and direction of the image display device, the obstacle being an obstacle to visual recognition of the monitored object;
The superimposed image is superimposed on the obstacle image so as to be visible on the obstacle image so as to correspond to the position and direction of the image display device, and is superimposed with additional information corresponding to the monitored object. And when the relative distance between the image display device and the monitoring object is a predetermined distance or less from the position of the image display device, the generation of the superimposed image is stopped and the additional information is Superimposing image generation means for generating only ,
An information processing program that functions as a transmission unit that transmits only the superimposed image or the additional information to the image display device.

Images