WO2019123971A1 - Ar information providing system and information processing device - Google Patents

Ar information providing system and information processing device Download PDF

Info

Publication number
WO2019123971A1
WO2019123971A1 PCT/JP2018/043209 JP2018043209W WO2019123971A1 WO 2019123971 A1 WO2019123971 A1 WO 2019123971A1 JP 2018043209 W JP2018043209 W JP 2018043209W WO 2019123971 A1 WO2019123971 A1 WO 2019123971A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
range
processing apparatus
unit
image data
Prior art date
Application number
PCT/JP2018/043209
Other languages
French (fr)
Japanese (ja)
Inventor
小坂　忠義
貴之 藤原
瑛 長坂
健太郎 大西
克朗 菊地
Original Assignee
株式会社日立システムズ
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社日立システムズ filed Critical 株式会社日立システムズ
Publication of WO2019123971A1 publication Critical patent/WO2019123971A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators

Definitions

  • the present invention relates to an AR information providing system and an information processing apparatus.
  • Patent Document 1 includes "Ensuring the promptness of information transmission in a remote communication system", “includes a first terminal and a second terminal that can mutually communicate with each other, and a first image captured by the first terminal is When an input operation for drawing a mark at a target position shown on both the terminal and the second terminal and displayed on the displayed image is performed on the second terminal, the first terminal and the second terminal A mark processing apparatus included in a remote communication system in which the mark is superimposed and displayed on the target position so as to move following the movement of the target position in a displayed image, and the user of the first terminal is the mark processing apparatus. It is judged that the presence / absence of necessity of notifying the presence of the mark is determined, and when it is determined that the necessity exists, the determination unit is provided to cause the first terminal to execute notification processing of notifying the user of the presence of the mark. It is done.
  • Patent Document 2 "allowing the instructor to instruct the operator to an object located outside the range of the current captured image", "the camera obtained from a plurality of images captured by the camera. The plurality of images are superimposed to generate a three-dimensional panoramic image based on the posture information of the image and the feature point map of the image, and the three-dimensional panoramic image is generated and displayed on the first display unit.
  • the designation of the present invention there is provided a work support method in which the computer performs processing for causing the second display unit to output the designated position information of the target based on the current posture information of the camera. It is done.
  • the image of an image captured on the camera of one of the terminals is instructed from the other terminal as an image without using a marker or a registered object image
  • the first terminal is photographed by the camera.
  • the first step of sequentially transmitting a moving image to the second terminal, and the second terminal displaying the received captured moving image on the display and causing the user to write a designated still image for the captured moving image
  • the second terminal transmits to the first terminal the instructed still image written to the user and the photographed still image obtained by trimming the photographed moving image as the still image within the predetermined range including the instructed still image
  • the first terminal matches the captured moving image captured by the camera with the captured still image, and superimposes the instructed still image on the captured moving image of the matched portion. It is described as a fourth step of displaying the play ".
  • JP 2017-37434 A JP 2017-58752 A JP, 2015-115723, A
  • Patent Documents 1 to 3 information for assisting work instruction and work from a terminal operated by a remote support person such as a skilled person or a manager located at a remote place to a terminal owned by a site worker is an AR object It is stated to be provided as.
  • an AR object created based on a video captured by the camera of the terminal of the site worker and transmitted to the terminal of the remote support is sent to the terminal of the site worker.
  • the position and direction of the field worker's terminal may change between the time it is displayed and displayed, and the AR object may not be displayed at the correct position.
  • Patent Document 3 since it is necessary to send image information from the administrator to the worker, the communication load is high, and the delay caused by the compression and expansion of the image data becomes a problem. Another problem is that the accuracy of the image matching is lowered due to repeated compression processing in each of the forward pass and the return pass.
  • An object of the present invention is to provide an AR information providing system and an information processing apparatus capable of providing AR information with high accuracy even in an environment where communication delay may occur.
  • the first information processing apparatus may include an AR information display unit which displays AR information superimposed on a predetermined position of a real predetermined range of scenery, and range specification information which is information for specifying the predetermined range.
  • the predetermined position is specified by an information receiving unit, an AR information setting unit that sets the AR information in association with the predetermined position in the predetermined range specified based on the range specifying information, and the set AR information.
  • An AR information transmission unit that transmits the first information processing apparatus together with predetermined position specification information, the first information processing apparatus includes an AR information reception unit that receives the predetermined position specification information together with the AR information
  • the range identification information acquisition unit acquires the current range identification information, and the AR information display unit receives the AR received based on the received predetermined position identification information and the acquired current range identification information.
  • the display position of information is specified, and the display of the AR information is controlled based on the specified display position.
  • AR information can be provided accurately even in an environment where communication delays may occur.
  • FIG. 1 shows schematic structure of AR information provision system of 1st Embodiment. It is a figure which shows the main structures of AR information display apparatus of 1st Embodiment. It is an external view which shows an example of AR information display apparatus of 1st Embodiment. It is a figure which shows the main structures of AR information provision apparatus of 1st Embodiment. It is a flowchart explaining the process performed in the AR information provision system of 1st Embodiment. It is an example of the screen which AR information display part displays. (A), (b) is an example of the screen which AR information display part displays. It is a figure which shows the main structures of AR information display apparatus of 2nd Embodiment.
  • AR Augmented Reality
  • database may be abbreviated as "DB”
  • program may be abbreviated as "PG”.
  • FIG. 1 shows a schematic configuration of the AR information providing system 1 described as the first embodiment.
  • the AR information providing system 1 provides an AR information display device 10 (first information processing device) worn by or carried by an AR information user 3 who uses (refers to) AR information 6 (AR object), and AR information 6 And an AR information providing apparatus 20 (second information processing apparatus) operated by the AR information provider 4.
  • first information processing device worn by or carried by an AR information user 3 who uses (refers to) AR information 6 (AR object), and AR information 6
  • an AR information providing apparatus 20 second information processing apparatus operated by the AR information provider 4.
  • the AR information display device 10 and the AR information providing device 20 perform bi-directional communication via the communication network 5.
  • the communication network 5 may be a wired or wireless communication means (Internet (Internet), LAN (Local Area Network), WAN (Wide Area Network), public communication network, dedicated line, etc.) or various communication devices (router, hub, It is configured using a NIC (Network Interface Card), a wireless communication module, and the like.
  • the AR information user 3 is, for example, a worker at a site who performs maintenance operation work of equipment.
  • the AR information user 3 performs work such as inspection and repair of equipment while referring to the AR information 6 sent from the AR information providing device 20 and displayed by the AR information display device 10.
  • the AR information provider 4 is, for example, a manager or a skilled person of the work performed by the AR information user 3.
  • the AR information provider 4 operates the AR information providing apparatus 20 and sets (generates) AR information 6 including information such as a work instruction, a work procedure, and various advices.
  • the AR information 6 set by the AR information provider 4 is transmitted from the AR information providing device 20 to the AR information display device 10 and provided to the AR information user 3.
  • the AR information display device 10 includes a processor 11, a storage device 12, a display device 13, a communication device 14, a camera 15, and various sensors 16.
  • the AR information display device 10 may be provided with an audio input / output device (microphone, speaker, etc.).
  • basic hardware such as a power storage device (battery) and a power supply circuit are omitted.
  • the processor 11 is configured using, for example, a central processing unit (CPU) or a micro processing unit (MPU). Various functions of the AR information display device 10 are realized by the processor 11 executing a program stored in the storage device 12.
  • CPU central processing unit
  • MPU micro processing unit
  • the storage device 12 includes, for example, a read only memory (ROM), a random access memory (RAM), a non-volatile memory (NVRAM), a solid state drive (SSD), a hard disk drive (HDD), and optical / magnetic storage. It is configured using an apparatus or the like.
  • ROM read only memory
  • RAM random access memory
  • NVRAM non-volatile memory
  • SSD solid state drive
  • HDD hard disk drive
  • the display device 13 superimposes and displays AR information on a real landscape (in this example, a scene of a site currently viewed by the AR information user 3).
  • the display device 13 uses, for example, a virtual image projection method of forming a virtual image by using a half mirror, and a retinal projection method of forming an image directly on a retina by using a human eye lens.
  • the display device 13 is, for example, a micro display (LCD (Liquid Crystal Display), organic EL (EL: Electro Luminescence), an LED (Light Emitting Diode) or the like), a projection lens, a light guide plate, an LED And other light emitting devices.
  • the method for the display device 13 to display AR information superimposed on the ambient light of the site is not necessarily limited.
  • the method of overlaying AR information on a real landscape does not necessarily have to be an optical transmission method that uses direct external light, and, for example, indirect reality captured by a camera equipped in the AR information display device 10 It may be a video transmission method in which AR information is superimposed on a landscape image.
  • the communication device 14 is a device that communicates with the AR information providing device 20 via the communication network 5 by a wireless method or a wired method, and is configured using, for example, a wireless communication module or a NIC (Network Interface Card).
  • a wireless communication module or a NIC (Network Interface Card).
  • the camera 15 is a device for capturing a landscape around the AR information display device 10, and may be, for example, an optical device, an imaging device of a CCD (Charged-coupled devices) system or a CMOS (Complementary metal-oxide-semiconductor) system, Image) processing unit, storage unit, etc.
  • CCD Charge-coupled devices
  • CMOS Complementary metal-oxide-semiconductor
  • the various sensors 16 detect and output information on the state (current position, posture, movement) of the AR information display device 10 and the surrounding environment.
  • the various sensors 16 are, for example, a depth camera (Time of Flight (TOF) camera, stereo camera, laser imaging (LiDAR: Laser imaging Detection and Ranging, infrared depth sensor, ultrasonic sensor, etc.), GPS (Global (Positioning System), gyro sensor, acceleration sensor, azimuth sensor, infrared sensor, illuminance sensor, etc.
  • the camera 15 can also function as the various sensors 16.
  • FIG. 3 is an appearance of a glasses-type wearable device (HMD (Head Mounted Display)) shown as an example of the AR information display device 10.
  • HMD Head Mounted Display
  • the components shown by reference numerals 11 to 16 of the AR information display device 10 shown in FIG. 2 are mounted.
  • AR information is displayed on the display screen 131 of the display device 13 attached to the glasses portion 132 which transmits external light.
  • the figure is merely an example of the AR information display device 10, and the AR information display device 10 may be another type of device such as a smartphone, a tablet, or various portable terminals.
  • the storage device 12 includes an AR information display program (hereinafter referred to as AR information display PG 111), a range identification information acquisition program (hereinafter referred to as range identification information acquisition PG 112), and a range identification information transmission program (
  • AR information display PG 111 a range identification information acquisition program
  • range identification information transmission program a range specification information transmission PG 113
  • AR information reception PG 114 an AR information reception program
  • image data transmission PG 115 an image data transmission program
  • the storage device 12 may further store a program for realizing an operating system (OS) or a database management system (DBMS).
  • OS operating system
  • DBMS database management system
  • the function implemented by the AR information display PG 111 is implemented by the AR information display unit
  • the function implemented by the range identification information acquisition PG 112 is implemented by the range identification information acquisition unit
  • the function implemented by the range identification information transmission PG 113 Is referred to as a range specifying information transmission unit
  • a function realized by the AR information reception PG 114 is referred to as an AR information reception unit
  • a function realized by the image data transmission PG 115 is referred to as an image data transmission unit.
  • the storage device 12 stores AR information 121, map information 122, and marker information 123.
  • the AR information 121, the map information 122, and the marker information 123 are managed, for example, in a database of a DBMS.
  • the AR information 121 is AR information sent from the AR information provision device 20, and includes character information which is a component of the AR information, graphic information (planar figure (geometrical figure), solid, etc.) and the like.
  • the map information 122 includes three-dimensional map information (for example, by a three-dimensional model) around the AR information display device 10 represented by the world coordinate system (world coordinate system).
  • the marker information 123 is for specifying a marker from the installation position information in the world coordinate system and the photographed image for each of one or more markers fixedly installed around the AR information display device 10 (in this example, a work site in this example) And matching information.
  • the AR information display unit described above displays AR information on the display screen 131 of the display device 13 so as to overlap with a predetermined position of a landscape in a predetermined actual range.
  • the AR information display unit also displays information on AR information such as the direction in which the AR information is present on the display screen 131. Specific display control of the AR information will be described later.
  • the range identification information acquisition unit acquires range identification information that is information for identifying the above-described predetermined range.
  • the above-described predetermined range corresponds to, for example, a shooting range (a range specified by an angle of view, a viewing angle or the like) of the camera 15 at a certain point in time.
  • the range specifying information matches, for example, a visual field range (field angle, visual field angle) where the AR information user 3 views the surroundings via the glasses portion 132 of the AR information display device 10 at a certain time.
  • the range specification information acquisition unit generates range specification information based on the information output from the various sensors 16, for example.
  • the range specifying information includes, for example, information indicating the current position of the AR information display device 10 in the world coordinate system, and the attitude of the AR information display device 10 (for example, the direction in which the camera 15 of the AR information display device 10 faces) And information to be shown.
  • the range identification information transmission unit transmits the range identification information transmitted by the range identification information acquisition unit to the AR information provision device 20.
  • the range specifying information transmission unit adds a time stamp (for example, a time when the range specifying information is acquired or a time when the range specifying information is generated) to the range specifying information to be transmitted. This time stamp is used, for example, to correspond to image data to be described later.
  • the AR information receiving unit receives AR information from the AR information providing device 20 and stores the AR information as AR information 121.
  • the predetermined position specifying information which is information for specifying the predetermined position described above, is attached to the AR information received by the AR information receiving unit.
  • the predetermined position specifying information includes, for example, information representing the predetermined position in the world coordinates.
  • the image data transmission unit transmits the image data captured and output by the camera 15 to the AR information provision device 20.
  • the image data may be a still image or each frame constituting moving image data.
  • the image data is accompanied by a time stamp (for example, the time when the image data or the frame is acquired or the time when these are generated).
  • the AR information provision device 20 includes a processor 21, a storage device 22, an input device 23, a display device 24, and a communication device 25.
  • the AR information provision device 20 may include an audio input / output device (microphone, speaker, etc.).
  • basic hardware such as a power supply circuit is omitted.
  • the processor 21 is configured using, for example, a CPU or an MPU.
  • the processor 21 executes the programs stored in the storage device 22 to realize various functions of the AR information provision device 20.
  • the storage device 22 is configured using, for example, a ROM, a RAM, an NVRAM, an SSD, an HDD, an optical / magnetic storage device, and the like.
  • the input device 23 is, for example, a mouse or a keyboard, and receives an input of information from the user.
  • the display device 24 is, for example, a liquid crystal display (LCD) or the like, and provides various information to the user.
  • the communication device 25 is a device that communicates with the AR information display device 10 via the communication network 5 by a wireless method or a wired method, and is configured using, for example, a wireless communication module or a NIC.
  • the storage device 22 is a range specification information reception program (hereinafter referred to as range specification information reception PG 211), an image data reception program (hereinafter referred to as image data reception PG 212), and an AR information setting program. (Hereinafter referred to as AR information setting PG 213), stores an AR information transmission program (hereinafter referred to as AR information transmission PG 214).
  • the storage device 12 may further store a program for realizing the OS and the DBMS.
  • range identification information receiving unit the functions implemented by the range identification information reception PG 211 will be referred to as “range identification information receiving unit”
  • the functions implemented by the image data reception PG 212 may be referred to as image data reception unit
  • the functions implemented by the AR information setting PG 213 The AR information setting unit and the function realized by the AR information transmission PG 214 are referred to as an AR information transmission unit.
  • the storage device 22 stores AR information 221 and map information 222.
  • the AR information 221 and the map information 222 are managed, for example, in a database of a DBMS.
  • the AR information 121 is AR information set by the AR information setting unit.
  • the map information 222 includes three-dimensional map information (for example, by a three-dimensional model) around the AR information display device 10, which is expressed in a world coordinate system (world coordinate system).
  • the map information 222 is common to the contents of the map information 122 stored in the storage device 12 of the AR information display device 10.
  • the range specifying information receiving unit receives and stores the range specifying information sent from the AR information display device 10.
  • the image data receiving unit receives and stores the image data sent from the AR information display device 10.
  • the AR information setting unit is an operation for editing AR information (new creation of AR information, correction of AR information, change of display position, etc.) through interactive processing with the user performed through the user interface (input device 23, display device 24) ) And set AR information.
  • the AR information setting unit superimposes the AR information being set on the camera image and displays it on the display device 24.
  • the AR information setting unit stores, as AR information 221, the AR information generated by the interactive process.
  • Each AR information to be stored is accompanied by the above-mentioned predetermined position specifying information specified based on the range specifying information received by the range specifying information receiving unit.
  • the AR information transmission unit transmits the AR information set by the AR information setting unit to the AR information display device 10 together with the predetermined position specifying information.
  • the image data transmission unit of the AR information display device 10 acquires image data from the camera 15 (S511), and transmits the acquired image data to the AR information provision device 20 (S512). A timestamp is attached to each image data to be transmitted.
  • the image data transmission unit 115 transmits image data by, for example, Real-time Transport Protocol (RTP) and User Datagram Protocol (UDP).
  • RTP Real-time Transport Protocol
  • UDP User Datagram Protocol
  • a time stamp of the RTP header may be used as the time stamp.
  • the time stamp may not necessarily be attached to all the image data.
  • the time stamp may be attached intermittently within a range in which the time shift is not a problem in practical use.
  • a time stamp may be transmitted for each frame. The processes of S511 to S512 are repeatedly performed.
  • the image data receiving unit of the AR information provision device 20 receives the image data sent from the AR information display device 10 (S513), and an image based on the received image data (hereinafter referred to as a camera image) is displayed. It displays on 24 (S514).
  • the image data receiving unit notifies the AR information setting unit of a time stamp attached to the received image data in order to take correspondence between the process performed by the image data receiving unit and the process performed by the AR information setting unit. .
  • the time stamp may not necessarily be notified each time image data is received. For example, a range in which the time shift between the process performed by the image data receiving unit and the process performed by the AR information setting unit does not pose a practical problem May be notified intermittently. Further, when the above time lag does not cause any practical problem (for example, when the processing capability of the AR information provision device 20 is high), the above notification may not be performed.
  • the processing of S513 to S514 is repeated.
  • the range identification information acquisition unit of the AR information display device 10 acquires range identification information (S515), and transmits the acquired range identification information to the AR information provision device 20 (S516).
  • the range specifying information to be transmitted is accompanied by a time stamp for taking correspondence with the image data.
  • the range specifying information includes information indicating the current position of the AR information display device 10 in the world coordinate system and information indicating the direction in which the camera 15 is facing.
  • the range identification information acquisition unit acquires range identification information based on, for example, information based on GPS and image data obtained by capturing a marker.
  • the range identification information acquisition unit collates the size of the photographed marker and the vertical and horizontal distortion of the marker with the marker information 123, and thereby corrects the correct current position of the AR information display device 10 and the accuracy at which the camera 15 is facing. Get the right direction.
  • the range specification information acquisition unit acquires, for example, the accurate current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by collating the data acquired from the depth camera with the map information 122. Do.
  • the range specification information acquisition unit acquires, for example, the correct current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by another known method. The processing of S515 to S516 is repeated.
  • the range identification information receiving unit of the AR information provision device 20 receives the range identification information sent from the AR information display device 10 (S517). Subsequently, the AR information setting unit of the AR information provision device 20 sets AR information for a predetermined range specified based on the received range specifying information by interactive processing via the user interface (the input device 23 and the display device 24). Start (S518). Specifically, the AR information setting unit accepts editing of the AR information via the user interface, and the AR information generated thereby is information (information indicated in the world coordinate system) indicating the predetermined position in the predetermined range. Information associated with certain predetermined position identification information is stored as AR information 221. The user can set AR information anew, or can set AR information 221 stored in advance.
  • the AR information setting unit When setting the AR information, the AR information setting unit superimposes the AR information being set on the camera image and displays it on the display device 24.
  • the AR information setting unit includes the position information 7 including the position and sight line direction of the AR information user 3 and the position and direction of the AR information, and the map information 8 in the vicinity of the AR information user 3 Display (S519).
  • the user can set the AR information at the correct position (the position of the world coordinate system) while checking the camera image of the site.
  • the AR information setting unit includes latitude and longitude. Coordinates of the world coordinate system may be displayed as position information 7.
  • the AR information transmitting unit of the AR information providing device 20 combines the generated AR information with the predetermined position specifying information (world coordinate system) of the AR information to display the AR information display device 10. (S521). The processes of S517 to S521 are repeated.
  • the AR information receiving unit of the AR information display device 10 receives the AR information sent from the AR information providing device 20 and the predetermined position specifying information (world coordinate system) (S522). Subsequently, the range specification information acquisition unit of the AR information display device 10 newly acquires range specification information (world coordinate system) at present (S523). Subsequently, the AR information display unit of the AR information display device 10 collates the newly acquired range identification information (world coordinate system) with the received predetermined position identification information (world coordinate system) to display the AR information display device 13 The display position in the coordinate system of the display screen 131 is determined (S524). At this time, the AR information display unit of the AR information display device 10 displays the current position of the AR information display device 10 included in the predetermined position specification information and the position information 7 including the direction in which the camera 15 is facing (S525) .
  • the AR information display unit of the AR information display device 10 determines whether the display position of the AR information is within the range specified from the newly acquired range specifying information (S526). If the display position of the AR information is within the range specified from the newly acquired range specifying information (S526: YES), the AR information display unit displays AR information at the display position obtained in S524 on the display screen 131. (S527). Thereafter, the process returns to S522.
  • the AR information display unit changes the information indicating the direction in which the display position obtained in S524 exists, and the AR information The alternative object such as the information indicating that it has been done is displayed on the display screen 131 (S528). Thereafter, the process returns to S522.
  • FIG. 7A shows an example of a screen displayed by the AR information display unit in S527
  • FIG. 7B shows an example of a screen displayed by the AR information display unit in S528.
  • the AR information symbol 711
  • the position information 7 are displayed at the display position obtained in S524 of the display screen 131. Is displayed.
  • information code 712 indicating the direction in which the display position obtained in S524 exists and the position information 7 are displayed. It is displayed on the screen 131.
  • the AR information display device 10 includes the AR information sent from the AR information providing device 20 and the predetermined position specifying information (world coordinate system). Since the display of AR information is controlled based on this, AR information can be provided with high accuracy without causing positional deviation. Further, only the image data and the range specifying information are transmitted from the AR information display device 10 to the AR information providing device 20, and the AR information and the predetermined position specifying information are transmitted from the AR information providing device 20 to the AR information display device. Only. Therefore, AR information can be efficiently provided from the AR information provision device 20 to the AR information display device 10 while suppressing an increase in the amount of communication.
  • image data is transmitted from the AR information display device 10 to the AR information provision device 20, and the AR information provision device 20 superimposes AR information on the camera image when setting AR information. It is displayed on the display device 24.
  • the AR information provision device 20 does not transmit image data from the AR information display device 10 to the AR information provision device 20, and the AR information provision device 20 overlaps AR information on the stored map information 222. While displaying on the display unit 24, AR information is set by interactive processing with the user.
  • the AR information display device 10 of the second embodiment includes a processor 11, a storage device 12, a display device 13, a communication device 14, a camera 15, and various sensors 16. These configurations are basically the same as the AR information display device 10 of the first embodiment. However, the AR information display device 10 of the second embodiment differs in configuration from the AR information display device 10 of the first embodiment in that the storage device 12 does not store the image data transmission PG 115 (image data transmission unit). .
  • FIG. 9 shows the main configuration of the AR information provision device 20 of the second embodiment.
  • the AR information provision device 20 of the second embodiment includes a processor 21, a storage device 22, an input device 23, a display device 24, and a communication device 25.
  • These configurations are basically the same as the AR information provision device 20 of the first embodiment.
  • the storage device 22 does not store the image data reception PG 212, and when the AR information setting unit receives editing of AR information by interactive processing with the user.
  • the configuration differs from that of the AR information providing device 20 of the first embodiment in that AR information being set is superimposed on a three-dimensional map image based on the map information 222 and displayed on the display device 24.
  • FIG. 10 is a flowchart for explaining the processing performed in the AR information providing system 1 of the second embodiment. Hereinafter, the process will be described with reference to FIG.
  • the range identification information acquisition unit of the AR information display device 10 acquires range identification information (S1015), and transmits the acquired range identification information to the AR information provision device 20 (S1016). These processes are the same as the processes of S511 and S512 in FIG.
  • the range specifying information receiving unit of the AR information providing device 20 receives the range specifying information sent from the AR information display device 10 (S1017).
  • the AR information setting unit of the AR information provision device 20 sets AR information for a predetermined range specified based on the received range specifying information by interactive processing via the user interface (the input device 23 and the display device 24).
  • the AR information setting unit accepts editing of the AR information via the user interface, and AR information generated thereby is specified position specification that is information indicating a predetermined position (world coordinate system) in the predetermined range.
  • Information associated with the information is stored as AR information 221.
  • the user can set the AR information anew or can use AR information 221 stored in advance.
  • the AR information setting unit superimposes the AR information being set on a three-dimensional map image based on the map information 222 and displays it on the display device 24. As a result, the user can set AR information at an accurate position while checking a three-dimensional map image.
  • S1018 to S1027 are the same as the processes of S518 to S528 in FIG. 5 except for the processes corresponding to S519 and S525, and thus the description thereof is omitted.
  • the AR information display device 10 includes the AR information and the predetermined position sent from the AR information providing device 20. Since the display of AR information is controlled based on the specific information (world coordinate system), AR information can be provided with high accuracy without causing positional deviation.
  • the AR information providing device 20 does not transmit image data from the AR information display device 10 to the AR information providing device 20, and the AR information providing device 20 superimposes the AR information being set on the three-dimensional map image based on the map information 222. While displaying it, AR information is set by interactive processing with the user. Therefore, AR information can be efficiently provided from the AR information provision device 20 to the AR information display device 10 while suppressing an increase in the amount of communication.
  • the range specifying information is transmitted from the AR information display device 10 to the AR information providing device 20, and the AR information providing device 20 receives the range specifying information, and the received range specifying information AR information is set for a predetermined range specified based on the above.
  • the AR information display device 10 stores the range specifying information as history information associated with the time stamp, and the AR information providing device 20 transmits the range specifying information to the AR information providing device 20. Do not send, the AR information provision device 20 sets AR information on the range (predetermined range) of the camera image by interactive processing with the user while displaying the AR information on the camera image while superimposing the AR information upon setting the AR information.
  • the set AR information is transmitted to the AR information display device 10 together with predetermined position specifying information correlated with the AR information, and a time stamp of a camera image (image data) superimposed upon setting of the AR information. Then, the AR information display device 10 acquires the history of the range specifying information corresponding to the time stamp received together with the AR information from the stored history information, and specifies the display position of the AR information in the world coordinate system.
  • FIG. 11 shows the main configuration of the AR information display device 10 of the third embodiment.
  • the AR information display device 10 of the third embodiment includes a processor 11, a storage device 12, a display device 13, a communication device 14, a camera 15, and various sensors 16. These configurations are basically the same as the AR information display device 10 of the first embodiment.
  • the storage device 12 does not store the range specification information transmission PG 113, and the storage device 12 stores the range specification information history storage PG 116, range specification information history specification.
  • the configuration is different from that of the AR information display device 10 of the first embodiment in that the PG 117 and the range specification information history 124 are stored.
  • the function realized by the processor 11 executing the range identification information history storage PG 116 is described by the range identification information history storage unit and the processor 11 executing the range identification information history identification PG 117.
  • the function to be realized is referred to as a range specifying information history specifying unit.
  • FIG. 12 shows the main configuration of the AR information provision device 20 of the third embodiment.
  • the AR information provision device 20 of the third embodiment includes a processor 21, a storage device 22, an input device 23, a display device 24, and a communication device 25. These configurations are basically the same as the AR information provision device 20 of the first embodiment.
  • the storage device 22 does not store the range specification information reception PG 211, and the AR information setting unit performs a coordinate system of a camera image based on image data together with AR information (
  • the point for generating predetermined position specifying information according to the camera image coordinate system) the AR information transmitting unit overlaps with the AR information when setting the AR information with the predetermined position specifying information (camera image coordinate system)
  • the configuration is different from that of the AR information providing device 20 of the first embodiment in that the time stamp of the displayed camera image is transmitted to the AR information display device 10.
  • FIG. 13 is a flowchart illustrating processing performed in the AR information provision system 1 of the third embodiment. Hereinafter, the process will be described with reference to FIG.
  • the range specification information acquisition unit of the AR information display device 10 acquires range specification information (S1315), and the range specification information history storage unit of the AR information display device 10 is acquired by the range specification information acquisition unit
  • the range specifying information is sequentially accumulated and stored as the range specifying information history 124 in the storage device 12 (S1316).
  • each range specifying information accumulated and stored as the range specifying information history 124 is accompanied by a time stamp for taking correspondence with the time stamp of the image data.
  • the range specification information includes information indicating the current position of the AR information display device 10 in the world coordinate system and information indicating the direction in which the camera 15 is facing.
  • the range identification information acquisition unit acquires range identification information based on, for example, information based on GPS and image data obtained by capturing a marker.
  • the range identification information acquisition unit collates the size of the photographed marker and the vertical and horizontal distortion of the marker with the marker information 123, and thereby corrects the correct current position of the AR information display device 10 and the accuracy at which the camera 15 is facing. Get the right direction.
  • the range specification information acquisition unit acquires, for example, the accurate current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by collating the data acquired from the depth camera with the map information 122. Do.
  • the range specification information acquisition unit acquires, for example, the correct current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by another known method.
  • the AR information setting unit of the AR information providing device 20 is a camera image (hereinafter referred to as a target camera image) displayed on the display device 24 by interactive processing via the user interface (the input device 23 and the display device 24).
  • the setting of AR information is started for the range (predetermined range) of (S1318).
  • the AR information setting unit receives editing of the AR information via the user interface, and generates AR information indicating information indicating a predetermined position of the predetermined range (a coordinate system of the target camera image (hereinafter referred to as Information associated with predetermined position specifying information (camera image coordinate system), which is information shown in the camera image coordinate system) is stored as AR information 221.
  • the user can set the AR information anew or can use AR information 221 stored in advance.
  • the AR information setting unit superimposes the AR information being set on the camera image and displays it on the display device 24. As a result, the user can set AR information at an accurate position while checking the camera image.
  • the AR information transmission unit of the AR information provision device 20 determines the generated AR information at a predetermined position represented by the camera image coordinate system associated with the AR information.
  • the identification information and the time stamp of the image data of the target camera image received in S1313 are transmitted to the AR information display device 10 (S1320).
  • the AR information receiving unit of the AR information display device 10 receives the AR information, the predetermined position specifying information (camera image coordinate system), and the time stamp sent from the AR information providing device 20 (S1321).
  • the range specification information history specification unit of the AR information display device 10 specifies (acquires) range specification information (world coordinate system) corresponding to the received time stamp from the range specification information history 124 (S1322).
  • the AR information display unit of the AR information display device 10 receives the predetermined position specification information (camera image coordinate system) received in S1321 based on the range specification information (world coordinate system) specified by the range specification information history specification unit. Are converted into information represented in the world coordinate system (S1323).
  • the range specification information acquisition unit of the AR information display device 10 newly acquires the current range specification information (world coordinate system) (S1324).
  • the AR information display unit of the AR information display device 10 collates the range identification information (world coordinate system) newly acquired in S1324 with the predetermined position identification information of the AR information converted into the world coordinate system in S1323, The display position of the display screen 131 of the display device 13 of AR information is obtained (S1325).
  • the AR information display unit of the AR information display device 10 determines whether the display position of the AR information is within the range specified from the range specification information (S1326). If the display position of the AR information is within the range specified from the range specification information (S1326: YES), the AR information display unit displays the AR information at the display position obtained in S1325 of the display screen 131 (S1327). Thereafter, the process returns to S1321.
  • the AR information display unit changes the information indicating the direction in which the display position obtained in S1325 exists, and the AR information
  • the alternative AR information (AR object) including the information indicating that it has been done is displayed on the display screen 131 (S1328). Thereafter, the process returns to S1321.
  • the AR information providing apparatus 20 does not transmit the range specifying information from the AR information display apparatus 10 to the AR information providing apparatus 20, and the AR information providing apparatus 20 displays the AR information display While superimposing a camera image based on image data sent from the apparatus 10 on AR information and displaying it on the display device 24, AR information is set by interactive processing with the user. Therefore, the AR information providing apparatus 20 does not have to correspond to the image data and the range specifying information, and the AR information providing apparatus 20 efficiently provides AR information to the AR information display apparatus 10 while suppressing an increase in the communication amount. Can.
  • each of the configurations, function units, processing units, processing means, etc. described above may be realized by hardware, for example, by designing part or all of them with an integrated circuit.
  • each configuration, function, etc. described above may be realized by software by the processor interpreting and executing a program that realizes each function.
  • Information such as a program, a table, and a file that implements each function can be placed in a memory, a hard disk, a recording device such as an SSD, or a recording medium such as an IC card, an SD card, or a DVD.
  • control lines and information lines indicate what is considered to be necessary for explanation, and not all control lines and information lines on mounting are necessarily shown. For example, in practice it may be considered that almost all configurations are mutually connected.
  • the arrangement of the various functional units, the various processing units, and the various databases of each device described above is merely an example.
  • the arrangement form of the various functional units, the various processing units, and the various databases can be changed to an optimum arrangement form in view of the performance, processing efficiency, communication efficiency, etc. of hardware and software provided in the AR information providing system 1.
  • the configuration of the database (schema or the like) can be flexibly changed from the viewpoint of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency and the like.
  • AR Information Display Device 111 AR Information Display PG (AR Information Display Unit 112, Range identification information acquisition PG (range identification information acquisition unit), 113 Range identification information transmission PG (range identification information transmission unit) 114 AR information reception PG (AR information reception unit) 115 Image data transmission PG (image data) Transmission unit), 116 range identification information history storage PG (range identification information history storage unit), 117 range identification information history identification PG (range identification information history identification unit), 121 AR information, 122 map information, 123 marker information, 124 range Specific information history, 20 AR information provision device, 211 range identification information reception PG (range identification information reception ), 212 image data receiving PG (image data receiving unit), 213 AR information setting PG (AR information setting unit), 214 AR information transmission PG (AR information transmitter), 221 AR information, 222 map information

Abstract

The purpose of the present invention is to provide AR information with good accuracy even in an environment in which a communication delay may occur. A first information processing device (AR information display device) overlaps AR information at a prescribed position in a scene of a prescribed range of reality, and transmits range specifying information that specifies the prescribed range to a second information processing device (AR information providing device). The second information processing device sets the AR information in association with a prescribed location in the prescribed range specified on the basis of the range specifying information, and transmits the set AR information to the first information processing device together with prescribed location specifying information that specifies the prescribed location. The first information processing device acquires current range specifying information, specifies a display location of the received AR information on the basis of the received prescribed location information and the acquired current range specifying information, and controls a display of the AR information on the basis of the specified display location.

Description

ＡＲ情報提供システム及び情報処理装置AR information providing system and information processing apparatus
　本発明は、ＡＲ情報提供システム及び情報処理装置に関する。 The present invention relates to an AR information providing system and an information processing apparatus.
　本出願は、２０１７年１２月１８日に出願された日本特許出願２０１７－２４１４３７号に基づく優先権を主張し、その開示全体を援用して本出願に取り込むものである。 This application claims priority based on Japanese Patent Application No. 2017-241437 filed on Dec. 18, 2017, the entire disclosure of which is incorporated herein by reference.
　特許文献１には、「遠隔通信システムにおいて情報伝達の迅速性を確保する。」、「互いに通信可能な第１端末と第２端末とを含み、第１端末にて撮影される映像が第１端末および第２端末の両方にて表示され、第２端末において、表示されている映像に示される目標位置にマークを描画するための入力操作が行われると、第１端末および第２端末にて表示される映像にて前記目標位置の動きに追随して動くように前記マークが前記目標位置に重畳表示される遠隔通信システムに含まれるマーク処理装置であって、前記第１端末のユーザへ前記マークの存在を通知する必要性の有無を判定し、必要性有と判定した場合、前記マークの存在を前記ユーザへ通知する通知処理を前記第１端末に実行させる判定部を備えた」と記載されている。 Patent Document 1 includes "Ensuring the promptness of information transmission in a remote communication system", "includes a first terminal and a second terminal that can mutually communicate with each other, and a first image captured by the first terminal is When an input operation for drawing a mark at a target position shown on both the terminal and the second terminal and displayed on the displayed image is performed on the second terminal, the first terminal and the second terminal A mark processing apparatus included in a remote communication system in which the mark is superimposed and displayed on the target position so as to move following the movement of the target position in a displayed image, and the user of the first terminal is the mark processing apparatus. It is judged that the presence / absence of necessity of notifying the presence of the mark is determined, and when it is determined that the necessity exists, the determination unit is provided to cause the first terminal to execute notification processing of notifying the user of the presence of the mark. It is done.
　特許文献２には、「指示者が、撮影された現在の画像の範囲外に位置する対象物を、作業員に指示可能とする」、「カメラが撮影した複数の画像から得られる、該カメラの姿勢情報及び該画像の特徴点マップに基づいて、該複数の画像を重ね合わせて３次元パノラマ画像を生成して第１の表示部に表示させ、前記３次元パノラマ画像上でなされた対象物の指定に応じて、前記カメラの現在の姿勢情報に基づいて、指定された該対象物の位置情報を第２の表示部に出力させる処理をコンピュータが行う作業支援方法が提供される」と記載されている。 According to Patent Document 2, "allowing the instructor to instruct the operator to an object located outside the range of the current captured image", "the camera obtained from a plurality of images captured by the camera. The plurality of images are superimposed to generate a three-dimensional panoramic image based on the posture information of the image and the feature point map of the image, and the three-dimensional panoramic image is generated and displayed on the first display unit. According to the designation of the present invention, there is provided a work support method in which the computer performs processing for causing the second display unit to output the designated position information of the target based on the current posture information of the camera. It is done.
　特許文献３には、「マーカや登録オブジェクト画像を用いることなく、一方の端末のカメラに写る映像に対して、他方の端末から画像的に指示する」、「第１の端末が、カメラによる撮影動画像を逐次、第２の端末へ送信する第１のステップと、第２の端末が、受信した撮影動画像をディスプレイに表示し、当該撮影動画像に対する指示静止画像をユーザに書き込ませる第２のステップと、第２の端末が、ユーザに書き込まれた指示静止画像と、当該指示静止画像を含む所定範囲で撮影動画像を静止画像としてトリミングした撮影静止画像とを、第１の端末へ送信する第３のステップと、第１の端末が、カメラによって撮影された撮影動画像と撮影静止画像とをマッチングさせ、一致した部分の撮影動画像に指示静止画像を重畳させてディスプレイに表示する第４のステップとを有する」と記載されている。 According to Patent Document 3, "the image of an image captured on the camera of one of the terminals is instructed from the other terminal as an image without using a marker or a registered object image", and "the first terminal is photographed by the camera. The first step of sequentially transmitting a moving image to the second terminal, and the second terminal displaying the received captured moving image on the display and causing the user to write a designated still image for the captured moving image And the second terminal transmits to the first terminal the instructed still image written to the user and the photographed still image obtained by trimming the photographed moving image as the still image within the predetermined range including the instructed still image And the first terminal matches the captured moving image captured by the camera with the captured still image, and superimposes the instructed still image on the captured moving image of the matched portion. It is described as a fourth step of displaying the play ".
特開２０１７－３７４３４号公報JP 2017-37434 A 特開２０１７－５８７５２号公報JP 2017-58752 A 特開２０１５－１１５７２３号公報JP, 2015-115723, A
　昨今、設備の保守運用、製品の製造、流通等の様々な業務分野においてＡＲ（Augmented Reality：拡張現実）技術の応用が進められている。特許文献１～３には、遠隔地に存在する熟練者や管理者等の遠隔支援者が操作する端末から、現場作業者が所持する端末に対して作業指示や作業を補助する情報をＡＲオブジェクトとして提供することが記載されている。 Recently, application of AR (Augmented Reality: Augmented Reality) technology has been promoted in various business fields such as maintenance and operation of equipment, production of products, and distribution. In Patent Documents 1 to 3, information for assisting work instruction and work from a terminal operated by a remote support person such as a skilled person or a manager located at a remote place to a terminal owned by a site worker is an AR object It is stated to be provided as.
　ここでこのように遠隔に存在する遠隔支援者の端末から現場作業者の端末にＡＲオブジェクトを提供する際は、通信の状態や端末の性能に起因する通信の遅延が問題となる。 Here, when providing the AR object from the terminal of the remote support who remotely exists to the terminal of the site worker, a delay in communication due to the state of communication and the performance of the terminal becomes a problem.
　例えば、特許文献１，２の構成では、通信遅延により、現場作業者の端末のカメラで撮影され遠隔支援者の端末に送信された映像に基づき作成されたＡＲオブジェクトが現場作業者の端末に送られて表示されるまでの間に現場作業者の端末の位置や方向が変化してしまうことがあり、ＡＲオブジェクトが正しい位置に表示されないことがある。 For example, in the configurations of Patent Documents 1 and 2, due to a communication delay, an AR object created based on a video captured by the camera of the terminal of the site worker and transmitted to the terminal of the remote support is sent to the terminal of the site worker. The position and direction of the field worker's terminal may change between the time it is displayed and displayed, and the AR object may not be displayed at the correct position.
　また特許文献３では、管理者から作業者に画像情報を送る必要があるため通信負荷が高く、画像データの圧縮や伸張に起因する遅延が問題となる。また往路と復路の夫々で圧縮処理が繰り返し行われることで画像マッチングの精度が低下してしまうことも問題となる。 Further, in Patent Document 3, since it is necessary to send image information from the administrator to the worker, the communication load is high, and the delay caused by the compression and expansion of the image data becomes a problem. Another problem is that the accuracy of the image matching is lowered due to repeated compression processing in each of the forward pass and the return pass.
　本発明は、通信の遅延が生じ得る環境でも精度よくＡＲ情報を提供することが可能なＡＲ情報提供システム及び情報処理装置を提供することを目的としている。 An object of the present invention is to provide an AR information providing system and an information processing apparatus capable of providing AR information with high accuracy even in an environment where communication delay may occur.
　上記課題を解決するための本発明のうちの一つは、第１情報処理装置と、前記第１情報処理装置と通信可能に接続する第２情報処理装置と、を含んで構成されるＡＲ情報提供システムであって、前記第１情報処理装置は、現実の所定範囲の風景の所定位置にＡＲ情報を重ねて表示するＡＲ情報表示部と、前記所定範囲を特定する情報である範囲特定情報を取得する範囲特定情報取得部と、前記範囲特定情報を前記第２情報処理装置に送信する範囲特定情報送信部と、を備え、前記第２情報処理装置は、前記範囲特定情報を受信する範囲特定情報受信部と、前記範囲特定情報に基づき特定される前記所定範囲の前記所定位置に対応づけて前記ＡＲ情報を設定するＡＲ情報設定部と、設定した前記ＡＲ情報を、前記所定位置を特定する情報である所定位置特定情報とともに前記第１情報処理装置に送信するＡＲ情報送信部と、を備え、前記第１情報処理装置は、前記ＡＲ情報とともに前記所定位置特定情報を受信するＡＲ情報受信部を有し、前記範囲特定情報取得部は、現在の前記範囲特定情報を取得し、前記ＡＲ情報表示部は、受信した前記所定位置特定情報と取得した前記現在の範囲特定情報とに基づき、受信した前記ＡＲ情報の表示位置を特定し、特定した前記表示位置に基づき前記ＡＲ情報の表示を制御する。 One of the aspects of the present invention for solving the above problems is AR information configured to include a first information processing apparatus and a second information processing apparatus communicably connected to the first information processing apparatus. In the providing system, the first information processing apparatus may include an AR information display unit which displays AR information superimposed on a predetermined position of a real predetermined range of scenery, and range specification information which is information for specifying the predetermined range. A range specifying information acquiring unit to acquire, and a range specifying information transmitting unit for transmitting the range specifying information to the second information processing apparatus, the second information processing apparatus specifying the range specifying information to be received The predetermined position is specified by an information receiving unit, an AR information setting unit that sets the AR information in association with the predetermined position in the predetermined range specified based on the range specifying information, and the set AR information. Information An AR information transmission unit that transmits the first information processing apparatus together with predetermined position specification information, the first information processing apparatus includes an AR information reception unit that receives the predetermined position specification information together with the AR information The range identification information acquisition unit acquires the current range identification information, and the AR information display unit receives the AR received based on the received predetermined position identification information and the acquired current range identification information. The display position of information is specified, and the display of the AR information is controlled based on the specified display position.
　その他、本願が開示する課題、及びその解決方法は、発明を実施するための形態の欄、及び図面により明らかにされる。 In addition, the subject which this application discloses, and its solution method are clarified by the column of the form for inventing, and drawing.
　本発明により、通信の遅延が生じ得る環境でも精度よくＡＲ情報を提供することができる。 According to the present invention, AR information can be provided accurately even in an environment where communication delays may occur.
第１実施形態のＡＲ情報提供システムの概略的な構成を示す図である。It is a figure which shows schematic structure of AR information provision system of 1st Embodiment. 第１実施形態のＡＲ情報表示装置の主な構成を示す図である。It is a figure which shows the main structures of AR information display apparatus of 1st Embodiment. 第１実施形態のＡＲ情報表示装置の一例を示す外観図である。It is an external view which shows an example of AR information display apparatus of 1st Embodiment. 第１実施形態のＡＲ情報提供装置の主な構成を示す図である。It is a figure which shows the main structures of AR information provision apparatus of 1st Embodiment. 第１実施形態のＡＲ情報提供システムにおいて行われる処理を説明するフローチャートである。It is a flowchart explaining the process performed in the AR information provision system of 1st Embodiment. ＡＲ情報表示部が表示する画面の例である。It is an example of the screen which AR information display part displays. （ａ），（ｂ）は、ＡＲ情報表示部が表示する画面の例である。(A), (b) is an example of the screen which AR information display part displays. 第２実施形態のＡＲ情報表示装置の主な構成を示す図である。It is a figure which shows the main structures of AR information display apparatus of 2nd Embodiment. 第２実施形態のＡＲ情報提供装置の主な構成を示す図である。It is a figure which shows the main structures of AR information provision apparatus of 2nd Embodiment. 第２実施形態のＡＲ情報提供システムにおいて行われる処理を説明するフローチャートである。It is a flowchart explaining the process performed in the AR information provision system of 2nd Embodiment. 第３実施形態のＡＲ情報表示装置の主な構成を示す図である。It is a figure which shows the main structures of AR information display apparatus of 3rd Embodiment. 第３実施形態のＡＲ情報提供装置の主な構成を示す図である。It is a figure which shows the main structures of AR information provision apparatus of 3rd Embodiment. 第３実施形態のＡＲ情報提供システムにおいて行われる処理を説明するフローチャートである。It is a flowchart explaining the process performed in the AR information provision system of 3rd Embodiment.
　以下、実施形態につき適宜図面を参照しつつ説明する。尚、以下の説明において、同一の又は類似する構成に同一の符号を付して重複した説明を省略することがある。また以下の説明において、拡張現実のことをＡＲ（Augmented Reality）と表記することがある。また以下の説明において、「データベース」のことを「ＤＢ」と、「プログラム」のことを「ＰＧ」と、夫々略記することがある。 Hereinafter, embodiments will be described with reference to the drawings as appropriate. In the following description, the same or similar configurations may be denoted by the same reference signs and redundant description may be omitted. In the following description, augmented reality may be referred to as AR (Augmented Reality). In the following description, the term "database" may be abbreviated as "DB", and the term "program" may be abbreviated as "PG".
［第１実施形態］
　図１に第１実施形態として説明するＡＲ情報提供システム１の概略的な構成を示している。ＡＲ情報提供システム１は、ＡＲ情報６（ＡＲオブジェクト）を利用（参照）するＡＲ情報利用者３が装着もしくは携帯するＡＲ情報表示装置１０（第１情報処理装置）と、ＡＲ情報６を提供するＡＲ情報提供者４が操作するＡＲ情報提供装置２０（第２情報処理装置）とを含む。 First Embodiment
FIG. 1 shows a schematic configuration of the AR information providing system 1 described as the first embodiment. The AR information providing system 1 provides an AR information display device 10 (first information processing device) worn by or carried by an AR information user 3 who uses (refers to) AR information 6 (AR object), and AR information 6 And an AR information providing apparatus 20 (second information processing apparatus) operated by the AR information provider 4.
　ＡＲ情報表示装置１０とＡＲ情報提供装置２０とは、通信ネットワーク５を介して双方向通信を行う。通信ネットワーク５は、有線方式や無線方式の通信手段（インターネット（Internet）、ＬＡＮ（Local Area Network）、ＷＡＮ（Wide Area Network）、公衆通信網、専用線等）や各種通信装置（ルータ、ハブ、ＮＩＣ（Network Interface Card）、無線通信モジュール等）を用いて構成される。 The AR information display device 10 and the AR information providing device 20 perform bi-directional communication via the communication network 5. The communication network 5 may be a wired or wireless communication means (Internet (Internet), LAN (Local Area Network), WAN (Wide Area Network), public communication network, dedicated line, etc.) or various communication devices (router, hub, It is configured using a NIC (Network Interface Card), a wireless communication module, and the like.
　ＡＲ情報利用者３は、例えば、設備の保守運用業務を行う現場の作業員である。ＡＲ情報利用者３は、ＡＲ情報表示装置１０が表示する、ＡＲ情報提供装置２０から送られてきたＡＲ情報６を参照しつつ、設備の点検や修理等の作業を遂行する。 The AR information user 3 is, for example, a worker at a site who performs maintenance operation work of equipment. The AR information user 3 performs work such as inspection and repair of equipment while referring to the AR information 6 sent from the AR information providing device 20 and displayed by the AR information display device 10.
　ＡＲ情報提供者４は、例えば、ＡＲ情報利用者３が行う業務の管理者や熟練者等である。ＡＲ情報提供者４は、ＡＲ情報提供装置２０を操作し、作業指示、作業手順、各種アドバイス等の情報を含むＡＲ情報６を設定（生成）する。ＡＲ情報提供者４によって設定されたＡＲ情報６は、ＡＲ情報提供装置２０からＡＲ情報表示装置１０に送信され、ＡＲ情報利用者３に提供される。 The AR information provider 4 is, for example, a manager or a skilled person of the work performed by the AR information user 3. The AR information provider 4 operates the AR information providing apparatus 20 and sets (generates) AR information 6 including information such as a work instruction, a work procedure, and various advices. The AR information 6 set by the AR information provider 4 is transmitted from the AR information providing device 20 to the AR information display device 10 and provided to the AR information user 3.
　図２にＡＲ情報表示装置１０の主な構成を示している。同図に示すように、ＡＲ情報表示装置１０は、プロセッサ１１、記憶装置１２、表示装置１３、通信装置１４、カメラ１５、及び各種センサ１６を備える。同図に示す構成の他、ＡＲ情報表示装置１０は、音声入出力装置（マイクロフォン、スピーカ等）を備えていてもよい。同図では蓄電装置（バッテリ）や電源回路等の基本的なハードウェアについては省略している。 The main configuration of the AR information display device 10 is shown in FIG. As shown in the figure, the AR information display device 10 includes a processor 11, a storage device 12, a display device 13, a communication device 14, a camera 15, and various sensors 16. In addition to the configuration shown in the figure, the AR information display device 10 may be provided with an audio input / output device (microphone, speaker, etc.). In the figure, basic hardware such as a power storage device (battery) and a power supply circuit are omitted.
　プロセッサ１１は、例えば、ＣＰＵ（Central Processing Unit）やＭＰＵ（Micro Processing Unit）を用いて構成される。プロセッサ１１が記憶装置１２に格納されているプログラムを実行することによりＡＲ情報表示装置１０の各種の機能が実現される。 The processor 11 is configured using, for example, a central processing unit (CPU) or a micro processing unit (MPU). Various functions of the AR information display device 10 are realized by the processor 11 executing a program stored in the storage device 12.
　記憶装置１２は、例えば、ＲＯＭ（Read Only Memory）、ＲＡＭ（Random Access Memory）、ＮＶＲＡＭ（Non-volatile memory））、ＳＳＤ（Solid State Drive）、ＨＤＤ（Hard Disk Drive）、光学式／磁気式記憶装置等を用いて構成される。 The storage device 12 includes, for example, a read only memory (ROM), a random access memory (RAM), a non-volatile memory (NVRAM), a solid state drive (SSD), a hard disk drive (HDD), and optical / magnetic storage. It is configured using an apparatus or the like.
　表示装置１３は、現実の風景（本例ではＡＲ情報利用者３が現在眺めている現場の風景）にＡＲ情報を重ねて表示する。表示装置１３は、例えば、ハーフミラーを利用して虚像を形成する虚像投影方式、人の目の水晶体を利用して網膜に直接結像させる網膜投影方式等の方式により、現実の風景にＡＲ情報を重ねて表示する。表示装置１３は、例えば、マイクロディスプレイ（ＬＣＤ（Liquid Crystal Display）や有機ＥＬ（EL:Electro Luminescence）、ＬＥＤ（Light Emitting Diode）等を用いて構成されたもの等）、投射レンズ、導光板、ＬＥＤ等の発光デバイス等を含んで構成される。尚、表示装置１３がＡＲ情報を現場の外界光に重ねて表示する方法は必ずしも限定されない。また現実の風景にＡＲ情報を重ねる方式は必ずしも直接的な外界光を利用する光学透過方式でなくてもよく、例えば、ＡＲ情報表示装置１０に装備されたカメラにより撮影された間接的な現実の風景映像にＡＲ情報を重ねるビデオ透過方式であってもよい。 The display device 13 superimposes and displays AR information on a real landscape (in this example, a scene of a site currently viewed by the AR information user 3). The display device 13 uses, for example, a virtual image projection method of forming a virtual image by using a half mirror, and a retinal projection method of forming an image directly on a retina by using a human eye lens. To display. The display device 13 is, for example, a micro display (LCD (Liquid Crystal Display), organic EL (EL: Electro Luminescence), an LED (Light Emitting Diode) or the like), a projection lens, a light guide plate, an LED And other light emitting devices. The method for the display device 13 to display AR information superimposed on the ambient light of the site is not necessarily limited. Also, the method of overlaying AR information on a real landscape does not necessarily have to be an optical transmission method that uses direct external light, and, for example, indirect reality captured by a camera equipped in the AR information display device 10 It may be a video transmission method in which AR information is superimposed on a landscape image.
　通信装置１４は、通信ネットワーク５を介してＡＲ情報提供装置２０と無線方式又は有線方式により通信する装置であり、例えば、無線通信モジュールやＮＩＣ（Network Interface Card）を用いて構成される。 The communication device 14 is a device that communicates with the AR information providing device 20 via the communication network 5 by a wireless method or a wired method, and is configured using, for example, a wireless communication module or a NIC (Network Interface Card).
　カメラ１５は、ＡＲ情報表示装置１０の周囲の風景を撮影する装置であり、例えば、光学機構、ＣＣＤ（Charged-coupled devices）方式やＣＭＯＳ(Complementary metal-oxide-semiconductor）方式の撮像素子、映像（画像）処理装置、記憶装置等を備えて構成される。 The camera 15 is a device for capturing a landscape around the AR information display device 10, and may be, for example, an optical device, an imaging device of a CCD (Charged-coupled devices) system or a CMOS (Complementary metal-oxide-semiconductor) system, Image) processing unit, storage unit, etc.
　各種センサ１６は、ＡＲ情報表示装置１０の状態（現在位置、姿勢、移動）や周囲環境に関する情報を検知して出力する。各種センサ１６は、例えば、深度カメラ（タイムオブフライト（TOF: Time Of Flight）カメラ、ステレオカメラ、レーザレーダ（LiDAR: Laser imaging Detection and Ranging）、赤外線深度センサ、超音波センサ等）、ＧＰＳ（Global Positioning System）、ジャイロセンサ、加速度センサ、方位センサ、赤外線センサ、照度センサ等である。カメラ１５も各種センサ１６として機能させることができる。 The various sensors 16 detect and output information on the state (current position, posture, movement) of the AR information display device 10 and the surrounding environment. The various sensors 16 are, for example, a depth camera (Time of Flight (TOF) camera, stereo camera, laser imaging (LiDAR: Laser imaging Detection and Ranging, infrared depth sensor, ultrasonic sensor, etc.), GPS (Global (Positioning System), gyro sensor, acceleration sensor, azimuth sensor, infrared sensor, illuminance sensor, etc. The camera 15 can also function as the various sensors 16.
　図３は、ＡＲ情報表示装置１０の一例として示す、眼鏡型のウェアラブルデバイス（ＨＭＤ（Head Mounted Display））の外観である。ＡＲ情報表示装置１０の本体フレーム１９には、図２に示したＡＲ情報表示装置１０の符号１１～１６で示す各構成が実装されている。同図に示すＡＲ情報表示装置１０の場合、外界光を透過する眼鏡部分１３２に装着された表示装置１３の表示画面１３１にＡＲ情報が表示される。尚、同図はＡＲ情報表示装置１０の一例に過ぎず、ＡＲ情報表示装置１０は、例えば、スマートフォン、タブレット、各種携帯端末等の他の種類の装置であってもよい。 FIG. 3 is an appearance of a glasses-type wearable device (HMD (Head Mounted Display)) shown as an example of the AR information display device 10. In the main body frame 19 of the AR information display device 10, the components shown by reference numerals 11 to 16 of the AR information display device 10 shown in FIG. 2 are mounted. In the case of the AR information display device 10 shown in the figure, AR information is displayed on the display screen 131 of the display device 13 attached to the glasses portion 132 which transmits external light. The figure is merely an example of the AR information display device 10, and the AR information display device 10 may be another type of device such as a smartphone, a tablet, or various portable terminals.
　図２に戻り、記憶装置１２は、ＡＲ情報表示プログラム（以下、ＡＲ情報表示ＰＧ１１１と称する。）、範囲特定情報取得プログラム（以下、範囲特定情報取得ＰＧ１１２と称する。）、範囲特定情報送信プログラム（以下、範囲特定情報送信ＰＧ１１３と称する。）、ＡＲ情報受信プログラム（以下、ＡＲ情報受信ＰＧ１１４と称する。）、及び画像データ送信プログラム（以下、画像データ送信ＰＧ１１５と称する。）を記憶する。記憶装置１２は、更にＯＳ（Operating System）やＤＢＭＳ（Database Management System）を実現するためのプログラムを記憶していてもよい。 Returning to FIG. 2, the storage device 12 includes an AR information display program (hereinafter referred to as AR information display PG 111), a range identification information acquisition program (hereinafter referred to as range identification information acquisition PG 112), and a range identification information transmission program ( Hereinafter, a range specification information transmission PG 113), an AR information reception program (hereinafter referred to as AR information reception PG 114), and an image data transmission program (hereinafter referred to as image data transmission PG 115) are stored. The storage device 12 may further store a program for realizing an operating system (OS) or a database management system (DBMS).
　以下、ＡＲ情報表示ＰＧ１１１によって実現される機能のことをＡＲ情報表示部と、範囲特定情報取得ＰＧ１１２によって実現される機能のことを範囲特定情報取得部と、範囲特定情報送信ＰＧ１１３によって実現される機能のことを範囲特定情報送信部と、ＡＲ情報受信ＰＧ１１４によって実現される機能のことをＡＲ情報受信部と、画像データ送信ＰＧ１１５によって実現される機能のことを画像データ送信部と、夫々称する。 Hereinafter, the function implemented by the AR information display PG 111 is implemented by the AR information display unit, the function implemented by the range identification information acquisition PG 112 is implemented by the range identification information acquisition unit, and the function implemented by the range identification information transmission PG 113 Is referred to as a range specifying information transmission unit, a function realized by the AR information reception PG 114 is referred to as an AR information reception unit, and a function realized by the image data transmission PG 115 is referred to as an image data transmission unit.
　同図に示すように、記憶装置１２は、ＡＲ情報１２１、地図情報１２２、及びマーカ情報１２３を記憶する。ＡＲ情報１２１、地図情報１２２、及びマーカ情報１２３は、例えば、ＤＢＭＳのデータベースに管理される。 As shown in the figure, the storage device 12 stores AR information 121, map information 122, and marker information 123. The AR information 121, the map information 122, and the marker information 123 are managed, for example, in a database of a DBMS.
　ＡＲ情報１２１は、ＡＲ情報提供装置２０から送られてきたＡＲ情報であり、ＡＲ情報の構成要素となる文字情報、図形情報（平面図形（幾何学図形）、立体等）等を含む。 The AR information 121 is AR information sent from the AR information provision device 20, and includes character information which is a component of the AR information, graphic information (planar figure (geometrical figure), solid, etc.) and the like.
　地図情報１２２は、世界座標系（ワールド座標系）で表されたＡＲ情報表示装置１０の周囲の３次元の地図情報（例えば、立体モデルによるもの）を含む。 The map information 122 includes three-dimensional map information (for example, by a three-dimensional model) around the AR information display device 10 represented by the world coordinate system (world coordinate system).
　マーカ情報１２３は、ＡＲ情報表示装置１０の周囲（本例では作業現場）に固定設置された一つ以上のマーカの夫々についての、世界座標系による設置位置情報と撮影画像からマーカを特定するためのマッチング情報とを含む。 The marker information 123 is for specifying a marker from the installation position information in the world coordinate system and the photographed image for each of one or more markers fixedly installed around the AR information display device 10 (in this example, a work site in this example) And matching information.
　前述したＡＲ情報表示部は、現実の所定範囲の風景の所定位置に重なるように、表示装置１３の表示画面１３１にＡＲ情報を表示する。またＡＲ情報表示部は、ＡＲ情報が存在する方向等のＡＲ情報に関する情報を表示画面１３１に表示する。ＡＲ情報の具体的な表示制御については後述する。 The AR information display unit described above displays AR information on the display screen 131 of the display device 13 so as to overlap with a predetermined position of a landscape in a predetermined actual range. The AR information display unit also displays information on AR information such as the direction in which the AR information is present on the display screen 131. Specific display control of the AR information will be described later.
　範囲特定情報取得部は、上記の所定範囲を特定する情報である範囲特定情報を取得する。上記の所定範囲は、例えば、ある時点におけるカメラ１５の撮影範囲（画角、視野角等で特定される範囲）と一致する。また範囲特定情報は、例えば、ある時点にＡＲ情報利用者３がＡＲ情報表示装置１０の眼鏡部分１３２を介して周囲を眺めた視野範囲（画角、視野角）と一致する。範囲特定情報取得部は、例えば、各種センサ１６が出力する情報に基づき範囲特定情報を生成する。範囲特定情報は、例えば、ＡＲ情報表示装置１０の現在位置を世界座標系で示した情報と、ＡＲ情報表示装置１０の姿勢（例えば、ＡＲ情報表示装置１０のカメラ１５が向いている方向）を示す情報とを含む。 The range identification information acquisition unit acquires range identification information that is information for identifying the above-described predetermined range. The above-described predetermined range corresponds to, for example, a shooting range (a range specified by an angle of view, a viewing angle or the like) of the camera 15 at a certain point in time. Further, the range specifying information matches, for example, a visual field range (field angle, visual field angle) where the AR information user 3 views the surroundings via the glasses portion 132 of the AR information display device 10 at a certain time. The range specification information acquisition unit generates range specification information based on the information output from the various sensors 16, for example. The range specifying information includes, for example, information indicating the current position of the AR information display device 10 in the world coordinate system, and the attitude of the AR information display device 10 (for example, the direction in which the camera 15 of the AR information display device 10 faces) And information to be shown.
　範囲特定情報送信部は、範囲特定情報取得部が送信した範囲特定情報をＡＲ情報提供装置２０に送信する。範囲特定情報送信部は、送信する範囲特定情報にタイムスタンプ（例えば、当該範囲特定情報を取得した時刻や当該範囲特定情報が生成された時刻）を付帯させる。このタイムスタンプは、例えば、後述する画像データと対応をとるために利用される。 The range identification information transmission unit transmits the range identification information transmitted by the range identification information acquisition unit to the AR information provision device 20. The range specifying information transmission unit adds a time stamp (for example, a time when the range specifying information is acquired or a time when the range specifying information is generated) to the range specifying information to be transmitted. This time stamp is used, for example, to correspond to image data to be described later.
　ＡＲ情報受信部は、ＡＲ情報提供装置２０からＡＲ情報を受信し、ＡＲ情報１２１として記憶する。ＡＲ情報受信部が受信するＡＲ情報には、前述した所定位置を特定する情報である所定位置特定情報が付帯する。所定位置特定情報は、例えば、上記所定位置を世界座標で表した情報を含む。 The AR information receiving unit receives AR information from the AR information providing device 20 and stores the AR information as AR information 121. The predetermined position specifying information, which is information for specifying the predetermined position described above, is attached to the AR information received by the AR information receiving unit. The predetermined position specifying information includes, for example, information representing the predetermined position in the world coordinates.
　画像データ送信部は、カメラ１５によって撮影され出力される画像データをＡＲ情報提供装置２０に送信する。画像データは、静止画でもよいし、動画データを構成している各フレームであってもよい。画像データにはタイムスタンプ（例えば、画像データ又はフレームを取得した時刻やこれらが生成された時刻）が付帯する。 The image data transmission unit transmits the image data captured and output by the camera 15 to the AR information provision device 20. The image data may be a still image or each frame constituting moving image data. The image data is accompanied by a time stamp (for example, the time when the image data or the frame is acquired or the time when these are generated).
　図４にＡＲ情報提供装置２０の主な構成を示している。同図に示すように、ＡＲ情報提供装置２０は、プロセッサ２１、記憶装置２２、入力装置２３、表示装置２４、及び通信装置２５を備える。同図に示す構成の他、ＡＲ情報提供装置２０は、音声入出力装置（マイクロフォン、スピーカ等）を備えていてもよい。同図では電源回路等の基本的なハードウェアについては省略している。 The main configuration of the AR information provision device 20 is shown in FIG. As shown in the figure, the AR information provision device 20 includes a processor 21, a storage device 22, an input device 23, a display device 24, and a communication device 25. In addition to the configuration shown in the figure, the AR information provision device 20 may include an audio input / output device (microphone, speaker, etc.). In the figure, basic hardware such as a power supply circuit is omitted.
　プロセッサ２１は、例えば、ＣＰＵやＭＰＵを用いて構成される。プロセッサ２１が記憶装置２２に格納されているプログラムを実行することによりＡＲ情報提供装置２０の各種の機能が実現される。記憶装置２２は、例えば、ＲＯＭ、ＲＡＭ、ＮＶＲＡＭ、ＳＳＤ、ＨＤＤ、光学式／磁気式記憶装置等を用いて構成される。入力装置２３は、例えば、マウスやキーボード等であり、ユーザから情報の入力を受け付ける。表示装置２４は、例えば、ＬＣＤ（Liquid Crystal Display）等であり、各種の情報をユーザに提供する。通信装置２５は、通信ネットワーク５を介してＡＲ情報表示装置１０と無線方式又は有線方式により通信する装置であり、例えば、無線通信モジュールやＮＩＣを用いて構成される。 The processor 21 is configured using, for example, a CPU or an MPU. The processor 21 executes the programs stored in the storage device 22 to realize various functions of the AR information provision device 20. The storage device 22 is configured using, for example, a ROM, a RAM, an NVRAM, an SSD, an HDD, an optical / magnetic storage device, and the like. The input device 23 is, for example, a mouse or a keyboard, and receives an input of information from the user. The display device 24 is, for example, a liquid crystal display (LCD) or the like, and provides various information to the user. The communication device 25 is a device that communicates with the AR information display device 10 via the communication network 5 by a wireless method or a wired method, and is configured using, for example, a wireless communication module or a NIC.
　同図に示すように、記憶装置２２は、範囲特定情報受信プログラム（以下、範囲特定情報受信ＰＧ２１１と称する。）、画像データ受信プログラム（以下、画像データ受信ＰＧ２１２と称する。）、ＡＲ情報設定プログラム（以下、ＡＲ情報設定ＰＧ２１３と称する。）、ＡＲ情報送信プログラム（以下、ＡＲ情報送信ＰＧ２１４と称する。）を記憶する。記憶装置１２は、更にＯＳやＤＢＭＳを実現するためのプログラムを記憶してもよい。 As shown in the figure, the storage device 22 is a range specification information reception program (hereinafter referred to as range specification information reception PG 211), an image data reception program (hereinafter referred to as image data reception PG 212), and an AR information setting program. (Hereinafter referred to as AR information setting PG 213), stores an AR information transmission program (hereinafter referred to as AR information transmission PG 214). The storage device 12 may further store a program for realizing the OS and the DBMS.
　以下、範囲特定情報受信ＰＧ２１１によって実現される機能のことを範囲特定情報受信部と、画像データ受信ＰＧ２１２によって実現される機能のことを画像データ受信部と、ＡＲ情報設定ＰＧ２１３によって実現される機能のことをＡＲ情報設定部と、ＡＲ情報送信ＰＧ２１４によって実現される機能のことをＡＲ情報送信部と、夫々称する。 Hereinafter, the functions implemented by the range identification information reception PG 211 will be referred to as “range identification information receiving unit”, the functions implemented by the image data reception PG 212 may be referred to as image data reception unit, and the functions implemented by the AR information setting PG 213 The AR information setting unit and the function realized by the AR information transmission PG 214 are referred to as an AR information transmission unit.
　記憶装置２２は、ＡＲ情報２２１及び地図情報２２２を記憶する。ＡＲ情報２２１及び地図情報２２２は、例えば、ＤＢＭＳのデータベースに管理される。ＡＲ情報１２１は、ＡＲ情報設定部により設定されたＡＲ情報である。地図情報２２２は、世界座標系（ワールド座標系）で表された、ＡＲ情報表示装置１０の周囲の３次元の地図情報（例えば、立体モデルによるもの）を含む。地図情報２２２は、ＡＲ情報表示装置１０の記憶装置１２が記憶する地図情報１２２の内容と共通している。 The storage device 22 stores AR information 221 and map information 222. The AR information 221 and the map information 222 are managed, for example, in a database of a DBMS. The AR information 121 is AR information set by the AR information setting unit. The map information 222 includes three-dimensional map information (for example, by a three-dimensional model) around the AR information display device 10, which is expressed in a world coordinate system (world coordinate system). The map information 222 is common to the contents of the map information 122 stored in the storage device 12 of the AR information display device 10.
　範囲特定情報受信部は、ＡＲ情報表示装置１０から送られてくる範囲特定情報を受信して記憶する。 The range specifying information receiving unit receives and stores the range specifying information sent from the AR information display device 10.
　画像データ受信部は、ＡＲ情報表示装置１０から送られてくる画像データを受信して記憶する。 The image data receiving unit receives and stores the image data sent from the AR information display device 10.
　ＡＲ情報設定部は、ユーザインタフェース（入力装置２３、表示装置２４）を介して行われるユーザとの対話処理によりＡＲ情報の編集操作（ＡＲ情報の新規作成、ＡＲ情報の修正、表示位置の変更等）を受け付けてＡＲ情報を設定する。ＡＲ情報設定部は、設定中のＡＲ情報をカメラ画像に重ねて表示装置２４に表示する。ＡＲ情報設定部は、上記対話処理により生成されたＡＲ情報をＡＲ情報２２１として記憶する。記憶する各ＡＲ情報には、範囲特定情報受信部が受信した範囲特定情報に基づき特定された、前述の所定位置特定情報が付帯する。 The AR information setting unit is an operation for editing AR information (new creation of AR information, correction of AR information, change of display position, etc.) through interactive processing with the user performed through the user interface (input device 23, display device 24) ) And set AR information. The AR information setting unit superimposes the AR information being set on the camera image and displays it on the display device 24. The AR information setting unit stores, as AR information 221, the AR information generated by the interactive process. Each AR information to be stored is accompanied by the above-mentioned predetermined position specifying information specified based on the range specifying information received by the range specifying information receiving unit.
　ＡＲ情報送信部は、ＡＲ情報設定部が設定したＡＲ情報を所定位置特定情報とともにＡＲ情報表示装置１０に送信する。 The AR information transmission unit transmits the AR information set by the AR information setting unit to the AR information display device 10 together with the predetermined position specifying information.
　続いて、図５に示すフローチャートとともにＡＲ情報提供システム１において行われる処理について説明する。 Subsequently, processing performed in the AR information providing system 1 will be described with reference to the flowchart shown in FIG.
　ＡＲ情報表示装置１０の画像データ送信部は、カメラ１５から画像データを取得し（Ｓ５１１）、取得した画像データをＡＲ情報提供装置２０に送信する（Ｓ５１２）。送信される各画像データにはタイムスタンプが付帯する。画像データ送信部１１５は、例えば、ＲＴＰ（Real-time Transport Protocol）及びＵＤＰ(User Datagram Protocol)により画像データを送信する。その場合、上記タイムスタンプとしてＲＴＰヘッダのタイムスタンプを用いてもよい。尚、タイムスタンプは必ずしも全ての画像データに付帯させなくてもよく、例えば、時間ずれが実用上問題にならない範囲で間欠的に付帯させるようにしてもよい。また送信する画像データが映像データに基づくものである場合、例えば、フレーム毎にタイムスタンプを送信してもよい。Ｓ５１１～Ｓ５１２の処理は繰り返し行われる。 The image data transmission unit of the AR information display device 10 acquires image data from the camera 15 (S511), and transmits the acquired image data to the AR information provision device 20 (S512). A timestamp is attached to each image data to be transmitted. The image data transmission unit 115 transmits image data by, for example, Real-time Transport Protocol (RTP) and User Datagram Protocol (UDP). In that case, a time stamp of the RTP header may be used as the time stamp. The time stamp may not necessarily be attached to all the image data. For example, the time stamp may be attached intermittently within a range in which the time shift is not a problem in practical use. When the image data to be transmitted is based on video data, for example, a time stamp may be transmitted for each frame. The processes of S511 to S512 are repeatedly performed.
　ＡＲ情報提供装置２０の画像データ受信部は、ＡＲ情報表示装置１０から送られてくる画像データを受信し（Ｓ５１３）、受信した画像データに基づく画像（以下、カメラ画像と称する。）を表示装置２４に表示する（Ｓ５１４）。このとき、画像データ受信部は、画像データ受信部が行う処理とＡＲ情報設定部が行う処理との間の対応をとるため、受信した画像データに付帯するタイムスタンプをＡＲ情報設定部に通知する。尚、タイムスタンプは必ずしも画像データを受信する度に通知しなくてもよく、例えば、画像データ受信部が行う処理とＡＲ情報設定部が行う処理との間の時間ずれが実用上問題にならない範囲で間欠的に通知するようにしてもよい。また上記の時間ずれが実用上問題にならない場合（例えば、ＡＲ情報提供装置２０の処理能力が高い場合）は上記通知は行わなくてもよい。Ｓ５１３～Ｓ５１４の処理は繰り返し行われる。 The image data receiving unit of the AR information provision device 20 receives the image data sent from the AR information display device 10 (S513), and an image based on the received image data (hereinafter referred to as a camera image) is displayed. It displays on 24 (S514). At this time, the image data receiving unit notifies the AR information setting unit of a time stamp attached to the received image data in order to take correspondence between the process performed by the image data receiving unit and the process performed by the AR information setting unit. . The time stamp may not necessarily be notified each time image data is received. For example, a range in which the time shift between the process performed by the image data receiving unit and the process performed by the AR information setting unit does not pose a practical problem May be notified intermittently. Further, when the above time lag does not cause any practical problem (for example, when the processing capability of the AR information provision device 20 is high), the above notification may not be performed. The processing of S513 to S514 is repeated.
　ＡＲ情報表示装置１０の範囲特定情報取得部は、範囲特定情報を取得し（Ｓ５１５）、取得した範囲特定情報をＡＲ情報提供装置２０に送信する（Ｓ５１６）。送信される上記範囲特定情報には画像データと対応をとるためのタイムスタンプが付帯する。本例では、範囲特定情報は、ＡＲ情報表示装置１０の現在位置を世界座標系で示した情報とカメラ１５が向いている方向を示す情報を含む。範囲特定情報取得部は、例えば、ＧＰＳに基づく情報とマーカを撮影した画像データに基づき範囲特定情報を取得する。また範囲特定情報取得部は、例えば、撮影されたマーカのサイズとマーカの縦横の歪みをマーカ情報１２３と照合することにより、ＡＲ情報表示装置１０の正確な現在位置やカメラ１５が向いている正確な方向を取得する。また範囲特定情報取得部は、例えば、深度カメラから取得されるデータと地図情報１２２とを照合することにより、ＡＲ情報表示装置１０の正確な現在位置やカメラ１５が向いている正確な方向を取得する。また範囲特定情報取得部は、例えば、その他の公知の方法により、ＡＲ情報表示装置１０の正確な現在位置やカメラ１５が向いている正確な方向を取得する。Ｓ５１５～Ｓ５１６の処理は繰り返し行われる。 The range identification information acquisition unit of the AR information display device 10 acquires range identification information (S515), and transmits the acquired range identification information to the AR information provision device 20 (S516). The range specifying information to be transmitted is accompanied by a time stamp for taking correspondence with the image data. In this example, the range specifying information includes information indicating the current position of the AR information display device 10 in the world coordinate system and information indicating the direction in which the camera 15 is facing. The range identification information acquisition unit acquires range identification information based on, for example, information based on GPS and image data obtained by capturing a marker. In addition, the range identification information acquisition unit, for example, collates the size of the photographed marker and the vertical and horizontal distortion of the marker with the marker information 123, and thereby corrects the correct current position of the AR information display device 10 and the accuracy at which the camera 15 is facing. Get the right direction. Further, the range specification information acquisition unit acquires, for example, the accurate current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by collating the data acquired from the depth camera with the map information 122. Do. The range specification information acquisition unit acquires, for example, the correct current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by another known method. The processing of S515 to S516 is repeated.
　ＡＲ情報提供装置２０の範囲特定情報受信部は、ＡＲ情報表示装置１０から送られてくる範囲特定情報を受信する（Ｓ５１７）。続いて、ＡＲ情報提供装置２０のＡＲ情報設定部は、ユーザインタフェース（入力装置２３及び表示装置２４）を介した対話処理により、受信した範囲特定情報に基づき特定される所定範囲についてＡＲ情報の設定を開始する（Ｓ５１８）。具体的には、ＡＲ情報設定部は、ユーザインタフェースを介してＡＲ情報の編集を受け付け、それにより生成されたＡＲ情報を上記所定範囲の所定位置を示す情報（世界座標系で示した情報）である所定位置特定情報に対応づけた情報をＡＲ情報２２１として記憶する。ユーザはＡＲ情報を新規に設定することもできるし、予め記憶しているＡＲ情報２２１を利用して設定することもできる。 The range identification information receiving unit of the AR information provision device 20 receives the range identification information sent from the AR information display device 10 (S517). Subsequently, the AR information setting unit of the AR information provision device 20 sets AR information for a predetermined range specified based on the received range specifying information by interactive processing via the user interface (the input device 23 and the display device 24). Start (S518). Specifically, the AR information setting unit accepts editing of the AR information via the user interface, and the AR information generated thereby is information (information indicated in the world coordinate system) indicating the predetermined position in the predetermined range. Information associated with certain predetermined position identification information is stored as AR information 221. The user can set AR information anew, or can set AR information 221 stored in advance.
　ＡＲ情報の設定に際し、ＡＲ情報設定部は、設定中のＡＲ情報をカメラ画像に重ねて表示装置２４に表示する。またＡＲ情報設定部は、図６に示すように、ＡＲ情報利用者３の位置と視線方向やＡＲ情報の位置と方向を含む位置情報７と、ＡＲ情報利用者３の近傍の地図情報８を表示する（Ｓ５１９）。これによりユーザは現場のカメラ画像を確認しながら正確な位置（世界座標系の位置）にＡＲ情報を設定することができる。尚、本実施形態では、ＡＲ情報設定部にＡＲ情報利用者３の近くに任意に設定した原点からの相対位置を位置情報７として表示させているが、ＡＲ情報設定部に緯度、経度からなる世界座標系の座標を位置情報７として表示させてもよい。 When setting the AR information, the AR information setting unit superimposes the AR information being set on the camera image and displays it on the display device 24. In addition, as shown in FIG. 6, the AR information setting unit includes the position information 7 including the position and sight line direction of the AR information user 3 and the position and direction of the AR information, and the map information 8 in the vicinity of the AR information user 3 Display (S519). As a result, the user can set the AR information at the correct position (the position of the world coordinate system) while checking the camera image of the site. In the present embodiment, although the relative position from the origin arbitrarily set near the AR information user 3 is displayed as the position information 7 in the AR information setting unit, the AR information setting unit includes latitude and longitude. Coordinates of the world coordinate system may be displayed as position information 7.
　ＡＲ情報の設定が終了すると（Ｓ５２０：ＹＥＳ）、ＡＲ情報提供装置２０のＡＲ情報送信部は、生成したＡＲ情報を、当該ＡＲ情報の所定位置特定情報（世界座標系）とともにＡＲ情報表示装置１０に送信する（Ｓ５２１）。Ｓ５１７～Ｓ５２１の処理は繰り返し行われる。 When the setting of the AR information is completed (S520: YES), the AR information transmitting unit of the AR information providing device 20 combines the generated AR information with the predetermined position specifying information (world coordinate system) of the AR information to display the AR information display device 10. (S521). The processes of S517 to S521 are repeated.
　ＡＲ情報表示装置１０のＡＲ情報受信部は、ＡＲ情報提供装置２０から送られてくるＡＲ情報と所定位置特定情報（世界座標系）とを受信する（Ｓ５２２）。続いて、ＡＲ情報表示装置１０の範囲特定情報取得部は、現在における範囲特定情報（世界座標系）を新たに取得する（Ｓ５２３）。続いて、ＡＲ情報表示装置１０のＡＲ情報表示部は、新たに取得した範囲特定情報（世界座標系）と受信した所定位置特定情報（世界座標系）とを照合し、ＡＲ情報の表示装置１３の表示画面１３１の座標系における表示位置を求める（Ｓ５２４）。このとき、ＡＲ情報表示装置１０のＡＲ情報表示部は、所定位置特定情報に含まれるＡＲ情報表示装置１０の現在位置とカメラ１５が向いている方向からなる位置情報７とを表示する（Ｓ５２５）。 The AR information receiving unit of the AR information display device 10 receives the AR information sent from the AR information providing device 20 and the predetermined position specifying information (world coordinate system) (S522). Subsequently, the range specification information acquisition unit of the AR information display device 10 newly acquires range specification information (world coordinate system) at present (S523). Subsequently, the AR information display unit of the AR information display device 10 collates the newly acquired range identification information (world coordinate system) with the received predetermined position identification information (world coordinate system) to display the AR information display device 13 The display position in the coordinate system of the display screen 131 is determined (S524). At this time, the AR information display unit of the AR information display device 10 displays the current position of the AR information display device 10 included in the predetermined position specification information and the position information 7 including the direction in which the camera 15 is facing (S525) .
　続いて、ＡＲ情報表示装置１０のＡＲ情報表示部は、ＡＲ情報の表示位置が、新たに取得した範囲特定情報から特定される範囲内か否かを判断する（Ｓ５２６）。ＡＲ情報の表示位置が新たに取得した範囲特定情報から特定される範囲内である場合（Ｓ５２６：ＹＥＳ）、ＡＲ情報表示部は、表示画面１３１における、Ｓ５２４で求めた表示位置にＡＲ情報を表示する（Ｓ５２７）。その後、処理はＳ５２２に戻る。 Subsequently, the AR information display unit of the AR information display device 10 determines whether the display position of the AR information is within the range specified from the newly acquired range specifying information (S526). If the display position of the AR information is within the range specified from the newly acquired range specifying information (S526: YES), the AR information display unit displays AR information at the display position obtained in S524 on the display screen 131. (S527). Thereafter, the process returns to S522.
　一方、ＡＲ情報の表示位置が範囲特定情報から特定される範囲内でない場合（Ｓ５２６：ＮＯ）、ＡＲ情報表示部は、Ｓ５２４で求めた表示位置が存在する方向を示す情報や、ＡＲ情報が変更された旨の情報等の代替のオブジェクトを表示画面１３１に表示する（Ｓ５２８）。その後、処理はＳ５２２に戻る。 On the other hand, when the display position of the AR information is not within the range specified from the range specifying information (S526: NO), the AR information display unit changes the information indicating the direction in which the display position obtained in S524 exists, and the AR information The alternative object such as the information indicating that it has been done is displayed on the display screen 131 (S528). Thereafter, the process returns to S522.
　図７（ａ）に、Ｓ５２７でＡＲ情報表示部が表示する画面の一例を、図７（ｂ）に、Ｓ５２８でＡＲ情報表示部が表示する画面の一例を、夫々示す。図７（ａ）に示すように、ＡＲ情報の表示位置が範囲特定情報から特定される範囲内であるため、表示画面１３１のＳ５２４で求めた表示位置にＡＲ情報（符号７１１）と位置情報７が表示されている。一方、図７（ｂ）では、ＡＲ情報の表示位置が範囲特定情報から特定される範囲内でないため、Ｓ５２４で求めた表示位置が存在する方向を示す情報（符号７１２）と位置情報７が表示画面１３１に表示されている。 FIG. 7A shows an example of a screen displayed by the AR information display unit in S527, and FIG. 7B shows an example of a screen displayed by the AR information display unit in S528. As shown in FIG. 7A, since the display position of the AR information is within the range specified from the range specifying information, the AR information (symbol 711) and the position information 7 are displayed at the display position obtained in S524 of the display screen 131. Is displayed. On the other hand, in FIG. 7B, since the display position of the AR information is not within the range specified from the range specifying information, information (code 712) indicating the direction in which the display position obtained in S524 exists and the position information 7 are displayed. It is displayed on the screen 131.
　以上に説明したように、第１実施形態のＡＲ情報提供システム１においては、ＡＲ情報表示装置１０が、ＡＲ情報提供装置２０から送られてくるＡＲ情報と所定位置特定情報（世界座標系）に基づきＡＲ情報の表示を制御するので、位置ずれを生じさせずに精度よくＡＲ情報を提供することができる。またＡＲ情報表示装置１０からＡＲ情報提供装置２０へは、画像データと範囲特定情報を送信するのみであり、ＡＲ情報提供装置２０からＡＲ情報表示装置へは、ＡＲ情報と所定位置特定情報を送信するのみである。そのため、通信量の増大を抑えつつ効率よくＡＲ情報提供装置２０からＡＲ情報表示装置１０にＡＲ情報を提供することができる。 As described above, in the AR information providing system 1 according to the first embodiment, the AR information display device 10 includes the AR information sent from the AR information providing device 20 and the predetermined position specifying information (world coordinate system). Since the display of AR information is controlled based on this, AR information can be provided with high accuracy without causing positional deviation. Further, only the image data and the range specifying information are transmitted from the AR information display device 10 to the AR information providing device 20, and the AR information and the predetermined position specifying information are transmitted from the AR information providing device 20 to the AR information display device. Only. Therefore, AR information can be efficiently provided from the AR information provision device 20 to the AR information display device 10 while suppressing an increase in the amount of communication.
［第２実施形態］
　第１実施形態のＡＲ情報提供システム１では、ＡＲ情報表示装置１０からＡＲ情報提供装置２０に画像データを送信し、ＡＲ情報提供装置２０が、ＡＲ情報の設定に際してＡＲ情報をカメラ画像に重ねて表示装置２４に表示する。第２実施形態のＡＲ情報提供システム１では、ＡＲ情報表示装置１０からＡＲ情報提供装置２０に画像データを送信せず、ＡＲ情報提供装置２０は、記憶している地図情報２２２にＡＲ情報を重ねて表示装置２４に表示しつつ、ユーザとの対話処理によりＡＲ情報の設定を行う。 Second Embodiment
In the AR information provision system 1 of the first embodiment, image data is transmitted from the AR information display device 10 to the AR information provision device 20, and the AR information provision device 20 superimposes AR information on the camera image when setting AR information. It is displayed on the display device 24. In the AR information provision system 1 of the second embodiment, the AR information provision device 20 does not transmit image data from the AR information display device 10 to the AR information provision device 20, and the AR information provision device 20 overlaps AR information on the stored map information 222. While displaying on the display unit 24, AR information is set by interactive processing with the user.
　図８に第２実施形態のＡＲ情報表示装置１０の主な構成を示している。同図に示すように、第２実施形態のＡＲ情報表示装置１０は、プロセッサ１１、記憶装置１２、表示装置１３、通信装置１４、カメラ１５、及び各種センサ１６を備える。これらの構成は基本的に第１実施形態のＡＲ情報表示装置１０と同様である。但し、第２実施形態のＡＲ情報表示装置１０は、記憶装置１２が、画像データ送信ＰＧ１１５（画像データ送信部）を記憶していない点で第１実施形態のＡＲ情報表示装置１０と構成が異なる。 The main structure of AR information display apparatus 10 of 2nd Embodiment is shown in FIG. As shown in the figure, the AR information display device 10 of the second embodiment includes a processor 11, a storage device 12, a display device 13, a communication device 14, a camera 15, and various sensors 16. These configurations are basically the same as the AR information display device 10 of the first embodiment. However, the AR information display device 10 of the second embodiment differs in configuration from the AR information display device 10 of the first embodiment in that the storage device 12 does not store the image data transmission PG 115 (image data transmission unit). .
　図９に第２実施形態のＡＲ情報提供装置２０の主な構成を示している。同図に示すように、第２実施形態のＡＲ情報提供装置２０は、プロセッサ２１、記憶装置２２、入力装置２３、表示装置２４、及び通信装置２５を備える。これらの構成は基本的に第１実施形態のＡＲ情報提供装置２０と同様である。但し、第２実施形態のＡＲ情報提供装置２０は、記憶装置２２が画像データ受信ＰＧ２１２を記憶していない点、及び、ＡＲ情報設定部が、ユーザとの対話処理によりＡＲ情報の編集を受け付ける際、設定中のＡＲ情報を、地図情報２２２に基づく３次元の地図画像に重ねて表示装置２４に表示する点で第１実施形態のＡＲ情報提供装置２０と構成が異なる。 FIG. 9 shows the main configuration of the AR information provision device 20 of the second embodiment. As shown in the drawing, the AR information provision device 20 of the second embodiment includes a processor 21, a storage device 22, an input device 23, a display device 24, and a communication device 25. These configurations are basically the same as the AR information provision device 20 of the first embodiment. However, in the AR information provision device 20 of the second embodiment, the storage device 22 does not store the image data reception PG 212, and when the AR information setting unit receives editing of AR information by interactive processing with the user. The configuration differs from that of the AR information providing device 20 of the first embodiment in that AR information being set is superimposed on a three-dimensional map image based on the map information 222 and displayed on the display device 24.
　図１０は、第２実施形態のＡＲ情報提供システム１において行われる処理を説明するフローチャートである。以下、同図とともに上記処理について説明する。 FIG. 10 is a flowchart for explaining the processing performed in the AR information providing system 1 of the second embodiment. Hereinafter, the process will be described with reference to FIG.
　まずＡＲ情報表示装置１０の範囲特定情報取得部は、範囲特定情報を取得し（Ｓ１０１５）、取得した範囲特定情報をＡＲ情報提供装置２０に送信する（Ｓ１０１６）。これらの処理は図５のＳ５１１，Ｓ５１２の処理と同様であるので説明を省略する。 First, the range identification information acquisition unit of the AR information display device 10 acquires range identification information (S1015), and transmits the acquired range identification information to the AR information provision device 20 (S1016). These processes are the same as the processes of S511 and S512 in FIG.
　ＡＲ情報提供装置２０の範囲特定情報受信部は、ＡＲ情報表示装置１０から送られてくる範囲特定情報を受信する（Ｓ１０１７）。 The range specifying information receiving unit of the AR information providing device 20 receives the range specifying information sent from the AR information display device 10 (S1017).
　続いて、ＡＲ情報提供装置２０のＡＲ情報設定部は、ユーザインタフェース（入力装置２３及び表示装置２４）を介した対話処理により、受信した範囲特定情報に基づき特定される所定範囲についてＡＲ情報の設定を開始する（Ｓ１０１８）。具体的には、ＡＲ情報設定部は、ユーザインタフェースを介してＡＲ情報の編集を受け付け、それにより生成されたＡＲ情報を上記所定範囲の所定位置（世界座標系）を示す情報である所定位置特定情報に対応づけた情報をＡＲ情報２２１として記憶する。尚、ユーザは、上記ＡＲ情報を新規に設定することもできるし、予め記憶しているＡＲ情報２２１を利用して設定することもできる。ＡＲ情報の設定に際し、ＡＲ情報設定部は、設定中のＡＲ情報を地図情報２２２に基づく３次元の地図画像に重ねて表示装置２４に表示する。これによりユーザは３次元の地図画像を確認しながら正確な位置にＡＲ情報を設定することができる。 Subsequently, the AR information setting unit of the AR information provision device 20 sets AR information for a predetermined range specified based on the received range specifying information by interactive processing via the user interface (the input device 23 and the display device 24). Start (S1018). Specifically, the AR information setting unit accepts editing of the AR information via the user interface, and AR information generated thereby is specified position specification that is information indicating a predetermined position (world coordinate system) in the predetermined range. Information associated with the information is stored as AR information 221. The user can set the AR information anew or can use AR information 221 stored in advance. When setting the AR information, the AR information setting unit superimposes the AR information being set on a three-dimensional map image based on the map information 222 and displays it on the display device 24. As a result, the user can set AR information at an accurate position while checking a three-dimensional map image.
　Ｓ１０１８～Ｓ１０２７の処理は、Ｓ５１９及びＳ５２５に相当する処理を除いて図５のＳ５１８～Ｓ５２８の処理と同様であるので説明を省略する。 The processes of S1018 to S1027 are the same as the processes of S518 to S528 in FIG. 5 except for the processes corresponding to S519 and S525, and thus the description thereof is omitted.
　以上に説明したように、第２実施形態のＡＲ情報提供システム１においては、第１実施形態と同様に、ＡＲ情報表示装置１０が、ＡＲ情報提供装置２０から送られてくるＡＲ情報と所定位置特定情報（世界座標系）とに基づきＡＲ情報の表示を制御するので、位置ずれを生じさせずに精度よくＡＲ情報を提供することができる。またＡＲ情報表示装置１０からＡＲ情報提供装置２０に画像データを送信せず、ＡＲ情報提供装置２０は、設定中のＡＲ情報を地図情報２２２に基づく３次元の地図画像に重ねて表示装置２４に表示しつつ、ユーザとの対話処理によりＡＲ情報の設定を行う。そのため、通信量の増大を抑えつつ効率よくＡＲ情報提供装置２０からＡＲ情報表示装置１０にＡＲ情報を提供することができる。 As described above, in the AR information providing system 1 according to the second embodiment, as in the first embodiment, the AR information display device 10 includes the AR information and the predetermined position sent from the AR information providing device 20. Since the display of AR information is controlled based on the specific information (world coordinate system), AR information can be provided with high accuracy without causing positional deviation. In addition, the AR information providing device 20 does not transmit image data from the AR information display device 10 to the AR information providing device 20, and the AR information providing device 20 superimposes the AR information being set on the three-dimensional map image based on the map information 222. While displaying it, AR information is set by interactive processing with the user. Therefore, AR information can be efficiently provided from the AR information provision device 20 to the AR information display device 10 while suppressing an increase in the amount of communication.
［第３実施形態］
　第１実施形態のＡＲ情報提供システム１では、ＡＲ情報表示装置１０からＡＲ情報提供装置２０に範囲特定情報を送信し、ＡＲ情報提供装置２０が、範囲特定情報を受信し、受信した範囲特定情報に基づき特定される所定範囲についてＡＲ情報の設定を行っている。第３実施形態のＡＲ情報提供システム１では、ＡＲ情報表示装置１０は範囲特定情報をタイムスタンプと対応づけた履歴情報として記憶し、ＡＲ情報表示装置１０からＡＲ情報提供装置２０に範囲特定情報を送信しない。一方、ＡＲ情報提供装置２０は、ＡＲ情報の設定に際してＡＲ情報をカメラ画像に重ねて表示装置２４に表示しつつユーザとの対話処理によりカメラ画像の範囲（所定範囲）についてＡＲ情報の設定を行い、設定したＡＲ情報を、当該ＡＲ情報に対応づけられている所定位置特定情報、及びＡＲ情報の設定に際して重ねたカメラ画像（画像データ）のタイムスタンプとともにＡＲ情報表示装置１０に送信する。そしてＡＲ情報表示装置１０は、記憶している履歴情報からＡＲ情報とともに受信したタイムスタンプに対応する範囲特定情報の履歴を取得し、ＡＲ情報の世界座標系における表示位置を特定する。 Third Embodiment
In the AR information providing system 1 of the first embodiment, the range specifying information is transmitted from the AR information display device 10 to the AR information providing device 20, and the AR information providing device 20 receives the range specifying information, and the received range specifying information AR information is set for a predetermined range specified based on the above. In the AR information providing system 1 of the third embodiment, the AR information display device 10 stores the range specifying information as history information associated with the time stamp, and the AR information providing device 20 transmits the range specifying information to the AR information providing device 20. Do not send On the other hand, the AR information provision device 20 sets AR information on the range (predetermined range) of the camera image by interactive processing with the user while displaying the AR information on the camera image while superimposing the AR information upon setting the AR information. The set AR information is transmitted to the AR information display device 10 together with predetermined position specifying information correlated with the AR information, and a time stamp of a camera image (image data) superimposed upon setting of the AR information. Then, the AR information display device 10 acquires the history of the range specifying information corresponding to the time stamp received together with the AR information from the stored history information, and specifies the display position of the AR information in the world coordinate system.
　図１１に、第３実施形態のＡＲ情報表示装置１０の主な構成を示している。同図に示すように、第３実施形態のＡＲ情報表示装置１０は、プロセッサ１１、記憶装置１２、表示装置１３、通信装置１４、カメラ１５、及び各種センサ１６を備える。これらの構成は基本的に第１実施形態のＡＲ情報表示装置１０と同様である。但し、第３実施形態のＡＲ情報表示装置１０は、記憶装置１２が、範囲特定情報送信ＰＧ１１３を記憶していない点、並びに、記憶装置１２が、範囲特定情報履歴記憶ＰＧ１１６、範囲特定情報履歴特定ＰＧ１１７、及び範囲特定情報履歴１２４を記憶している点で第１実施形態のＡＲ情報表示装置１０と構成が異なる。 FIG. 11 shows the main configuration of the AR information display device 10 of the third embodiment. As shown in the drawing, the AR information display device 10 of the third embodiment includes a processor 11, a storage device 12, a display device 13, a communication device 14, a camera 15, and various sensors 16. These configurations are basically the same as the AR information display device 10 of the first embodiment. However, in the AR information display device 10 of the third embodiment, the storage device 12 does not store the range specification information transmission PG 113, and the storage device 12 stores the range specification information history storage PG 116, range specification information history specification. The configuration is different from that of the AR information display device 10 of the first embodiment in that the PG 117 and the range specification information history 124 are stored.
　尚、以下の説明において、プロセッサ１１が範囲特定情報履歴記憶ＰＧ１１６を実行することにより実現される機能のことを範囲特定情報履歴記憶部と、プロセッサ１１が範囲特定情報履歴特定ＰＧ１１７を実行することにより実現される機能のことを範囲特定情報履歴特定部と、夫々称する。 In the following description, the function realized by the processor 11 executing the range identification information history storage PG 116 is described by the range identification information history storage unit and the processor 11 executing the range identification information history identification PG 117. The function to be realized is referred to as a range specifying information history specifying unit.
　図１２に第３実施形態のＡＲ情報提供装置２０の主な構成を示している。同図に示すように、第３実施形態のＡＲ情報提供装置２０は、プロセッサ２１、記憶装置２２、入力装置２３、表示装置２４、及び通信装置２５を備える。これらの構成は基本的に第１実施形態のＡＲ情報提供装置２０と同様である。但し、第２実施形態のＡＲ情報提供装置２０は、記憶装置２２が範囲特定情報受信ＰＧ２１１を記憶していない点、並びに、ＡＲ情報設定部がＡＲ情報とともに画像データに基づくカメラ画像の座標系（以下、カメラ画像座標系と称する。）による所定位置特定情報を生成する点、ＡＲ情報送信部が、ＡＲ情報ととともに、所定位置特定情報（カメラ画像座標系）と、ＡＲ情報の設定に際し重ねて表示したカメラ画像のタイムスタンプとをＡＲ情報表示装置１０に送信する点で、第１実施形態のＡＲ情報提供装置２０と構成が異なる。 FIG. 12 shows the main configuration of the AR information provision device 20 of the third embodiment. As shown in the drawing, the AR information provision device 20 of the third embodiment includes a processor 21, a storage device 22, an input device 23, a display device 24, and a communication device 25. These configurations are basically the same as the AR information provision device 20 of the first embodiment. However, in the AR information provision device 20 of the second embodiment, the storage device 22 does not store the range specification information reception PG 211, and the AR information setting unit performs a coordinate system of a camera image based on image data together with AR information ( Hereinafter, the point for generating predetermined position specifying information according to the camera image coordinate system), the AR information transmitting unit overlaps with the AR information when setting the AR information with the predetermined position specifying information (camera image coordinate system) The configuration is different from that of the AR information providing device 20 of the first embodiment in that the time stamp of the displayed camera image is transmitted to the AR information display device 10.
　図１３は、第３実施形態のＡＲ情報提供システム１において行われる処理を説明するフローチャートである。以下、同図とともに上記処理について説明する。 FIG. 13 is a flowchart illustrating processing performed in the AR information provision system 1 of the third embodiment. Hereinafter, the process will be described with reference to FIG.
　まず同図におけるＳ１３１１～Ｓ１３１４の処理は、図５におけるＳ５１１～Ｓ５１４の処理と同様であるので説明を省略する。 First, the processing of S1311 to S1314 in the same figure is the same as the processing of S511 to S514 in FIG.
　Ｓ１３１５～Ｓ１３１６では、ＡＲ情報表示装置１０の範囲特定情報取得部が、範囲特定情報を取得し（Ｓ１３１５）、ＡＲ情報表示装置１０の範囲特定情報履歴記憶部が、範囲特定情報取得部が取得した範囲特定情報を記憶装置１２に範囲特定情報履歴１２４として逐次蓄積記憶する（Ｓ１３１６）。ここで範囲特定情報履歴１２４として蓄積記憶される各範囲特定情報には、画像データのタイムスタンプとの対応をとるためのタイムスタンプが付帯する。本例では、範囲特定情報は、ＡＲ情報表示装置１０の現在位置を世界座標系で示した情報とカメラ１５が向いている方向を示す情報を含むものとする。範囲特定情報取得部は、例えば、ＧＰＳに基づく情報とマーカを撮影した画像データに基づき範囲特定情報を取得する。また範囲特定情報取得部は、例えば、撮影されたマーカのサイズとマーカの縦横の歪みをマーカ情報１２３と照合することにより、ＡＲ情報表示装置１０の正確な現在位置やカメラ１５が向いている正確な方向を取得する。また範囲特定情報取得部は、例えば、深度カメラから取得されるデータと地図情報１２２とを照合することにより、ＡＲ情報表示装置１０の正確な現在位置やカメラ１５が向いている正確な方向を取得する。また範囲特定情報取得部は、例えば、その他の公知の方法により、ＡＲ情報表示装置１０の正確な現在位置やカメラ１５が向いている正確な方向を取得する。 In S1315-S1316, the range specification information acquisition unit of the AR information display device 10 acquires range specification information (S1315), and the range specification information history storage unit of the AR information display device 10 is acquired by the range specification information acquisition unit The range specifying information is sequentially accumulated and stored as the range specifying information history 124 in the storage device 12 (S1316). Here, each range specifying information accumulated and stored as the range specifying information history 124 is accompanied by a time stamp for taking correspondence with the time stamp of the image data. In this example, the range specification information includes information indicating the current position of the AR information display device 10 in the world coordinate system and information indicating the direction in which the camera 15 is facing. The range identification information acquisition unit acquires range identification information based on, for example, information based on GPS and image data obtained by capturing a marker. In addition, the range identification information acquisition unit, for example, collates the size of the photographed marker and the vertical and horizontal distortion of the marker with the marker information 123, and thereby corrects the correct current position of the AR information display device 10 and the accuracy at which the camera 15 is facing. Get the right direction. Further, the range specification information acquisition unit acquires, for example, the accurate current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by collating the data acquired from the depth camera with the map information 122. Do. The range specification information acquisition unit acquires, for example, the correct current position of the AR information display device 10 and the correct direction in which the camera 15 is facing, by another known method.
　ＡＲ情報提供装置２０のＡＲ情報設定部は、ユーザインタフェース（入力装置２３及び表示装置２４）を介した対話処理により、表示装置２４に表示されているカメラ画像（以下、対象カメラ画像と称する。）の範囲（所定範囲）についてＡＲ情報の設定を開始する（Ｓ１３１８）。具体的には、ＡＲ情報設定部は、ユーザインタフェースを介してＡＲ情報の編集を受け付け、それにより生成されたＡＲ情報を上記所定範囲の所定位置を示す情報（対象カメラ画像の座標系（以下、カメラ画像座標系と称する。）で示した情報）である所定位置特定情報（カメラ画像座標系）に対応づけた情報をＡＲ情報２２１として記憶する。尚、ユーザは、上記ＡＲ情報を新規に設定することもできるし、予め記憶しているＡＲ情報２２１を利用して設定することもできる。ＡＲ情報の設定に際し、ＡＲ情報設定部は、設定中のＡＲ情報をカメラ画像に重ねて表示装置２４に表示する。これによりユーザはカメラ画像を確認しながら正確な位置にＡＲ情報を設定することができる。 The AR information setting unit of the AR information providing device 20 is a camera image (hereinafter referred to as a target camera image) displayed on the display device 24 by interactive processing via the user interface (the input device 23 and the display device 24). The setting of AR information is started for the range (predetermined range) of (S1318). Specifically, the AR information setting unit receives editing of the AR information via the user interface, and generates AR information indicating information indicating a predetermined position of the predetermined range (a coordinate system of the target camera image (hereinafter referred to as Information associated with predetermined position specifying information (camera image coordinate system), which is information shown in the camera image coordinate system) is stored as AR information 221. The user can set the AR information anew or can use AR information 221 stored in advance. When setting the AR information, the AR information setting unit superimposes the AR information being set on the camera image and displays it on the display device 24. As a result, the user can set AR information at an accurate position while checking the camera image.
　ＡＲ情報の編集が終了すると（Ｓ１３１９：ＹＥＳ）、ＡＲ情報提供装置２０のＡＲ情報送信部は、生成したＡＲ情報を、当該ＡＲ情報に対応づけられているカメラ画像座標系で表された所定位置特定情報、及びＳ１３１３で受信した対象カメラ画像の画像データのタイムスタンプとともにＡＲ情報表示装置１０に送信する（Ｓ１３２０）。 When the editing of the AR information is completed (S1319: YES), the AR information transmission unit of the AR information provision device 20 determines the generated AR information at a predetermined position represented by the camera image coordinate system associated with the AR information. The identification information and the time stamp of the image data of the target camera image received in S1313 are transmitted to the AR information display device 10 (S1320).
　ＡＲ情報表示装置１０のＡＲ情報受信部は、ＡＲ情報提供装置２０から送られてくるＡＲ情報、所定位置特定情報（カメラ画像座標系）、及びタイムスタンプを受信する（Ｓ１３２１）。 The AR information receiving unit of the AR information display device 10 receives the AR information, the predetermined position specifying information (camera image coordinate system), and the time stamp sent from the AR information providing device 20 (S1321).
　続いて、ＡＲ情報表示装置１０の範囲特定情報履歴特定部は、受信したタイムスタンプに対応する範囲特定情報（世界座標系）を範囲特定情報履歴１２４から特定（取得）する（Ｓ１３２２）。 Subsequently, the range specification information history specification unit of the AR information display device 10 specifies (acquires) range specification information (world coordinate system) corresponding to the received time stamp from the range specification information history 124 (S1322).
　続いて、ＡＲ情報表示装置１０のＡＲ情報表示部は、範囲特定情報履歴特定部が特定した上記範囲特定情報（世界座標系）に基づき、Ｓ１３２１で受信した所定位置特定情報（カメラ画像座標系）を世界座標系で表された情報に変換する（Ｓ１３２３）。 Subsequently, the AR information display unit of the AR information display device 10 receives the predetermined position specification information (camera image coordinate system) received in S1321 based on the range specification information (world coordinate system) specified by the range specification information history specification unit. Are converted into information represented in the world coordinate system (S1323).
　続いて、ＡＲ情報表示装置１０の範囲特定情報取得部は、現在での範囲特定情報（世界座標系）を新たに取得する（Ｓ１３２４）。 Subsequently, the range specification information acquisition unit of the AR information display device 10 newly acquires the current range specification information (world coordinate system) (S1324).
　続いて、ＡＲ情報表示装置１０のＡＲ情報表示部は、Ｓ１３２４で新たに取得した範囲特定情報（世界座標系）とＳ１３２３で世界座標系に変換したＡＲ情報の所定位置特定情報とを照合し、ＡＲ情報の表示装置１３の表示画面１３１の表示位置を求める（Ｓ１３２５）。 Subsequently, the AR information display unit of the AR information display device 10 collates the range identification information (world coordinate system) newly acquired in S1324 with the predetermined position identification information of the AR information converted into the world coordinate system in S1323, The display position of the display screen 131 of the display device 13 of AR information is obtained (S1325).
　続いて、ＡＲ情報表示装置１０のＡＲ情報表示部は、ＡＲ情報の表示位置が、範囲特定情報から特定される範囲内か否かを判断する（Ｓ１３２６）。ＡＲ情報の表示位置が範囲特定情報から特定される範囲内である場合（Ｓ１３２６：ＹＥＳ）、ＡＲ情報表示部は、表示画面１３１のＳ１３２５で求めた表示位置にＡＲ情報を表示する（Ｓ１３２７）。その後、処理はＳ１３２１に戻る。 Subsequently, the AR information display unit of the AR information display device 10 determines whether the display position of the AR information is within the range specified from the range specification information (S1326). If the display position of the AR information is within the range specified from the range specification information (S1326: YES), the AR information display unit displays the AR information at the display position obtained in S1325 of the display screen 131 (S1327). Thereafter, the process returns to S1321.
　一方、ＡＲ情報の表示位置が範囲特定情報から特定される範囲内でない場合（Ｓ１３２６：ＮＯ）、ＡＲ情報表示部は、Ｓ１３２５で求めた表示位置が存在する方向を示す情報や、ＡＲ情報が変更された旨の情報等を含む代替のＡＲ情報（ＡＲオブジェクト）を表示画面１３１に表示する（Ｓ１３２８）。その後、処理はＳ１３２１に戻る。 On the other hand, when the display position of the AR information is not within the range specified from the range specification information (S1326: NO), the AR information display unit changes the information indicating the direction in which the display position obtained in S1325 exists, and the AR information The alternative AR information (AR object) including the information indicating that it has been done is displayed on the display screen 131 (S1328). Thereafter, the process returns to S1321.
　以上に説明したように、第３実施形態のＡＲ情報提供システム１においては、ＡＲ情報表示装置１０からＡＲ情報提供装置２０に範囲特定情報を送信せず、ＡＲ情報提供装置２０は、ＡＲ情報表示装置１０から送られてくる画像データに基づくカメラ画像をＡＲ情報に重ねて表示装置２４に表示しつつ、ユーザとの対話処理によりＡＲ情報の設定を行う。そのため、ＡＲ情報提供装置２０側では画像データと範囲特定情報の対応をとる必要がなく、通信量の増大を抑えつつ効率よくＡＲ情報提供装置２０からＡＲ情報表示装置１０にＡＲ情報を提供することができる。 As described above, in the AR information providing system 1 of the third embodiment, the AR information providing apparatus 20 does not transmit the range specifying information from the AR information display apparatus 10 to the AR information providing apparatus 20, and the AR information providing apparatus 20 displays the AR information display While superimposing a camera image based on image data sent from the apparatus 10 on AR information and displaying it on the display device 24, AR information is set by interactive processing with the user. Therefore, the AR information providing apparatus 20 does not have to correspond to the image data and the range specifying information, and the AR information providing apparatus 20 efficiently provides AR information to the AR information display apparatus 10 while suppressing an increase in the communication amount. Can.
　ところで、本発明は以上に説明した実施形態に限定されるものではなく、その要旨を逸脱しない範囲で種々変更可能であることはいうまでもない。例えば、上記の実施の形態は本発明を分かりやすく説明するために詳細に説明したものであり、必ずしも説明した全ての構成を備えるものに限定されるものではない。また上記実施形態の構成の一部について、他の構成の追加・削除・置換をすることが可能である。 The present invention is not limited to the embodiments described above, and it goes without saying that various modifications can be made without departing from the scope of the invention. For example, the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Further, with respect to a part of the configuration of the above-described embodiment, it is possible to add, delete, and replace other configurations.
　また上記の各構成、機能部、処理部、処理手段等は、それらの一部又は全部を、例えば、集積回路で設計する等によりハードウェアで実現してもよい。また上記の各構成、機能等は、プロセッサがそれぞれの機能を実現するプログラムを解釈し、実行することによりソフトウェアで実現してもよい。各機能を実現するプログラム、テーブル、ファイル等の情報は、メモリやハードディスク、ＳＳＤ等の記録装置、又はＩＣカード、ＳＤカード、ＤＶＤ等の記録媒体に置くことができる。 In addition, each of the configurations, function units, processing units, processing means, etc. described above may be realized by hardware, for example, by designing part or all of them with an integrated circuit. Further, each configuration, function, etc. described above may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as a program, a table, and a file that implements each function can be placed in a memory, a hard disk, a recording device such as an SSD, or a recording medium such as an IC card, an SD card, or a DVD.
　また上記の各図において、制御線や情報線は説明上必要と考えられるものを示しており、必ずしも実装上の全ての制御線や情報線を示しているとは限らない。例えば、実際にはほとんど全ての構成が相互に接続されていると考えてもよい。 Further, in each of the above-mentioned drawings, control lines and information lines indicate what is considered to be necessary for explanation, and not all control lines and information lines on mounting are necessarily shown. For example, in practice it may be considered that almost all configurations are mutually connected.
　また以上に説明した各装置の各種機能部、各種処理部、各種データベースの配置形態は一例に過ぎない。各種機能部、各種処理部、各種データベースの配置形態はＡＲ情報提供システム１が備えるハードウェアやソフトウェアの性能、処理効率、通信効率等の観点から最適な配置形態に変更し得る。 The arrangement of the various functional units, the various processing units, and the various databases of each device described above is merely an example. The arrangement form of the various functional units, the various processing units, and the various databases can be changed to an optimum arrangement form in view of the performance, processing efficiency, communication efficiency, etc. of hardware and software provided in the AR information providing system 1.
　また前述したデータベースの構成（スキーマ（Schema）等）は、リソースの効率的な利用、処理効率向上、アクセス効率向上、検索効率向上等の観点から柔軟に変更し得る。 Further, the configuration of the database (schema or the like) can be flexibly changed from the viewpoint of efficient use of resources, improvement of processing efficiency, improvement of access efficiency, improvement of search efficiency and the like.
１　ＡＲ情報提供システム、３　ＡＲ情報利用者、４　ＡＲ情報提供者、５　通信ネットワーク、６　ＡＲ情報、７　位置情報、８　地図情報、１０　ＡＲ情報表示装置、１１１　ＡＲ情報表示ＰＧ（ＡＲ情報表示部）、１１２　範囲特定情報取得ＰＧ（範囲特定情報取得部）、１１３　範囲特定情報送信ＰＧ（範囲特定情報送信部）、１１４　ＡＲ情報受信ＰＧ（ＡＲ情報受信部）、１１５　画像データ送信ＰＧ（画像データ送信部）、１１６　範囲特定情報履歴記憶ＰＧ（範囲特定情報履歴記憶部）、１１７　範囲特定情報履歴特定ＰＧ（範囲特定情報履歴特定部）、１２１　ＡＲ情報、１２２　地図情報、１２３　マーカ情報、１２４　範囲特定情報履歴、２０　ＡＲ情報提供装置、２１１　範囲特定情報受信ＰＧ（範囲特定情報受信部）、２１２　画像データ受信ＰＧ（画像データ受信部）、２１３　ＡＲ情報設定ＰＧ（ＡＲ情報設定部）、２１４　ＡＲ情報送信ＰＧ（ＡＲ情報送信部）、２２１　ＡＲ情報、２２２　地図情報 1 AR Information Providing System, 3 AR Information User, 4 AR Information Provider, 5 Communication Network, 6 AR Information, 7 Position Information, 8 Map Information, 10 AR Information Display Device, 111 AR Information Display PG (AR Information Display Unit 112, Range identification information acquisition PG (range identification information acquisition unit), 113 Range identification information transmission PG (range identification information transmission unit) 114 AR information reception PG (AR information reception unit) 115 Image data transmission PG (image data) Transmission unit), 116 range identification information history storage PG (range identification information history storage unit), 117 range identification information history identification PG (range identification information history identification unit), 121 AR information, 122 map information, 123 marker information, 124 range Specific information history, 20 AR information provision device, 211 range identification information reception PG (range identification information reception ), 212 image data receiving PG (image data receiving unit), 213 AR information setting PG (AR information setting unit), 214 AR information transmission PG (AR information transmitter), 221 AR information, 222 map information

Claims (16)

  1. 　第１情報処理装置と、
    　前記第１情報処理装置と通信可能に接続する第２情報処理装置と、
    　を含んで構成されるＡＲ情報提供システムであって、
    　前記第１情報処理装置は、
    　現実の所定範囲の風景の所定位置にＡＲ情報を重ねて表示するＡＲ情報表示部と、
    　前記所定範囲を特定する情報である範囲特定情報を取得する範囲特定情報取得部と、
    　前記範囲特定情報を前記第２情報処理装置に送信する範囲特定情報送信部と、
    　を備え、
    　前記第２情報処理装置は、
    　前記範囲特定情報を受信する範囲特定情報受信部と、
      前記範囲特定情報に基づき特定される前記所定範囲の前記所定位置に対応づけて前記ＡＲ情報を設定するＡＲ情報設定部と、
    　設定した前記ＡＲ情報を、前記所定位置を特定する情報である所定位置特定情報とともに前記第１情報処理装置に送信するＡＲ情報送信部と、
    　を備え、
    　前記第１情報処理装置は、
    　前記ＡＲ情報とともに前記所定位置特定情報を受信するＡＲ情報受信部を有し、
    　前記範囲特定情報取得部は、現在の前記範囲特定情報を取得し、
    　前記ＡＲ情報表示部は、受信した前記所定位置特定情報と取得した前記現在の範囲特定情報とに基づき、受信した前記ＡＲ情報の表示位置を特定し、特定した前記表示位置に基づき前記ＡＲ情報の表示を制御する、
    　ＡＲ情報提供システム。     A first information processing apparatus;
    A second information processing apparatus communicably connected to the first information processing apparatus;
    AR information providing system configured to include
    The first information processing apparatus is
    An AR information display unit that displays AR information superimposed on a predetermined position of a real predetermined range of scenery;
    A range identification information acquisition unit that acquires range identification information that is information for identifying the predetermined range;
    A range identification information transmission unit that transmits the range identification information to the second information processing apparatus;
    Equipped with
    The second information processing apparatus is
    A range identification information receiving unit that receives the range identification information;
    An AR information setting unit configured to set the AR information in association with the predetermined position of the predetermined range specified based on the range specification information;
    An AR information transmission unit that transmits the set AR information to the first information processing apparatus together with predetermined position specifying information that is information for specifying the predetermined position;
    Equipped with
    The first information processing apparatus is
    It has an AR information receiving unit that receives the predetermined position specifying information together with the AR information,
    The range identification information acquisition unit acquires the current range identification information;
    The AR information display unit specifies a display position of the received AR information based on the received predetermined position specification information and the acquired current range specification information, and the AR information display unit determines the AR information based on the specified display position. Control the display,
    AR information provision system.
  2. 　請求項１に記載のＡＲ情報提供システムであって、
    　前記範囲特定情報は、前記第１情報処理装置の位置を世界座標系で示した情報、及び前記第１情報処理装置の姿勢を示す情報を含む、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 1, wherein
    The range specification information includes information indicating the position of the first information processing device in a world coordinate system, and information indicating an attitude of the first information processing device.
    AR information provision system.
  3. 　請求項１に記載のＡＲ情報提供システムであって、
    　前記所定位置特定情報は、前記所定位置を世界座標系で示した情報を含む、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 1, wherein
    The predetermined position specification information includes information indicating the predetermined position in a world coordinate system.
    AR information provision system.
  4. 　請求項１に記載のＡＲ情報提供システムであって、
    　前記第１情報処理装置は、
    　現実の風景の画像データを取得する画像データ取得部と、
    　取得した前記画像データを前記第２情報処理装置に送信する送信部と、
    を有し、
    　前記第２情報処理装置は、
    　前記画像データを受信する画像データ受信部と、
    　受信した前記画像データに基づく画像を表示しつつ前記ＡＲ情報の設定に関する操作入力を受け付けるユーザインタフェースと、
    　を有する、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 1, wherein
    The first information processing apparatus is
    An image data acquisition unit for acquiring image data of a real landscape;
    A transmitter configured to transmit the acquired image data to the second information processing apparatus;
    Have
    The second information processing apparatus is
    An image data receiving unit that receives the image data;
    A user interface that receives an operation input related to setting of the AR information while displaying an image based on the received image data;
    Have
    AR information provision system.
  5. 　請求項２に記載のＡＲ情報提供システムであって、
    　前記第１情報処理装置は、前記第１情報処理装置の前記位置及び前記姿勢のうちの少なくともいずれかに関する情報を出力する一つ以上のセンサを有する、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 2, wherein
    The first information processing apparatus includes one or more sensors that output information regarding at least one of the position and the attitude of the first information processing apparatus.
    AR information provision system.
  6. 　請求項５に記載のＡＲ情報提供システムであって、
    　前記センサは、カメラ、深度カメラ、ＧＰＳ、ジャイロセンサ、加速度センサ、及び方位センサのうちの少なくともいずれかである、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 5, wherein
    The sensor is at least one of a camera, a depth camera, a GPS, a gyro sensor, an acceleration sensor, and an orientation sensor.
    AR information provision system.
  7. 　請求項１に記載のＡＲ情報提供システムであって、
    　前記第２情報処理装置は、
    　前記所定範囲を含む地図情報を記憶する記憶部と、
    　前記地図情報に基づく画像を表示しつつ前記ＡＲ情報の設定に関する操作入力を受け付けるユーザインタフェースと、
    　を有する、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 1, wherein
    The second information processing apparatus is
    A storage unit that stores map information including the predetermined range;
    A user interface that receives an operation input related to setting of the AR information while displaying an image based on the map information;
    Have
    AR information provision system.
  8. 　請求項２に記載のＡＲ情報提供システムであって、
    　前記第２情報処理装置は表示装置と接続し、
    　前記ＡＲ情報設定部は、前記第１情報処理装置の位置及び姿勢のうちの少なくともいずれかを示す情報を前記表示装置に表示する、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 2, wherein
    The second information processing apparatus is connected to a display device,
    The AR information setting unit displays information indicating at least one of a position and an attitude of the first information processing apparatus on the display device.
    AR information provision system.
  9. 　第１情報処理装置と、
    　前記第１情報処理装置と通信可能に接続する第２情報処理装置と、
    　を含んで構成されるＡＲ情報提供システムであって、
    　前記第１情報処理装置は、
    　現実の所定範囲の風景の所定位置にＡＲ情報を重ねて表示するＡＲ情報表示部と、
    　前記所定範囲を特定する情報である範囲特定情報を取得する範囲特定情報取得部と、
    　前記範囲特定情報をタイムスタンプに対応づけて蓄積記憶する範囲特定情報履歴記憶部と、
    　前記所定範囲の画像データを取得する画像データ取得部と、
    　取得した前記画像データと前記画像データのタイムスタンプとを前記第２情報処理装置に送信する送信部と、
    　を有し、
    　前記第２情報処理装置は、
    　前記画像データを前記タイムスタンプとともに受信する画像データ受信部と、
    　受信した前記画像データに基づく画像を表示しつつ前記ＡＲ情報の設定に関する操作入力を受け付けるユーザインタフェースと、
    　設定した前記ＡＲ情報を、前記所定位置を特定する情報である所定位置特定情報及び前記タイムスタンプとともに前記第１情報処理装置に送信するＡＲ情報送信部と、
    　を有し、
    　前記第１情報処理装置は、
    　前記ＡＲ情報とともに前記所定位置特定情報及び前記タイムスタンプを受信するＡＲ情報受信部と、
    　蓄積記憶している前記範囲特定情報から、受信した前記タイムスタンプに対応する前記範囲特定情報を特定する範囲特定情報履歴特定部と、
    　を有し、
    　前記範囲特定情報取得部は、現在の前記範囲特定情報を取得し、
    　前記ＡＲ情報表示部は、前記特定した範囲特定情報の履歴、受信した前記所定位置特定情報、及び取得した前記現在の範囲特定情報に基づき、受信した前記ＡＲ情報の表示位置を特定し、特定した前記表示位置に基づき前記ＡＲ情報の表示を制御する、
    　ＡＲ情報提供システム。     A first information processing apparatus;
    A second information processing apparatus communicably connected to the first information processing apparatus;
    AR information providing system configured to include
    The first information processing apparatus is
    An AR information display unit that displays AR information superimposed on a predetermined position of a real predetermined range of scenery;
    A range identification information acquisition unit that acquires range identification information that is information for identifying the predetermined range;
    A range identification information history storage unit for storing the range identification information in association with a time stamp;
    An image data acquisition unit for acquiring image data of the predetermined range;
    A transmitter configured to transmit the acquired image data and a time stamp of the image data to the second information processing apparatus;
    Have
    The second information processing apparatus is
    An image data receiving unit that receives the image data together with the time stamp;
    A user interface that receives an operation input related to setting of the AR information while displaying an image based on the received image data;
    An AR information transmission unit that transmits the set AR information to the first information processing apparatus together with predetermined position specifying information that is information for specifying the predetermined position and the time stamp;
    Have
    The first information processing apparatus is
    An AR information receiving unit that receives the predetermined position specifying information and the time stamp together with the AR information;
    A range specifying information history specifying unit for specifying the range specifying information corresponding to the received time stamp from the range specifying information stored and stored;
    Have
    The range identification information acquisition unit acquires the current range identification information;
    The AR information display unit specifies and specifies the display position of the received AR information based on the history of the specified range specification information, the received predetermined position specification information, and the acquired current range specification information. Controlling display of the AR information based on the display position;
    AR information provision system.
  10. 　請求項９に記載のＡＲ情報提供システムであって、
    　前記所定位置特定情報は、前記所定位置を前記画像データに基づくカメラ画像の座標系で示した情報を含み、
    　前記ＡＲ情報表示部は、範囲特定情報履歴特定部が特定した前記範囲特定情報の履歴に基づき、前記ＡＲ情報受信部が受信した前記所定位置特定情報を世界座標系で表された情報に変換する、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 9, wherein
    The predetermined position specification information includes information indicating the predetermined position in a coordinate system of a camera image based on the image data.
    The AR information display unit converts the predetermined position identification information received by the AR information reception unit into information represented in the world coordinate system, based on the history of the range identification information identified by the range identification information history identification unit. ,
    AR information provision system.
  11. 　請求項１又は９に記載のＡＲ情報提供システムであって、
    　前記ＡＲ情報表示部は、特定した前記表示位置が前記現在の範囲特定情報で特定される範囲内であれば、前記受信したＡＲ情報を前記表示位置に表示する、
    　ＡＲ情報提供システム。     The AR information providing system according to claim 1 or 9,
    The AR information display unit displays the received AR information on the display position if the specified display position is within the range specified by the current range specification information.
    AR information provision system.
  12. 　請求項１１に記載のＡＲ情報提供システムであって、
    　前記ＡＲ情報表示部は、特定した前記表示位置が前記現在の範囲特定情報で特定される範囲内でなければ、前記受信したＡＲ情報が存在する方向を示すＡＲ情報を前記範囲内に表示する、
    　ＡＲ情報提供システム。
    The AR information providing system according to claim 11, wherein
    If the specified display position is not within the range specified by the current range specification information, the AR information display unit displays AR information indicating the direction in which the received AR information is present within the range.
    AR information provision system.
  13. 　請求項１に記載のＡＲ情報提供システムにおける前記第１情報処理装置であって、
    　現実の所定範囲の風景の所定位置にＡＲ情報を重ねて表示するＡＲ情報表示部と、
    　前記所定範囲を特定する情報である範囲特定情報を取得する範囲特定情報取得部と、
    　前記範囲特定情報を前記第２情報処理装置に送信する範囲特定情報送信部と、
    　前記ＡＲ情報とともに前記所定位置特定情報を受信するＡＲ情報受信部と、
    　を備え、
    　前記範囲特定情報取得部は、現在の前記範囲特定情報を取得し、
    　前記ＡＲ情報表示部は、受信した前記所定位置特定情報と取得した前記現在の範囲特定情報とに基づき、受信した前記ＡＲ情報の表示位置を特定し、特定した前記表示位置に基づき前記ＡＲ情報の表示を制御する、
    　情報処理装置。     The first information processing apparatus in the AR information provision system according to claim 1, wherein
    An AR information display unit that displays AR information superimposed on a predetermined position of a real predetermined range of scenery;
    A range identification information acquisition unit that acquires range identification information that is information for identifying the predetermined range;
    A range identification information transmission unit that transmits the range identification information to the second information processing apparatus;
    An AR information receiving unit that receives the predetermined position specifying information together with the AR information;
    Equipped with
    The range identification information acquisition unit acquires the current range identification information;
    The AR information display unit specifies a display position of the received AR information based on the received predetermined position specification information and the acquired current range specification information, and the AR information display unit determines the AR information based on the specified display position. Control the display,
    Information processing device.
  14. 　請求項１に記載のＡＲ情報提供システムにおける前記第２情報処理装置であって、
    　前記範囲特定情報を受信する範囲特定情報受信部と、
      前記範囲特定情報に基づき特定される前記所定範囲の前記所定位置に対応づけて前記ＡＲ情報を設定するＡＲ情報設定部と、
    　設定した前記ＡＲ情報を、前記所定位置を特定する情報である所定位置特定情報とともに前記第１情報処理装置に送信するＡＲ情報送信部と、
    　を備える、
    　情報処理装置。     The second information processing apparatus in the AR information provision system according to claim 1, wherein
    A range identification information receiving unit that receives the range identification information;
    An AR information setting unit configured to set the AR information in association with the predetermined position of the predetermined range specified based on the range specification information;
    An AR information transmission unit that transmits the set AR information to the first information processing apparatus together with predetermined position specifying information that is information for specifying the predetermined position;
    Equipped with
    Information processing device.
  15. 　請求項９に記載のＡＲ情報提供システムにおける前記第１情報処理装置であって、
    　現実の所定範囲の風景の所定位置にＡＲ情報を重ねて表示するＡＲ情報表示部と、
    　前記所定範囲を特定する情報である範囲特定情報を取得する範囲特定情報取得部と、
    　前記範囲特定情報をタイムスタンプに対応づけて蓄積記憶する範囲特定情報履歴記憶部と、
    　前記所定範囲の画像データを取得する画像データ取得部と、
    　取得した前記画像データと前記画像データのタイムスタンプとを前記第２情報処理装置に送信する送信部と、
    、
    　前記ＡＲ情報とともに前記所定位置特定情報及び前記タイムスタンプを受信するＡＲ情報受信部と、
    　蓄積記憶している前記範囲特定情報から、受信した前記タイムスタンプに対応する前記範囲特定情報を特定する範囲特定情報履歴特定部と、
    　を有し、
    　前記範囲特定情報取得部は、現在の前記範囲特定情報を取得し、
    　前記ＡＲ情報表示部は、前記特定した範囲特定情報の履歴、受信した前記所定位置特定情報、及び取得した前記現在の範囲特定情報に基づき、受信した前記ＡＲ情報の表示位置を特定し、特定した前記表示位置に基づき前記ＡＲ情報の表示を制御する、
    　情報処理装置。     The first information processing apparatus in the AR information provision system according to claim 9, wherein
    An AR information display unit that displays AR information superimposed on a predetermined position of a real predetermined range of scenery;
    A range identification information acquisition unit that acquires range identification information that is information for identifying the predetermined range;
    A range identification information history storage unit for storing the range identification information in association with a time stamp;
    An image data acquisition unit for acquiring image data of the predetermined range;
    A transmitter configured to transmit the acquired image data and a time stamp of the image data to the second information processing apparatus;
    ,
    An AR information receiving unit that receives the predetermined position specifying information and the time stamp together with the AR information;
    A range specifying information history specifying unit for specifying the range specifying information corresponding to the received time stamp from the range specifying information stored and stored;
    Have
    The range identification information acquisition unit acquires the current range identification information;
    The AR information display unit specifies and specifies the display position of the received AR information based on the history of the specified range specification information, the received predetermined position specification information, and the acquired current range specification information. Controlling display of the AR information based on the display position;
    Information processing device.
  16. 　請求項９に記載のＡＲ情報提供システムにおける前記第２情報処理装置であって、
    　前記画像データを前記タイムスタンプとともに受信する画像データ受信部と、
    　受信した前記画像データに基づく画像を表示しつつ前記ＡＲ情報の設定に関する操作入力を受け付けるユーザインタフェースと、
    　設定した前記ＡＲ情報を、前記所定位置を特定する情報である所定位置特定情報と前記タイムスタンプとともに前記第１情報処理装置に送信するＡＲ情報送信部と、
    　を有する、
    　情報処理装置。     The second information processing apparatus in the AR information provision system according to claim 9, wherein
    An image data receiving unit that receives the image data together with the time stamp;
    A user interface that receives an operation input related to setting of the AR information while displaying an image based on the received image data;
    An AR information transmitting unit that transmits the set AR information to the first information processing apparatus together with predetermined position specifying information that is information for specifying the predetermined position and the time stamp;
    Have
    Information processing device.
PCT/JP2018/043209 2017-12-18 2018-11-22 Ar information providing system and information processing device WO2019123971A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-241437 2017-12-18
JP2017241437A JP6849582B2 (en) 2017-12-18 2017-12-18 AR information provision system and information processing equipment

Publications (1)

Publication Number Publication Date
WO2019123971A1 true WO2019123971A1 (en) 2019-06-27

Family

ID=66993258

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/043209 WO2019123971A1 (en) 2017-12-18 2018-11-22 Ar information providing system and information processing device

Country Status (2)

Country Link
JP (1) JP6849582B2 (en)
WO (1) WO2019123971A1 (en)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005135355A (en) * 2003-03-28 2005-05-26 Olympus Corp Data authoring processing apparatus
JP2016012168A (en) * 2014-06-27 2016-01-21 株式会社デジタル Information sharing system
JP2016081225A (en) * 2014-10-15 2016-05-16 Kddi株式会社 Information presenting system
JP2016167688A (en) * 2015-03-09 2016-09-15 富士通株式会社 Image generation method, system, device, and terminal
JP2017037434A (en) * 2015-08-07 2017-02-16 シャープ株式会社 Mark processing unit and program

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005135355A (en) * 2003-03-28 2005-05-26 Olympus Corp Data authoring processing apparatus
JP2016012168A (en) * 2014-06-27 2016-01-21 株式会社デジタル Information sharing system
JP2016081225A (en) * 2014-10-15 2016-05-16 Kddi株式会社 Information presenting system
JP2016167688A (en) * 2015-03-09 2016-09-15 富士通株式会社 Image generation method, system, device, and terminal
JP2017037434A (en) * 2015-08-07 2017-02-16 シャープ株式会社 Mark processing unit and program

Also Published As

Publication number Publication date
JP6849582B2 (en) 2021-03-24
JP2019109319A (en) 2019-07-04

Similar Documents

Publication Publication Date Title
US10579320B2 (en) Display system, display device, information display method, and program
EP3619599B1 (en) Virtual content displayed with shared anchor
EP2613296B1 (en) Mixed reality display system, image providing server, display apparatus, and display program
US10474228B2 (en) Display system for remote control of working machine
US9275079B2 (en) Method and apparatus for semantic association of images with augmentation data
EP3340177B1 (en) Method and system for creating virtual message onto a moving object and searching the same
JP5582548B2 (en) Display method of virtual information in real environment image
US8115768B2 (en) Methods and system for communication and displaying points-of-interest
JP7047394B2 (en) Head-mounted display device, display system, and control method for head-mounted display device
US10868977B2 (en) Information processing apparatus, information processing method, and program capable of adaptively displaying a video corresponding to sensed three-dimensional information
JPWO2018088037A1 (en) Control device for movable imaging device, control method for movable imaging device, and program
JP7109395B2 (en) WORK SUPPORT SYSTEM, WORK SUPPORT DEVICE, AND WORK SUPPORT METHOD
JP2006197068A (en) Image display and display method
CN113467731B (en) Display system, information processing apparatus, and display control method of display system
JP2021018710A (en) Site cooperation system and management device
WO2019123971A1 (en) Ar information providing system and information processing device
US11238658B2 (en) AR space image projecting system, AR space image projecting method, and user terminal
KR20220165948A (en) Method and system for remote collaboration
KR102143616B1 (en) System for providing performance contents using augmented reality and Method for providing the same
KR20160022482A (en) Military traning monitoring system
KR102098225B1 (en) Omnidirectional image display apparatus and method
TWI611307B (en) Method for establishing location-based space object, method for displaying space object, and application system thereof
JP6348750B2 (en) Electronic device, display method, program, and communication system
JP7179583B2 (en) Image processing device and image processing method
CN115223384B (en) Vehicle data display method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18891791

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18891791

Country of ref document: EP

Kind code of ref document: A1