JP2020098568A - Information management device, information management system, information management method, and information management program - Google Patents

Abstract

To provide an information management device, an information management system, an information management method, and an information management program with which it is possible to correlate an actual existing object with information relating thereto, associate each of discretionary objects with information in a discretionary data format, and record (preserve) and manage the positional relationship of the objects or a positional relationship, etc., when the objects are overlooked from a wide range.SOLUTION: Provided is a portable information management device 10 comprising imaging means 15 capable of obtaining a live-view video of a target area that includes an actually existing object, display means 13 capable of displaying the live-view video, storage means 14, and information management means 20. The information management means 20 includes position specification means 23 for specifying the position of an object actually existing in a prescribed space, mark addition means 21 for displaying a mark image that corresponds to the object on top of the live-view video on the display means 13, and association means 25 for storing information related with the actually existing object in the storage means 14 in association with the mark image.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information management device, an information management system, an information management method, and an information management program.

Conventionally, an information management device (for example, a personal computer (PC)) is associated with a real space (place) or an object by associating digital information (for example, position information, image data, character data, or the like) related to them. ) Is known to process and save data that can be managed.

For example, especially at construction sites such as construction (construction) and civil engineering, take a picture of the situation at each stage of construction and submit a photo ledger with the photograph to the owner such as a public office to confirm the work status of the construction. Is required to be reported. In this case, in order to add information such as construction name, work type, measuring point, date, name of the company performing the construction, work status, etc. in the photograph, write this information on the actual blackboard and set it up on site. The method of taking a picture is used from the old days.

In order to improve the convenience of such a method, there is known a digital camera capable of taking a picture of a construction site where images of blackboards are combined without using a real blackboard (for example, refer to Patent Document 1). ..

There is also known a program for sequentially taking photographs of construction sites and automatically arranging the image files of the photographs to reduce the burden of creating a photo ledger and a process chart (for example, see Patent Document 2).

Furthermore, in a plant facility where the position information using the global radio positioning system (GPS) cannot be used, as a sorting operation of the captured images, the captured image (inspection image) and the location (inspection target device There is also known a technique of efficiently associating a position) with an object to be imaged (device to be inspected) (for example, Patent Document 3).

Japanese Patent No. 3288019 Patent No. 5835780 JP, 2016-206825, A

However, the techniques described in Patent Documents 1 and 2 provide information related to a specific position of a shooting target (specific position on site) (information such as the situation on the construction date (shooting date) and construction content). Since they are associated with each other, it is difficult to grasp the positional relationship between the photographing positions (fields) of each other (the photographing positions overlooking from a wide range).

Further, the technique described in Patent Document 3 is limited to photographing at a predetermined inspection location, and photographing/recording at an arbitrary location is not assumed.

In addition, when sharing information on the building/construction work site and information on the site of repairs/repairs and inspections with multiple people, including those in remote areas other than the site, what is the work or work? It is highly necessary to leave surrounding environment information that is not directly related, and for example, character (document) data other than image data is also required, which cannot be sorted out by conventional techniques.

In particular, it is routine that the person sharing the information includes a person who is new to the scene (has never visited the scene), and trying to convey to those people accurately and easily, The amount of surrounding environment information will increase and become more complex.

For example, specifically, a case will be described where a plurality of people share information such as the degree and site of the scratch in order to repair the scratch on the indoor wall. In order to record the condition of the target scratch, the photographer who has visited the scene first takes a close-up photograph (a photograph close to the subject) of the scratched part. However, since it is a close-up photograph, it is difficult for anyone other than the photographer to understand the shooting location, and the photographer himself may not know the shooting location.

Further, in order to accurately or easily convey or grasp the situation of the site including the target (part of the wound), detailed information about the target (for example, a close-up photograph of the target) and overhead information (for example, , Both the information of the surrounding environment of the target) is required, and there is a problem that the work of associating (associating) these with each other is troublesome.

As information on the surrounding environment, information (documents) in multiple types of data formats is created, such as information that is difficult to convey only with photographs (image data), created with documents (character data), or written in drawings. However, there is a problem in that the work of creating such information, associating the information with each other, and dividing into folders for organizing becomes complicated and complicated.

In addition, it may be necessary to record the progress information of the repair of the scratch, or to record unexpectedly unplanned objects, for example, when a new site of the wound is found on the spot. There is also a high demand for flexible recording and management in this data format.

Further, it is common to combine these pieces of information to create a report or the like, but the creation of the report or the like has a problem that it takes a lot of time and labor.

Note that these points are limited to information management at construction/construction sites where blackboards have been used up to now, or at inspection sites that patrol a predetermined site, or to a large or small scale or to be managed. However, it is also a problem in the information management at the office work site such as a company or a store.

In view of such an actual situation, the present invention associates information of an arbitrary data format with each of arbitrary objects in the case of managing an existing object and information related thereto by an information management device (PC or the like), Further, an information management device, an information management system, and an information management method capable of efficiently (easily) recording (saving) and managing the positional relationship between the objects, or the positional relationship when the objects are viewed from a wide range. And an information management program.

The present invention provides portable information including an image pickup unit capable of acquiring a live view image of a target area including an existing target, a display unit capable of displaying the live view image, a storage unit, and an information management unit. In the management device, the information management unit includes a position specifying unit that specifies a position of the target, and a mark giving unit that displays a mark image corresponding to the target on the live view image on the display unit. And an associating unit that stores information related to the target in the storage unit in association with the mark image.

Further, the present invention is an information management system comprising the above information management device and another device capable of managing the information.

Further, the present invention is an information management method of adding predetermined information to an existing object and storing the data as data that can be managed by the information management device, wherein a live view image of a target area including the existing object is captured by the imaging means. An image capturing step of acquiring, a display step of displaying the live view image on a display unit, a position identifying step of identifying a position of the target in a predetermined space, and a mark image corresponding to the target overlaid on the live view image. An information management method comprising: a mark giving step of displaying on the display means; and an associating step of storing information related to the target in a storage means in association with the mark image.

Further, the present invention provides an image capturing step of acquiring a live view image of a target area including an existing target by an image capturing means, a display step of displaying the live view image on a display means, and a position of the target in a predetermined space. Position identifying step, a mark providing step of displaying a mark image corresponding to the target on the display means in an overlapping manner with the live view image, and information relating to the target in association with the mark image, the storage means. An information management program characterized by causing a computer to execute the associating step stored in.

According to the information management device, the information management system, the information management method, and the information management program of the present invention, when an information management device (such as a PC) manages a real target and information related thereto by an arbitrary target It is possible to associate (retain) information in any data format with respect to each of the above, and to efficiently and easily record (save) and manage the positional relationship between the objects, or the positional relationship when the objects are overlooked from a wide range. The excellent effect of

It is an external view showing the outline of the information management device concerning the embodiment of the present invention. It is a block diagram which shows the outline|summary of the information management apparatus based on embodiment of this invention. It is a figure showing an example of a display of an information management device concerning an embodiment of the present invention. It is a figure showing an example of a display of an information management device concerning an embodiment of the present invention. It is a figure showing an example of a display of an information management device concerning an embodiment of the present invention. It is a figure which shows an example of the information which the information management apparatus which concerns on embodiment of this invention manages. It is a block diagram which shows the outline|summary of the (A) general-view means based on embodiment of this invention, and is a figure which shows an example of a display of (B) information management apparatus. It is a figure showing an example of a display of an information management device concerning an embodiment of the present invention. It is a figure showing an outline of an information management method (information management processing) concerning an embodiment of the present invention. It is a schematic diagram showing an example of an information management system concerning an embodiment of the present invention. It is a block diagram which shows the outline|summary of the information management apparatus based on embodiment of this invention. It is a schematic diagram of the cooperation means based on embodiment of this invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention. It is a schematic diagram showing an example of cooperation processing concerning an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

<Information management device/Overall configuration>
1 and 2 are diagrams showing an overall configuration of an information management apparatus 10 which is an example of an embodiment of the present invention, FIG. 1 is a schematic view of the appearance, and FIG. It is a top view, and the same figure (B) is a top view of the other surface side. FIG. 2 is a block diagram showing the outline of the hardware configuration.

As shown in FIGS. 1 and 2, the information management device 10 according to the present exemplary embodiment includes a control unit 11, an input unit 12, a display unit 13, a storage unit 14, an imaging unit 15, and an information communication unit 16. The status acquisition unit 17 and the information management unit 20 are included.

As shown in FIG. 1, the information management device 10 of the present embodiment is a portable device (mobile terminal device). In the following embodiments, the case where the information management device 10 is the tablet terminal 101 in which the program (application program) according to the present embodiment is installed is described as an example, but the information management device 10 is not limited to this, and the information management device 10 is not limited to this. For example, a smartphone, a mobile phone, a notebook PC, a personal digital assistant (PDA), a handy terminal, a GPS terminal, a digital camera, an HMD (Head) in which the program according to the present embodiment is installed or incorporated. Mounted Display) or the like.

The tablet terminal 101 is a portable information processing terminal device that uses wireless communication, and includes an input unit 12, a display unit 13, an imaging unit 15, a microphone 18, a speaker 19, and the like.

In this example, the display unit 13 is a touch screen (touch panel display) of the tablet terminal 101, and for example, a liquid crystal display (Liquid Crystal Display), an organic EL display (Organic Electro-Luminescence Display.), or an OLED display (Organic Light Emitting Diode). (Also referred to as Display).) and the like, a touch panel that is the input means 12 is superimposed.

The display unit 13 can display the image captured by the image capturing unit 15 and the live view image LV to be captured. Further, it is possible to display an image as an execution result of various applications, an icon as the input unit 12, an image of a button, and other images (including characters). The “image” in this specification includes a still image and a moving image, unless otherwise specified.

As the operation of the input unit 12, by touching (displaying, tapping, dragging, pinching out, pinching in, swiping, etc.) the displayed image with a finger or a touch screen pen, selection, determination, enlargement, reduction, Various operations such as movement can be performed. The input means 12 also includes a physical button 121. The physical button 121 is, for example, a home button or a volume control button.

Note that, as in the case where the information management apparatus 10 is, for example, a mobile phone or a notebook PC, the display unit (display) 15 and the input unit (for example, keyboard, physical key) 12 may have independent configurations. Good.

The imaging unit 15 is, for example, a unit (camera unit) that realizes a camera function in cooperation with an imaging application program, and the lens 151 is arranged on a surface different from the surface on which the display 13 is provided ( The same figure (B)).

Further, the tablet terminal 101 has a known configuration such as the microphone 18, the speaker 19, and other sensors (not shown).

With reference to FIG. 2, the control unit 11 includes, for example, an arithmetic processing unit (CPU; central processing unit, MPU; microprocessor unit, etc.), and is an operating system stored (installed) in the storage unit 14 by the control unit 11 or the like. An apparatus that provides various functions including the functions described below by executing (OS; Operating System), various application programs, and browser software.

The storage unit 14 is a built-in memory (main memory (RAM; Random Access Memory), storage, ROM (Read Only Memory), etc.) in this example, but may include an external storage device. The storage unit 14 stores and saves various data (data that can be processed by various application programs such as image data and text data, documents) in addition to the OS, firmware, browser software, various application programs, and various drivers.

The imaging unit 15 is, for example, a unit (camera unit) that realizes a camera function in cooperation with an application program for imaging, and includes at least a lens 151, an imaging element (image sensor) 153, an image processor 155, and the like. The object recognized through the lens 151 is imaged on the image sensor 153, and image (video) information obtained by photoelectric conversion of the image sensor 153 is stored in the storage unit 14 as image data.

Specifically, by activating the camera function of the tablet terminal 101, the image capturing unit 15 can capture an image of a target area including an existing target, acquire it as a live view image LV, and display it on the display unit 13. When the operator operates the shooting button (for example, the button image displayed on the display unit 13 or the physical button 131), the image information displayed in the live view image LV is stored in the storage unit 14 as image data. The stored image data may be displayed on the display unit 13.

Here, the live view image LV in the present embodiment means that the object OB recognized by the imaging unit 15 through the lens 151 is imaged on the image sensor 153 in real time, and image information obtained by photoelectric conversion of the image sensor 153 is non-volatile. A video (image) to be displayed on the display unit 13 without going through the sex storage unit 14. Specifically, the live view image LV in the present embodiment means that a real image of the target area including the target OB is displayed in real time on the display unit 13 via the imaging unit 15 in a state where the shooting button or the like is not operated. Refers to the video that is present.

The information communication unit 16 is an interface for the tablet terminal 101 to perform data communication with other devices (such as other tablet terminals and server devices) via a network (communication line). The data communication is, for example, short-distance wireless communication such as Wi-Fi (registered trademark), Bluetooth (registered trademark), NFC (Near Field Communication), or the like, for example, LTE (Long Term Evolution) method, 3G (3rd Generation) method, 4G (4th Generation) system, 5G (5th Generation) system, FDMA system, TDMA system, CDMA system, W-CDMA, and various wireless communication systems via mobile communication networks such as PHS (Personal Handyphone System). Done.

The state acquisition unit 17 is a unit that acquires and records information indicating the state of the own device (the direction in which the lens 151 is facing), the posture, the position, and the like, and although detailed illustration is omitted, for example, 3 An electronic compass that detects the geomagnetism in the axial direction to detect the direction, an acceleration sensor that detects the tilt angle of the tablet terminal 101 with respect to the three axial directions, a gyro (angular velocity) sensor, a depth sensor, and a SLAM (Simultaneous) using a monocular camera. Localization and Mapping). In the tablet terminal 101, based on the azimuth detected by the electronic compass and the tilt angle/posture of the own device detected by the triaxial acceleration sensor or the gyro sensor, the imaging direction in which the lens 151 of the imaging unit 15 is directed ( The imaging direction) and the imaging angle (angle of view) can be detected. In SLAM, the own position can be estimated and a map of the surrounding environment can be created by moving the own device from a reference position (initial position) and acquiring a plurality of images of the surrounding environment by the imaging unit 15. ..

The state acquisition unit 17 may be a SLAM using a stereo camera, or a unit that detects reflected light from an object such as infrared rays or a laser to detect the posture and position of the own device. In addition, the state acquisition unit 17 detects the direction, posture, and position (absolute position) of the own device by, for example, the radio base station of a mobile phone or the radio field intensity from an access point of Wi-Fi (registered trademark) communication. In addition, an absolute position information acquisition module that can detect the direction, attitude, and position (absolute position) of the own device by a signal from a satellite is included, such as a global positioning satellite system.

The information management means 20 is an application program (information management program 20 of this embodiment) installed in the storage means 14 of the tablet terminal 101. That is, the information management program 20 is installed in the tablet terminal 101, and the information management function of the information management apparatus 10 is realized by cooperating with the hardware of the tablet terminal 101 described above.

Continuing to refer to FIG. 2, the information management means 20 and the information management function realized thereby will be described.

The information management means 20, for example, a virtual visual processing (virtual audiovisual processing, the same applies below) means 27 that is a means for realizing virtual visual (and auditory) processing, a mark giving means 21, a position specifying means 23, It has an associating means 25 and an overall view display means 29. The virtual visual processing means 27 is, for example, an augmented reality (AR) processing means that realizes augmented reality (AR), and is specifically a so-called AR framework or AR library.

That is, in the present embodiment, the virtual visual processing means (AR processing means) 27 is utilized to make a mark giving function (mark giving means) 21, a position specifying function (position specifying means) 23, and an associating function (associating means) 25. An information management program (information management means) 20 for realizing the overall display function (overall display means 29) is configured.

In the present embodiment, the case where the virtual visual processing means 27 is an AR processing means will be described below as an example. However, the virtual audiovisual processing means 27, for example, realizes virtual reality (VR). It may be a processing means (VR platform) or a mixed reality (MR) processing means (MR platform) that realizes mixed reality (MR).

Since the AR processing means (AR framework (AR library, the same applies hereinafter)) 27 has a known configuration, a detailed description thereof will be omitted. For example, the target area (existing place, real environment) using the imaging means 15 is used. By capturing, the information of the space of the target area (such as the position information of the target area and the information of the object included in the target area) is stored, recognized, estimated, analyzed, and the like, and the target area being captured ( This is a technology that visually adds the information to, removes, emphasizes, and attenuates it from the image of the (real environment) to make it appear as if the real environment was expanded.

Examples of the AR framework 27 include ARKit (registered trademark) provided by Apple Inc. in the US, ARCore (registered trademark) provided by Google Inc. in the US, and open source ARTToolKit. As the AR framework 27 in this embodiment. A case of using ARKit (registered trademark) will be described as an example. In this case, the AR framework 27 uses the image pickup means 15 to store, estimate, analyze, etc. spatial information about the target area (existing place, real environment), and the image of the target area (live view video LV). By storing the arrangement of information of "characteristic points" among them, the shape (outside) from the "characteristic points" of an object (for example, a real wall, chair, desk, etc.) in a real environment (for example, indoors) is stored. The shape can be grasped, and an object image based on digital information (computer graphics; CG) or the like can be synthesized along the shape and displayed in real time on the display means 13. Alternatively, a surface including the object ( It is possible to detect a flat surface (horizontal surface, vertical surface, etc.) or a curved surface, synthesize object images along the surface, and display them on the display unit 13 in real time.

It should be noted that digital information (for example, an image such as a grid) indicating the surface including the object (horizontal surface, vertical surface, curved surface, etc.) detected and detected in the image of the object displayed in the live view image LV of the real environment. May be combined (superposed) and displayed on the display means 13 in real time.

Further, the AR framework 27 recognizes the shape of the object and compares it with the information of the object stored in the storage means, for example, to attribute the object (not only the outer shape such as a rectangle or a circle, but also the “wall” and the “chair”). “Attributes such as “desk”) may be recognized, and digital information (such as an object image related to the object) is combined (superposed) with the image of the object displayed in the live view image LV of the real environment. The display means 13 may be configured to display in real time.

Further, the arrangement and the situation of the combined (superposed) object images may be recorded and saved. In this case, when the target area (real environment) is visited at a later timing (a few days, a few weeks, etc.) and the application program using the AR framework 27 is started, the previously placed object image is placed at the same position. Can be viewed and edited again. Also, an association operation (described later) can be performed.

The AR processing means 27 of the present embodiment is not limited to ARKit (registered trademark) and may be not a specific framework (library) as long as it can realize the same function as described above (individually). It may be a program or the like created to have the above function).

In the present embodiment, the position specifying unit 23, the mark assigning unit 21, the associating unit 25, and the overall view displaying unit 29 cooperate with the AR processing unit 27 to perform the position identifying process, the mark assigning process, the associating process, the overall view displaying process, etc. I do.

The position specifying means 23 specifies the reference position (initial position) and the mark giving position by the mark giving means 21 described later by the image pick-up means 15 (activating the camera function) and stores it in the storage means 14.

The information management device 10 of the present embodiment grasps the state of itself (direction in which the lens 151 is facing, posture, position) at any time. Therefore, when the information management program 20 is executed (started), the image pickup unit 15 is started to display the live view image LV, and the position specifying unit 23 resets the reference position (initial position) (new reference position (new position). Obtain the initial position). To acquire the initial position, for example, the image capturing unit 15 captures a plurality of images of the real environment (peripheral space) around the information management device 10 from different viewpoints (states in which live view images LV with different angles of view are displayed). This is performed by grasping (estimating) the position of the own device in the real environment based on the plurality of captured images.

Alternatively, the initial position may be acquired by using information (environmental data) on the space of the real environment that is stored in the storage unit 14 in advance. Specifically, the imaging means 15 scans the space of the real environment in advance and stores the information (environment data) in the storage means 14. Then, when the information management program 20 is activated, the environment data is loaded from the storage unit 14, and the image of the real environment captured by the image capturing unit 15 is collated with the environment data. When the two match, the relative position from the origin of the environmental data may be set as the initial position as the current position of the own device.

In any of the above cases, the position of the own device may be grasped by detecting the reflected light of infrared rays or laser.

Furthermore, after the initial position is acquired, the position specifying unit 23 uses the state acquiring unit 17 (or the position information acquiring module as necessary) at any time (sequentially) until the initial resetting of the initial position is performed next time. Acquires information (change in horizontal distance, vertical distance, angle, direction, posture, etc.) from the position (relative position information from the initial position; hereinafter referred to as "terminal movement information") ( (Specified) and stored in the storage means 14.

Particularly in the case of a building (indoor), the state of the own device may not be accurately acquired by GPS or the like, but in the present embodiment, the initial position is set and the terminal movement information from the initial position is acquired. There is no problem even if there is an error from the position information (absolute position information).

However, the present invention is not limited to this, and an absolute position acquisition module such as GPS may be used to acquire the absolute position of the own device.

The position specifying unit 23 also specifies the position (relative position from the initial position) of the target OB in the predetermined space based on a mark giving operation described later, and stores it in the storage unit 14.

As described above, the position specifying means 23 acquires (specifies) the terminal movement information from the initial position at any time using the state acquisition means 17.

Further, the position specifying unit 23 (state acquisition unit 17) specifies a certain point of the live view image LV being displayed on the display unit 13 (for example, the position of the target OB on the live view image LV), It is configured to be able to detect (estimate) the distance of the point from its own machine in the real environment. Specifically, for example, the position specifying unit 23 (state acquisition unit 17) is configured to be able to set the distance (adjustment value) between the point specified by the user on the live view image LV and the own device. The adjustment value is set by the user in advance or appropriately. When the user specifies (tap, touch, etc.) a certain point in the live view image LV, the position specifying unit 23 sets the set adjustment value as the distance from the own device to the point and determines the position of the own device. The position of the point in the real environment is specified based on the position. For example, when the adjustment value is set to 30 cm, when the user specifies a certain point in the live view image LV (tap, touch, etc.), the position specifying unit 23 moves the point 30 cm ahead of the own device. Then, the position of the point in the real environment is specified based on the position of the own device. A fixed value (for example, 30 cm, 1 m, etc.) may be set as the adjustment value, or the adjustment value may be appropriately changed by the user.

Further, the position specifying means 23 can detect (estimate) the distance from the own device by detecting a predetermined shape. Specifically, the AR framework 27 can detect a shape, for example, a flat surface (a flat surface) by recognizing a “characteristic point” in the live view image LV. That is, for example, a floor or wall in the real environment can be regarded as a “characteristic point”, and a plane including the floor or wall can be detected. Based on the detection of the plane, the position specifying means 23 uses a guide display G (for example, a grid or a dot grid to show the state of the plane or a state in which the detected plane is virtually extended further in the distance (for example, the depth direction)). A label image indicating the size, width, inclination, depth, etc. of the detected plane can be displayed in a superimposed manner on the live view image LV. Then, the position specifying means 23 can recognize (estimate) the distance from the own device with respect to a certain position on the guide display. The shape detection is not limited to flat surface detection, and may be a shape having irregularities.

In other words, the position identifying means 23 performs the mark attaching operation corresponding to the target OB while capturing the target OB as the live view image LV, so that the position of the real environment corresponding to the mark attaching position on the live view image LV. The distance between (the position of the target OB in the real environment) and the own device is specified (estimated). Then, based on the specified distance and the terminal movement information from the initial position in the mark applying operation position, the mark applying position, that is, the relative position from the initial position of the target OB in the real environment can be specified.

The mark giving unit 21 superimposes on the live view image LV displayed on the display unit 13 based on the mark giving operation by the user, and the object image corresponding to the target OB to which various kinds of information is given (hereinafter referred to as “mark image M”). Is displayed).

This will be specifically described with reference to FIG. This figure is a diagram showing an example of a display on the display means 13 of the information management device 10 at the time of the mark giving operation.

The user is configured to be able to select whether to use an adjustment value or shape detection (in this example, plane detection) at the time of a mark giving operation. When the user selects to use the adjustment value at the time of the mark giving operation, the position specifying means 23 specifies the position by the method using the adjustment value described above. On the other hand, when the user selects to use the plane detection during the marking operation, the position specifying unit 23 specifies the position by the method using the above-described plane detection.

FIG. 3A is a display example when the adjustment value is used to specify the mark application position. In this state, when the adjustment value is set in advance to, for example, 30 cm, after the initial position is set, the user selects to use the adjustment value at the time of the mark giving operation, and is a display example in the state of being close to the target OB. is there. The target area (real environment) including the actual target OB is photographed by the imaging means 15, and real-time video information of the target area is displayed on the display means 13 as the live view video LV.

As shown in FIG. 7A, the mark giving means 21 movably displays the mark image M overlaid on the live view image LV.

In this state, the user performs a mark giving operation of selecting the target OB to be given the mark image M, which is being displayed on the display means 13. That is, the user selects the mark-added position by, for example, sliding the mark image M to a desired position (for example, a position overlapping the target OB on the live view image LV or a position in the vicinity) while touching the mark image M ( decide.

In addition, in the mark giving operation, for example, after touching (tapping) the mark giving operation button 122 for instructing the start of mark giving, a desired position of the live view image LV being displayed (for example, the position of the target OB, or The surface of the display unit 13 may be touched (for example, tapped) at a position (around the target). The mark addition operation button 122 may be the button image displayed on the display unit 13 or the physical button 121. Thereby, the mark giving unit 21 detects the coordinates of the touch position on the display unit 13 and displays the mark image M at the touch position.

When the adjustment value is used, the mark image M displayed by the mark giving unit 21 indicates that the mark image M is located ahead of the own device by a predetermined adjustment value (30 cm in this example). That is, the user appropriately moves the mark image M so as to be superimposed on (or near) the target OB, and secures the distance between the target OB and the own device according to the adjustment value. That is, when the adjustment value is set to 30 cm, the target OB is appropriately moved so that the distance between the target OB and its own machine is 30 cm.

Then, for example, the position of the mark image M is determined by the operation for determining the mark providing position by the user (mark providing position determining operation). The mark giving position determination operation is, for example, a (re)operation of the mark giving operation button 122 (or the mark image M) (tap, touch, double tap, long press, etc.). Further, the mark giving position determining operation can be said to be an operation of acquiring the position of the target OB corresponding to the mark image M (target position acquiring operation).

When the adjustment value is used as shown in FIG. 9A, the position specifying unit 23 determines that the determined position of the mark image M (for example, the position of the target OB) is adjusted by the adjustment value (in this example, by the adjustment value). 30 cm) ahead, the position in the real environment of the target OB corresponding to the mark image M is specified based on the position of the own device. In this way, when the adjustment value is used, even if the target OB is an arbitrary point in a space where an object having a predetermined shape does not exist, the mark image M is given to specify the position of the target OB. You can

FIGS. 8B and 8C are display examples when shape detection (for example, plane detection) is used to specify the mark application position. In this state, the user selects to use the plane detection at the time of the mark giving operation, and the target OB is placed on a shape that can be detected as a plane, such as a wall (the same figure (B)) and a floor (the same figure (C)). It is an example of a display when located. The target area (real environment) including the actual target OB is photographed by the imaging means 15, and real-time video information of the target area is displayed on the display means 13 as the live view video LV.

As shown in FIGS. 2B and 2C, the mark giving unit 21 displays a guide display G (for example, a grid-shaped label image showing the detected plane) overlaid on the live view image LV, for example. , The mark image M is fixedly displayed near its center. In this case, the user arbitrarily moves the attitude, orientation, or position of the information management apparatus (tablet terminal) 10 so that the fixedly displayed mark image M is superimposed on (or located in the vicinity of) the target OB.

Then, for example, the position of the mark image M is determined by the user's mark giving position determining operation (for example, the mark giving operation button 122 (or the mark image M) operation).

When the plane detection is used, an existing object (floor, wall, desk, etc.) having a predetermined shape is required, but the mark giving operation along the detected plane can be performed. For example, if the detected plane is an inclined surface, the mark giving operation can be performed along the inclined surface.

It should be noted that the mark giving operation using the adjustment value shown in FIG. 9A and the mark giving operation using plane detection shown in FIG. 9B may be switched manually by a button operation or the like. It may be configured to be automatically switched based on the presence or absence of "point" and the shape, and when it is possible to switch automatically, it may be possible to switch manually by button operation or the like.

When the mark giving position determining operation is performed by the user, the mark giving means 21 can change the display mode of the mark image M (for example, display it in a blinking manner) and associate the information by the associating means 25 described later. This suggests to the user (photographer).

After the information management program 20 is first installed after being installed in the tablet terminal 101, the information management program 20 is terminated until the information management program 20 ends, or after the information management program 20 is terminated last time, the information management program 20 is restarted. In the present specification, a period from when the information management program 20 is started to when the information management program 20 is terminated (including power interruption) is referred to as “the information management program 20 is being executed” in this specification. Then, during execution of the information management program 20, the mark giving unit 21 can give the mark image M to each of a plurality of different target OBs.

When the mark images M are successively given to the plurality of target OBs, for example, the mark giving operation button 122 is operated (touched (tapped)) to instruct the start of mark giving. As a result, the mark image M, which indicates to the user (photographer) that the information can be associated by the associating unit 25 described later, for example, by blinking, is displayed as a still image and a new mark is displayed. The image M is displayed.

The position specifying means 23 specifies the relative position of the target OB in the predetermined space when the mark giving position determining operation is performed.

Specifically, in the case of the mark giving position determination operation using the adjustment value shown in FIG. 3(A), the position specifying means 23 (state acquisition means 17) is based on the position of the mark image M, The relative distance between the aircraft and the target OB is specified as the adjustment value. Further, as already described, the relative terminal movement information from the initial position to the position where the mark giving position determination operation is performed is calculated based on the terminal movement information acquired from time to time, and the relative movement of the target OB with respect to the initial position is calculated. The relative position information (information for specifying the relative position of the target OB with respect to the initial position by the horizontal distance, vertical distance, angle, and direction, hereinafter referred to as “target position information”) is specified. Then, it is stored in the storage means 14.

Further, in the case of the mark giving position determining operation by plane detection shown in FIG. 7B, the position specifying means 23 determines the relative position between the own device and the target OB in the real environment based on the positions of the guide display G and the mark image M. The actual distance. Further, as in the case of FIG. 7A, the position of the target OB is specified based on the terminal movement information acquired at any time and the relative distance between the own device and the target OB, and the target position information is stored in the storage unit 14. To do.

Further, the position identifying means 23 assigns a plurality of mark images M indicating at which position the target to which the mark image M is attached is located in a space (for example, a room) having a certain size in the real environment. In this case, the positional relationship with other mark images M is grasped and stored in the storage means 14.

As a result, the target OB associated with the mark image M deviates from the imaging target of the imaging unit 15 (out of the angle of view of the lens 151) and is also framed out from the display area of the display unit 13 in the live view image LV. However, when the target OB is imaged again, the mark image M is displayed over the target OB in the live view image LV (on the target OB or in the vicinity of the target).

Further, the target position information can be used in the overall display (map creation) described later.

Further, when the mark giving position determining operation is performed, the mark giving means 21 acquires and analyzes the information of the “characteristic point” of the target OB (in the vicinity of the mark giving position) in the live view image LV to realize the reality. The existence of the target OB in the environment may be grasped. The “characteristic points” are, for example, a large number of points that are shown corresponding to the outer shape of the target OB. The mark giving unit 21 may store the information of the grasped target OB (for example, external shape information such as the size of the target OB), the target position information and the mark image M in the storage unit 14 in association with each other. ..

It should be noted that the user (photographer) operates the photographing button 123 (or the physical button 121) which activates the image pickup means (camera function) 15 in the mark giving operation, whereby image data is generated and stored in the storage means 14. Ordinary photographing (hereinafter referred to as “photographing”) may be performed. In this case, the photographed photograph does not include the mark image M, and only the real environment is photographed.

FIG. 4 is a schematic diagram showing a state in which the user U is using the information management device 10 (information management program 20) in a certain space S. The live view video of the space S is displayed on the display means 13 of the information management device 10. The state which is displaying LV is shown. Further, in this state, the mark image M has been added to each of the four target OBs in the space S by the above-described processing, and the user stands in a position overlooking the space S and shoots at a wide angle of view. It is in the state of being. In this case, four objects OB1 to OB4 are stored in the angle of view of the image pickup means 15 (lens 151), and the display means 13 of the information management device 10 (tablet terminal 101) corresponds to the objects OB1 to OB4. The mark images M1 to M4 are displayed (AR display) overlaid on the live view image.

The display of the mark images M1 to M4 when the live view video LV is displayed in the space S is reset when the information management program 20 being executed is terminated. However, for example, the "characteristic point" of the space may be grasped, and the relative position from the initial position of each target OB may be stored in the storage unit 14 in association with the corresponding mark images M1 to M4. Accordingly, when the information management program 20 is restarted (re-executed) in the same space S by revisiting the site (the relevant space), the mark images M1 to M4 are displayed as live view images as shown in FIG. Can be overlaid on the display (AR display). As a result, even the user who first visited the site can accurately grasp the position of the target OB.

When the user performs an association start operation, the associating unit 25 stores information related to the existing target OB (hereinafter referred to as “target related information”) in the storage unit 14 in association with the mark image M. It is a means. Here, the association start operation is an operation (touch, tap operation, etc.) of the mark image M displayed as a still image, or an operation of a separately provided association operation button 126, and when the association start operation is performed. For example, it is suggested to the user (photographer) that the mark image M displayed as a still image has its display mode changed (by blinking display, etc.), and information can be associated by the associating means 25. Is configured to. In the present embodiment, even when the mark providing position determining operation is performed, the display mode of the mark image M is changed (blinking display or the like), and it becomes possible to associate the information by the associating unit 25. It is configured to suggest to the user (photographer). That is, this mark giving position determination operation is also included in the association start operation.

The target-related information is, for example, a target that indicates information about the target OB itself that can be generated/acquired by the information management program 20 (for example, position information of the target OB, external shape information, corresponding mark image, etc.). It includes basic information and target incidental information that can be associated with the target OB arbitrarily by the user.

Specifically, the associating unit 25 is assigned to the target OB, information indicating the presence of the target in a predetermined space (real environment), that is, target position information indicating the position of the target OB specified by the position specifying unit 23. The mark image M (image data thereof) and the identification number (mark identification number MID) unique to the mark image M are the target basic information, and one information group (data unit) using the mark identification number MID as a key, It is registered in the storage means 14 (for example, the related information database 30 shown in FIGS. 2 and 6). The target basic information may include information indicating the outer shape of the target OB analyzed from “characteristic points”.

Further, the associating unit 25 additionally registers the target incidental information arbitrarily selected for each target OB (mark identification number MID) by the user in the information group (data unit) corresponding to the mark identification number MID. For example, when the association start operation is performed by the user, the display mode of the mark image M is changed (for example, blinking is displayed), and the target accompanying information can be accepted by the associating unit 25 (reception of the target accompanying information. Suggest that the user is in the period). Then, the associating unit 25 waits for the input of the target incidental information by displaying candidates of the target incidental information and additionally registers the target incidental information input (selected) by the user in the data unit.

Target incidental information is, for example, image data (still images, moving images) such as photographs and videos, character data (text data, document data created by text creation software, tabular data created by spreadsheet software, etc.) , Voice data, and other data. Further, the target accompanying information also includes various data generated by the information generating means 2251 (see FIG. 2). Specifically, the information generation unit 2251 recognizes a voice input via the voice input unit 18 of the information management device 10, and generates character data. In addition, analysis (Artificial Intelligence analysis; AI analysis) using artificial intelligence is performed on captured images (still images, moving images) such as photographs and videos, and the name of the shooting target (for example, “scratch”, “door”, etc.) Of the tag and the image data with the tag added to the image is generated. This information is also included in the target incidental information.

FIG. 5 is a diagram showing an example of a display in the case of associating the target incidental information. The associating unit 25 registers various types of data selected or input by the user during the reception period of the target incidental information (for example, blinking display of the mark image) in the related information database 30 as the target incidental information. Specifically, as shown in the figure, the associating means 25 includes a predetermined storage means 14 (a storage area in which various data not associated with the mark identification number MID is stored, the tablet terminal 101). Data (file) DD that can be associated (selected) is acquired from a library or the like) and displayed on the display unit 13 in a list format, for example.

When the user arbitrarily selects (for example, taps) one or more data from the displayed list, the associating unit 25 uses the mark identification number MID corresponding to the mark image M as a key to associate the selected data with related information. Register in the database 30.

The association process for a certain mark image M is terminated by, for example, an association end operation by the user (for example, the operation of the mark image M again or the operation of the association operation button 126 again). By selecting the mark image and performing the association start operation and the association end operation in the display state of the mark image M (such as FIG. 3), the association process can be repeatedly performed at any timing.

In this way, the various data displayed in the list by the associating unit 25 for the user to select are other application programs (camera application program, document creation application program, etc.) and other functions of the tablet terminal 101 (recording function, etc.). ), the data stored in the storage unit 14 at an arbitrary timing (a timing different from the timing when the information management program 20 is being executed).

However, the present invention is not limited to this, and data created in real time during execution of the information management program 20 (data generated by the information generating means 2251) can also be associated.

For example, when the target incidental information is image data, the camera function of the tablet terminal 101 is activated by a user's association start operation, and new image data (still image data) captured by the image capturing unit 15 while the information management program 20 is being executed. , Moving image data), that is, data that has not been registered in the related information database 30 may be additionally registered in the related information database 30 in association with the mark identification number MID. Further, the image data stored in the storage unit 14 (library) of the tablet terminal 101 can be selected, for example, by displaying a thumbnail, and the data selected by the user is associated with the mark identification number MID to associate the related information database. It may be additionally registered in 30. Also, both of them may be selectable.

When the target related information is voice data, the microphone (recording) function of the tablet terminal 101 may be activated by the user's association start operation, or the data (file) may be operated by the operation of the mark image M. May be configured so that the activation of the microphone function can be selected, and in this case, the voice information recorded during the execution of the information management program 20 is associated with the mark identification number MID and the related information database. 30 can be additionally registered.

Further, when the target related information is character data, the memo function of the tablet terminal 101 is activated by the user's association start operation (the memo input field 31 is displayed as shown in FIG. 5). Alternatively, it may be configured such that the start of the memo function can be selected together with the selection of the data (file) by the association start operation by the user, and in this case, it is input during the execution of the information management program 20. Character data (text data) can be additionally registered in the related information database 30 in real time in association with the mark identification number MID.

By the associating unit 25, it is possible to collectively associate and manage a plurality of types of target related information regarding the target OB with the mark image M. Further, even at a later timing, by performing the association start operation, the target related information can be added to the mark image M (target OB).

An example of the related information database 30 will be described with reference to FIG. FIG. 6(A) is a diagram showing a display example of the display means 13 that superimposes and displays the mark images M1 to M4 on the live view image LV as shown in FIG. 4, and for each of the targets OB1 to OB4. It also shows a state in which the target related information is also associated.

FIG. 3B is a diagram showing an example of the related information database 30. As described above, a mark image M having a unique identification number (mark identification number MID) is associated with a certain one target OB. Further, target related information (target basic information and, if necessary, target accompanying information) is associated with the certain target OB (one mark identification number MID), and can be managed as one information group (data package). Becomes For example, the information management device 10 is capable of collectively reading the data package for each mark identification number MID from the storage means 14 and transmitting/receiving the data package to/from another device.

In the illustrated example, the target basic information and the target accompanying information are clearly classified, but the two are not necessarily clearly classified, and information related to the target OB is collectively treated as target related information. May be

As shown in FIG. 1B, one target (for example, a scratch on the wall surface) OB1 is provided with unique mark identification information MID01, and target related information (target basic information and target accompanying information) about the target OB1 is added to this. Information) is associated.

The target basic information is, for example, information indicating the presence of a target (for example, a scratch) OB1 in a predetermined space (for example, a room) (positional information, information indicating a shape based on "characteristic points", etc.).

The target-related information includes, for example, a plurality of photographs (image data J11, J12, J13) of the target OB1 taken from different angles, character data W11, W12, and voice data R11 for explaining the state of the target OB1. .. The associating unit 25 associates the image data J11, J12, J13, the character data W11, W12, and the voice data R11 with the mark identification information MID01 as a key and records them in the related information database as one data unit. There is.

Similarly, other target (for example, a scratch on a different wall surface) OB2 is provided with unique mark identification information MID02, and the target related information (target basic information and target related information (image data J21) about the target OB2 is added thereto. , J22, character data W21, voice data R21) are associated with each other and recorded in the related information database as one data unit, and the other target OB3 and OB4 have their own mark identification information MID03, MID04 is added, and the corresponding target related information is associated and recorded in the related information database as one data unit.

FIG. 7 is a diagram showing an overview of the overview display means 29, FIG. 7A is a block diagram of the overview display means 29, and FIG. 7B shows the position of the mark image M in two dimensions. It is a display example of the information management device 10 showing a state of being mapped as an image and being visually recognized (overlooking).

As shown in FIG. 9A, the overall view display unit 29 includes at least a mapping unit 291. The mapping unit 291 refers to the related information database 30 and maps the positions of one or more target OBs (mark images M) in a predetermined space as shown in FIG. It is a means to see the whole thing. Here, the "overall view" means, for example, visually recognizing (overlooking) the position of the mark image M as a two-dimensional image, or visually recognizing (overlooking) the position of the mark image M as a three-dimensional image, or the mark image. It means to visually recognize the position of M as a 360-degree image (still image, moving image) taken by a 360-degree camera.

The result of mapping by the mapping unit 291 can be output as a two-dimensional image, a three-dimensional image or a 360-degree image (displayed on the display unit 13, printed, output as electronic data). For example, the map 35 shown in FIG. 3B is an example in which the mapping result is displayed as a two-dimensional image. In the present embodiment, the two-dimensional image shown in FIG. 1B is illustrated as an example of the map 35, but the map 35 of the present embodiment is not limited to this, and is displayed as a three-dimensional image or a 360-degree image. Including things. By mapping, it is possible to easily understand the position of a certain target OB in a predetermined space and the positional relationship between the plurality of targets OB1 to OB4 (mark images M1 to M4). Further, when the map 35 of the present embodiment is displayed on the display unit 13, for example, by operating (pressing, touching, tapping, etc.) the displayed mark image M, the map 35 is associated with the mark image M. The target related information is a so-called clickable map that can be displayed or edited, for example, as a thumbnail.

The map 35 can also be browsed by a Web browser or the like, and can be transmitted/received as data to/from other devices.

Further, as shown in FIG. 9A, it is preferable that the overall view display unit 29 includes a collation unit 292 and an environment information acquisition unit 293. With these, for example, information on the space (for example, a two-dimensional image such as a design drawing) corresponding to the map 35 shown in FIG.

The matching unit 292 uses the map 35 indicating the positional relationship between the information (environmental data 36) of the space (environment) in which the target OB corresponding to the mark image M is included and the mark image M in the space, or in the space. The coordinate data indicating the position of one or more target OBs (mark image M) is collated, and the environmental data 36 is superimposed on the background of the map 35 and output (displayed on the display unit 13, printed, PDF file or spreadsheet file). , Output in electronic data format such as image data). The environment data 36 is, for example, a two-dimensional image (still image, moving image) of the space including the target OB corresponding to the mark image M, a three-dimensional image (still image, moving image) of the space, or the space (for example, indoors). CAD data (for example, CAD data created by BIM (Building Information Modeling) and CIM (Construction Information Modeling)), point cloud data (3DCG) of the space, 360-degree image (omnidirectional image) of the space, and the like. Yes, specifically, for example, a design drawing, a floor plan, a floor plan, a captured image of a space (still image, moving image), or the like.

The environment data 36 is generated by, for example, the environment information acquisition unit 293 described later, or is prepared in advance by a separate device (camera, CAD, etc.) and stored in the storage unit 14. The environment data 36 is collated with the map 35 generated by the mapping unit 291 by being called by the collation unit 292 based on the user's operation, and is superimposed and output. In this case, the environmental data 36 and the map 35 are each provided with a matching reference point which is a reference for matching, and the matching means 292 automatically matches the matching reference points of the environment data 36 and the map 35. The scale and the rotation are performed as necessary, and the environment data 36 is superimposed on the background of the map 35 and output.

More specifically, when the space is a room in a building and the mark image M is associated with the object B in the room, the design drawing (two-dimensional image) of the room and the map 35 of the mark image M are displayed. An example will be described in which the layers are overlapped.

In this case, reference points for verification are given to the design drawing at the positions of the window frame and the position of the entrance door, for example, and are stored in the storage unit 14 in advance as the environmental data 36. The mapping means 291 also generates a map 35 as shown in FIG. In this map 35, reference points for verification are provided at positions corresponding to the window frame and the entrance door. Although the map 35 is displayed as a two-dimensional image having only (X,Y) coordinates as shown in FIG. 7B, the mark image M is (X,Y) as the target position information of the corresponding target OB. , X) coordinate values. Therefore, the matching unit 292 selects the coordinate axis of the map 35 according to the coordinate axis of the environment data 36 to be matched. For example, when the environment data 36 is a two-dimensional image, only the (X, Y) coordinates are selected as the coordinate axes of the map 35 and displayed as a two-dimensional image. When the environment data 36 is 3DCG, the coordinate axis of the map 35 is (X, Y, X), so that the mark image M is displayed at the correct position even within 3DCG.

The collating unit 292 displays the collable environment data 36 and the map 35 on the display unit 13 in a list format or the like. When the environment data 36 and the map 35 to be collated are selected by the user, the collation unit 292 calls the corresponding environment data 36 and the map 35 from the storage unit 14, and the collation reference point (window frame) of the map 35 and the environment. The matching reference point (window frame) of the data 36 is matched, and the matching reference point (entrance door) of the map 35 and the matching reference point (entrance door) of the environmental data 36 are matched. At this time, if the size and inclination of the environment data and the map 35 are different, the collating means 292 changes the scale of both or one, or performs rotation or the like to match the size, direction, and position of both. Let Further, the collation unit 292 displays the collation data 37 in which the environment data 36 is superimposed on the background of the map 35 on the display unit 13. The collation data 37 can be printed as it is, can be stored in the storage unit 14 as a set of data units, and can be transmitted/received between other devices.

Further, as described above, the map 35 of the present embodiment is a clickable map, and the collation data 37 obtained by superimposing the map 35 and the environment data 36 is also a clickable map.

A plurality of reference points for matching given to each of the environment data 36 and the map 35 are given according to the environment data 36. However, if there is a shortage in matching, even if a part is manually adjusted. Good.

If the environment data 36 is a 360-degree image, for example, the position (shooting position) of the camera (360-degree camera, omnidirectional camera) and the shooting direction when shooting the 360 image are specified on the map 35 (tap, etc.). By doing so, the collation data 37 in which the 360-degree image and the map 25 are superimposed is generated.

The map 35 and the environment data 36 created in advance may be stored in the storage unit 14 in association with each other in advance by, for example, a collation ID, and a collation operation by the user (for example, selection of a collation menu (not shown)). , Or the operation of the matching button), the matching means 292 automatically calls the corresponding environment data 36 and the map 35 from the storage means 14 based on the matching ID and performs matching based on the matching reference point. It is also possible to adopt a configuration in which both are output in an overlapping manner.

The matching means 292 is not limited to automatic matching (matching), and may be manually matched by a user operation. After the user selects the environment data 36 and the map 35, the two are manually overlapped, and either one or the other is enlarged, reduced, or rotated by, for example, a pinch operation or a slide operation, and collation data 37 is created. You can Further, both automatic collation and manual collation may be provided, only one of them may be provided, or either may be selectable.

The reference points for collation of the environmental data 36 and the map 35 can be changed (reset) at any time.

The environment information acquisition unit 293 is a unit that acquires and generates the environment data 36 by the imaging unit 15 included in the information management device 10 or a device (CAD, PC, camera, 360 degree camera, etc.) different from the information management device 10. is there. The environment information acquisition unit 293 acquires and generates the environment data 36 in advance and stores it in the storage unit 14.

Further, the overview display means 29 can collate the environment data 36 and the map 35 by the collating means 292 while the environment information acquiring means 293 acquires and generates the environment data 36 in real time.

Further, for example, the environment data 36 (for example, the drawing) acquired by the environment information acquisition unit 293 can be collated with the map 35 using the live view image LV as a guide. When the user performs an environmental information acquisition operation (for example, a button operation (not shown) or the like), the information management device 10 activates the image pickup means 15, picks up the live view image LV, and displays it on the display means 13. Then, the environment information acquisition unit 293 waits for reception of the reference point for collation.

The map 35 has scale and position information of an actual size. On the other hand, the environmental data 36 (for example, the drawing) does not have the actual size scale or the positional information, or even if it has the positional information, it is the positional information and the scale made on the basis different from the space. Therefore, one side on the live view image LV[0] is designated in order to match the scale and position of the environment data with the actual space corresponding to the map 35. The map 35 is a bird's-eye view (overall) of the space displayed by the live-view video LV. By designating one side on the live-view video LV[0], the corresponding side of the map 35 is specified. It Then, by designating one side corresponding to the one side on the environment data 36, the actual space and the environment data 36 can be collated.

Specifically, the user selects, for example, one side on the live view image LV in the room (for example, one side of the ceiling of the room being imaged) to perform a reference point providing operation for verification (for example, a slide operation at a corresponding position or Touch (tap) operation of two points on the one side is performed. When this operation is performed, the environment information acquisition unit 293 acquires and stores the reference point for verification (or a reference side that is a set of points or a reference side that connects two points), and if necessary, the live view image LV. A reference marker or the like is displayed on the top.

As a result, the reference point for comparison (side) is also added to the corresponding one side of the map 35 showing the space displayed by the live-view video LV in a bird's-eye view (overall).

Next, when the user performs a matching operation (for example, operation of a matching button (not shown)), the matching means 292 waits for selection by the user, for example, by displaying a list of environment data 36 that can be matched. .. In the environment data 36, reference points (sides) for verification are provided at corresponding positions (one of the upper sides of the drawing in this example). The matching reference point (side) may be given in advance, or after the matching reference point is added to the map 35, the matching reference point (side) corresponding to the environmental data 36 is (re)repeated. It may be one that can be set.

When the user selects the predetermined environment data 36, the matching means 292 causes the matching reference point (side) of the live view image LV (map 35) and the matching reference point (side) of the environment data 36 to match. As necessary, the scale is changed, the rotation is performed, and the like, and the collation data 37 is created and displayed on the display unit 13. That is, the corresponding map 35 is displayed on the display means 13 (on its upper surface) so as to be superimposed on the environmental data 36.

The matching map 35 is based on the position information of the reference point for matching of the live view image LV (the position information of the reference point for matching relative to the origin in the real environment specified by the position specifying means 23). 292 may be configured to automatically select from the storage means 14. The map 35 selected by the matching means 292 may be configured so that the user can manually change it.

In this way, the overall-view display unit 29 can display the live view image LV of the room or the like on the display unit 13 and display the corresponding space map 35 on the display.

Also, in the case where the target space is imaged (scanned) and the point data (3DCG) is acquired as the environment data 36 in real time by the environment information acquisition unit 293, the map 35 is displayed in an overlapping manner, the same as above. Can be implemented.

The case where the environment data 36 is a 360-degree image will be further described as an example. The 360-degree image is data captured by a 360-degree camera (omnidirectional camera). For example, in order to acquire a 360-degree image of a room, a 360-degree camera is arranged in advance, for example, in the approximate center of the room, and the entire room is photographed in a 360-degree range. This shooting position is, for example, the initial position when the mark image M is attached. Then, the shooting position of the 360-degree camera is used as a reference point for collation of the environment data 36, and the initial position of the operation of giving the mark image M is set as the origin of the map 35 to collate them. Alternatively, the shooting position of the 360-degree camera is set to an arbitrary position, and the position (shooting position) of the 360-degree camera is acquired by, for example, the GPS included in the 360-degree camera or the same function as the position specifying unit 23 of the present embodiment. However, it may be collated with the map 35 based on that. In this case, the 360-degree camera may be provided in the information management device 10 as a part of the image pickup means 15, or may be a 360-degree camera separate from the information management device 10. Further, in the case of a 360-degree camera which is separate from the information management device 10, the image management operation may be performed by a remote operation from the information management device 10.

Note that, as shown in FIG. 8, in the present embodiment, it is also possible to display a list of the added mark images M in a list format. This figure is a diagram showing an example of a list (thumbnail) display of the mark images M in a list format. FIG. 9A is a display example of a list display (mark image list), and FIG. 9B is a table in a state in which a specific mark image M (for example, mark image M3) is selected from the mark image list. It is an example.

In the present embodiment, it is possible to photograph the real environment with the mark image M added during the mark adding operation. Further, the photograph taken in that case can be associated with the mark image M as the target related information, and can be displayed as a thumbnail as the mark image list. In the present embodiment, even when the mark image M is displayed and photographed on the live view image LV, only the real environment that does not include the mark image M is photographed as the image captured and stored in the storage unit 14. It is an image. On the other hand, in the case of displaying in the mark image list (thumbnail) (FIG. 9A) or when selecting a specific image from the mark image list and displaying the entire display area of the display unit 13 (preview display) (FIG. In (B), the mark image M(M3) is added to the image of the real environment based on the target position information associated with the mark image M(M3) (the image is reconstructed) and displayed. I have to.

The information management program 20 may also include a document creating means 22. The document creating means 22 can acquire the data units stored in the related information database in units of mark image IDs individually or individually, and can create a document (for example, a report) as it is. The document creating means 22 may be provided with a predetermined format for, for example, a frequently used document (for example, a report, an estimate, a purchase order, etc.), and the selection of these formats and the necessary data unit The necessary documents may be created by the selection/download operation (only). Further, if necessary, the downloaded data may be processed or edited.

<Flow of information management method (information management processing)>
The information management processing by the information management means (information management program) 20 of this embodiment will be described with reference to FIG.

The mark giving means 21, the position specifying means 23, the associating means 25, the overall view displaying means 29, and the document creating means 22 of this embodiment together with the AR processing means 27 perform information management processing.

FIG. 9 is a schematic diagram showing the information management process. As shown in the figure, in the information management program 20, the initial process (step STA), the mark giving process by the mark giving means 21 (step S04), the associating process by the associating means 25 (step S05), and the overview display means 29 are performed. The overall display process (step S06), the document creation process by the document creating means 22 (step S07), and the like can be executed. Further, in the initial processing (step STA) and the mark giving processing (step S04), the image pickup processing by the image pickup means 15 (step S01), the display processing by the display means 13 (step S02), and the position identification by the position identification means 23 are performed at any time. The process (step S03) is performed. In addition, in the overview process (step S06), an imaging process (step S01), a display process (step S02), and a position specifying process (step S03) may be performed.

Each step (step S01 to step S07) after the initial process (step STA) can be executed in any order. In FIG. 9, numbers are given as the names of the steps for convenience (S01 to S07), but these do not indicate the order of execution.

First, the information management program 20 executes a predetermined initial process (step STA) every time it is started. This initial processing includes initialization and acquisition of a reference position (initial position) when performing information management processing.

In the initial processing (step STA), the image pickup means 15 is activated, and the information management apparatus 10 resets the reference position (initial position) by the position specifying means 23 (new reference position ( Obtain the initial position). To obtain the initial position, for example, while displaying the live view video LV on the display unit 13, the image capturing unit 15 captures a plurality of images of the real environment (peripheral space) around the information management device 10 (display processing in step S02). , The image capturing process of step S01), and the position of the player in the real environment is grasped (estimated) based on the captured image (position specifying process of step S03).

Alternatively, the image capturing unit 15 may perform a scan (image capturing) in advance, and the initial position may be acquired using the information about the space of the real environment (environment data 36) stored in the storage unit 14. Specifically, in the initial processing STA, the environment data 36 is loaded from the storage unit 14, and the image of the real environment captured by the image capturing unit 15 is compared with the environment data 36 (imaging process in step S01, step S02. Display processing). When the two match, the relative position from the origin of the environment data 36 may be set as the initial position as the current position of the own device (position specifying process of step S03).

In any of the above cases, the position of the own device may be grasped by detecting the reflected light of infrared rays or laser.

After acquiring the initial position, the position specifying unit 23 acquires (moves sequentially) the movement information of the own device from the initial position and stores it in the storage unit 14 until the initial position is reset.

In the mark giving process (step S04), marks are individually given to the target OBs to which information is to be given, and the position (relative position from the initial position) of the target OBs in the real environment is grasped. The mark is given by a mark giving operation by the user. When the mark giving operation is performed, the position specifying means 23 acquires the movement information from the initial position at that timing (imaging process of step S01, display process of step S02, position specifying process of step S03) Mark giving position Is stored in the storage means 14.

In the associating process (step S05), the associating unit 25 collectively associates and manages a plurality of types of target related information regarding the target OB with the mark image M. Even at a later timing, the target related information is added to the mark image M (target OB) by performing the association start operation.

In the overview process (step S06), the mark image M is displayed as the map 35, or the map 35 is displayed so as to overlap the environment data 36.

In the document creation process (step S07), for example, the target related information is arbitrarily embedded in a predetermined format to create a report or the like.

With continued reference to the figure, the information management method (processing of the information management program 20) according to the present embodiment will be described with a specific example. Here, as an example, a case where a repair place (for example, a scratch on a wall) or an inspection place in one room provided on one floor of a building is a target OB, and information is given to each target OB explain.

For example, at a building/construction site or an inspection/maintenance site in equipment management, for example, a reporter (on-site) about a repair part or an inspection part (target OB) that occurs in a predetermined space (indoor) in a building. It is common to share information about the target OB and its related information among multiple parties, such as workers who check the situation), repair personnel, inspection personnel, site managers, and construction personnel. ..

First, for example, a reporter (user) activates (executes) the information management program 20 of the present embodiment in a room where confirmation work is performed. As a result, the camera function is activated as an initial process (step STA), and the reference position is reset. Specifically, the camera function is activated, and the image pickup means 15 is ready for image pickup. The image capturing means 15 captures an image of the real environment in the room within the range of the angle of view of the lens 151, acquires the live view image LV (step S01), and displays it on the display means 13 of the tablet terminal 101 (step S02).

For example, the user uses the imaging unit 15 to view the real environment (peripheral space) around the information management device 10 based on a guide or the like displayed on the display unit 13 that prompts the user to take a plurality of images. The image is captured a plurality of times with the image LV displayed. The position specifying unit 23 specifies the initial position by grasping (estimating) the position of the own device in the real environment based on the plurality of captured images, and stores the initial position in the storage unit 14 (step S03). The method for acquiring the initial position is not limited to this, and the method described in the above-described embodiment is appropriately adopted. After setting the initial position, the position specifying means 23 acquires the terminal movement information at any time and stores it in the storage means 14. In addition to this, for example, movement tracking information (movement log, action log) of the terminal may be stored together in the storage unit 14 using motion tracking or the like, and may be reproducible.

After setting the initial position, the user attaches the mark image M to the existing object (eg, scratch on the wall) OB to which the information should be attached (step S04). When the user can approach the target OB, the user approaches the target OB, and when the user cannot approach the target OB, acquires the live view image LV of the target region including the target OB from a distance (step S01, imaging process (process)), and the display unit 13. (Step S02, display step (processing)).

When the user performs the mark giving operation using the adjustment value, the mark giving means 21 displays the mark image M movably so as to be superimposed on the live view image LV. Alternatively, the mark giving means 21 displays the mark image M based on an operation of the user giving an instruction to start the mark giving (mark giving operation button 122) or the like.

The user attaches the mark image M to the target OB included in the live-view video LV, and thus secures a distance corresponding to the adjustment value with respect to the target OB, and through the display unit 13 (from above the display unit 13). An operation such as sliding movement of the mark image M is performed so that the mark image M is superposed on or brought close to the position of the target OB. Then, the mark image M is moved to a desired mark application position, and a mark application position determination operation (operation such as tapping of the mark image M or operation of the mark application operation button 122) is performed to determine the position of the mark image M.

When the user performs the mark giving operation using the plane detection, the mark giving means 21 displays the guide display G (for example, a grid-shaped label image showing the detected plane) overlaid on the live view image LV, and omits it. The mark image M is fixedly displayed near the center.

The user adds the mark image M to the target OB included in the live-view video LV, and thus the fixedly displayed mark image M is superposed on (or located in the vicinity of) the target OB. The posture, direction, or position of the (terminal) 10 is arbitrarily moved. Then, for example, the position of the mark image M is decided by performing the mark giving position decision operation by the user.

When the mark giving position determining operation is performed by the user, the mark giving means 21 can change the display mode of the mark image M (for example, display it in a blinking manner) and associate the information by the associating means 25 described later. This suggests to the user (photographer).

Further, when the mark giving position determining operation is performed by the user, the position specifying means 23 specifies the relative position of the target OB in the predetermined space.

Specifically, when the mark providing position determining operation is performed using the adjustment value (see FIG. 3A), the position specifying unit 23 (state acquiring unit 17) actually calculates the mark based on the position of the mark image M. The relative distance between the own device and the target OB in the environment is specified as the adjustment value. Further, based on the terminal movement information acquired from time to time, the relative terminal movement information from the initial position to the position where the mark giving position determination operation is performed is calculated, and the relative position of the target OB with respect to the initial position is grasped, The target position information is specified and stored in the storage unit 14.

Further, when the mark giving position determining operation is performed by plane detection (see FIG. 3(B)), the position specifying means 23 determines the own device and the target OB in the real environment based on the positions of the guide display G and the mark image M. Estimate the relative distance of. In addition, the position of the target OB is specified based on the terminal movement information acquired at any time and the relative distance between the own device and the target OB, and the target position information is stored in the storage unit 14. (Step S03, position specifying step (processing))

In this way, the position identifying means 23 grasps the relative position (target position information) of the target (scratch) OB from the initial position.

When the mark giving position determining operation is performed by the user, the mark giving means 21 changes the display mode of the mark image M (for example, blinks the display), and the like, so that the target related information (subject accompanying) is given to the mark image M. It is suggested to the user that the association of information) has become possible (start of the association step (association process (step S05))). That is, when the mark image M is displayed in a blinking manner, the user can add target accompanying information such as related image data to the mark image M. Further, the image data (such as a photograph of the target OB in the real environment) taken by the image pickup unit 15 when the mark image M is displayed blinking, the voice data input through the voice input unit 18, and the like are It can be automatically associated with the mark image M.

When the mark images M are successively given to the plurality of target OBs, for example, the mark giving operation button 122 is operated (touched (tapped)) to instruct the start of mark giving. As a result, the mark image M, which indicates to the user (photographer) that the information can be associated by the associating unit 25 described later, for example, by blinking, is displayed as a still image and a new mark is displayed. The image M is displayed.

The mark giving step and the associating step can be performed at any timing. In other words, the association process may be performed on each mark image M after assigning a plurality of mark images, or a certain mark image M may be assigned and subsequently the association process may be performed on the mark image M. Alternatively, the mark giving operation button 122 may be operated (touched) to instruct the start of new mark giving, and the associating process may be performed on the new mark image M.

In the associating step (process) of step S05, the target related information of the scratch OB is stored in the storage unit 14 in association with the mark image M1.

When the user performs a mark attachment position determination operation or an association start operation (for example, a tap operation on the mark image M) on the mark image M, the target related information (particularly target accompanying information) is attached to the mark image M. be able to. The associating unit 25 selects one or more data (files) DD stored in the storage unit 14 that can be associated with each other in response to the association start operation, for example, in a list format. It is displayed on the display means 13 (see FIG. 5). When the user performs an operation of selecting the data DD to be associated (for example, a tap operation of the data), the associating unit 25 associates the data as the target accompanying information with the mark identification number MID unique to the mark image M, and associates the data. It is stored in the information database 30. The selection of the target related information by the user and the associating process by the associating unit 25 are repeatedly executed until, for example, the association end operation (for example, the association operation button 126 (see FIG. 5)) is operated by the user.

Alternatively, when the target related information is image data, the camera function of the tablet terminal 101 may be activated by an operation for starting the association with the mark image M. Further, the association start operation for the mark image M may be configured to allow selection from a list display of selectable data (files) DD and shooting by a camera function. During the association process, the image captured by the camera function is automatically associated with the mark image M as target incidental information. In addition, the information generation unit 2251 (see FIG. 2) performs AI analysis on images (still images, moving images) such as photographs and videos, and generates tags of names of shooting targets (for example, “scratches”, “doors”, etc.). Then, the image data with a tag added to the image may be generated and associated as the target accompanying information.

In addition, when the target related information is voice data, the microphone (recording) function of the tablet terminal 101 may be activated by an association start operation for the mark image M1. During the association process, the voice data recorded by the microphone function is automatically associated with the mark image M as target accompanying information. Alternatively, the voice input by the information generation unit 2251 (see FIG. 2) may be recognized, character data may be generated, and the character data may be associated as target incidental information.

In addition, when the target related information is character data, the memo function of the tablet terminal 101 may be activated by an association start operation for the mark image M1. During the association process, the character data (text data) input by the memo function is automatically associated with the mark image M as target accompanying information.

It should be noted that the image data, voice data, character (text) data, etc. input during the association process are not automatically registered in the related information database 30, but are temporarily stored as related data in the related information database 30 as candidate data to be associated with the mark image M. You may make it memorize|store in the memory|storage means 14 other than.

Further, when the image pickup means 15 of the tablet terminal 101 captures an image while the information management program 20 is being executed, when the voice data is input by the microphone 18, or when the character data is input by the memo function. The data (image data, voice data, character (text) data, etc.) and the terminal movement information at the input points are associated with the mark identification number MID unique to the mark image M by the association means 25, and the associated information is obtained. It is registered in the database 30. Note that such information may be temporarily stored in a predetermined storage unit 14, associated with the mark identification number MID in the subsequent step of associating the target accompanying information, and registered in the associated information database 30.

In this way, one or more pieces of target-related information can be associated with the mark image M following the application of the mark image M. Thereby, for example, the target related information (target accompanying information) can be directly linked to the mark image M at the timing when the mark image M is attached.

Specifically, a photograph taken of the target OB1 from different angles or a photograph taken while gradually increasing the distance from the target OB (while gradually increasing the angle of view) is associated with the mark image M on the spot. be able to.

Even if a large number of image data (photos) are acquired in order to increase the amount of information, even if the photographer leaves the scene and the time passes, it is easy for the photographer to find out where the data was taken. Is cumbersome and labor intensive.

According to the present embodiment, it is possible to associate the target incidental information with the mark image M while staying at the site, for example, so that the work load on the user can be significantly reduced.

Even after a series of selection operations have been completed for one mark image M and the association end operation has been performed, the association start operation is performed again for the same mark image M, so that the mark image M is targeted. Related information can be associated (addition, correction, etc.).

While the information management program 20 is being executed, a corresponding mark image M is attached to another target OB (for example, a scratch OB generated at another location) in the same space (for example, a room), and the target relation to the mark image M is given. Information can be continuously linked.

When there are a plurality of mark images M, the position specifying process similar to the above is performed for each mark image M, and the position of each target OB in the space (for example, the room) is specified.

In addition, when the mark giving position determining operation is performed, the mark giving unit 21 and the AR processing unit 27 show “characteristic of the live view image LV of the target area including the target OB displayed on the display unit 13. You may acquire and analyze the information of "a point", and may acquire object basic information. The “characteristic points” are, for example, a large number of points indicating the outer shape of the target OB.

In the overview display step (process) of step S06, when the user performs a overview display operation (for example, an operation of a overview display button (not shown)), the mapping unit 291 of the overview display unit 29 causes the related information database 30. With reference to, a map 35 (see FIG. 7(B)) that allows the overall positional relationship of all the mark images M in the room to be created can be created and output. The output includes display on the display unit 13, printing, output in an electronic data format such as a PDF file or a spreadsheet file.

In the map 35 displayed on the display unit 13, it is possible to select each mark image M, and when a user selects a certain mark image M (for example, the mark image M1) (tap operation or the like), the target related information. (Target incidental information) is displayed as a thumbnail or a list. The user can perform operations such as viewing and editing such information, and adding and deleting data. Further, for example, the mark image M may be selectively displayed on the map 35. When there are too many mark images M, only the necessary images can be displayed on the map 35 at any time.

By mapping the mark image M, it is possible to easily understand the position of the target (for example, a scratch) OB in the room and the mutual positional relationship.

Further, in the overall display step (process), the collating means 292 shows the information (environmental data 36) of the space (environment) including the target OB corresponding to the mark image M and the positional relationship of the mark image M in the space. The map 35 shown can be collated and output as collation data 37 in which the environment data 36 is superimposed on the background of the map 35.

Further, in the overview display process (processing), the environment information acquisition unit 293 is used to perform the image pickup unit 15 included in the information management device 10 or a device (CAD, PC, camera, 360 degree camera, etc.) different from the information management device 10. The environmental data 36 can be acquired and generated. The environment information acquisition unit 293 can acquire and generate the environment data 36 in advance and store it in the storage unit 14, and the collation unit 292 stores the environment data 36 and the map 35 generated and stored by the environment information acquisition unit 293. The collation data 37 can be created by superimposing them. Further, the environmental information acquisition unit 293 can collate the environmental data 36 and the map 35 by the collation unit 292 while acquiring and generating the environmental data 36 in real time.

As a result, for example, an image (photograph) of the room, a design drawing, etc. can be superimposed on the map 35 and output, making it easier to understand the situation at the site, and especially for a person who has never been to the site. , It becomes easier to assume the situation at the site.

In addition, it can be transmitted/received to/from another device in a map state (as map format data) or as collation data 37. It is desirable that the map format data or the matching data 37 include corresponding target related information (data unit) for all the mark images M. When the information management program 20 is installed, the other terminal (for example, another tablet terminal) that has received the map format data or the verification data 37 receives the map 35 or the verification data shown in FIG. 7B. 37 can be output at any time (displayed on the display means, printed, output as electronic data), and when the target-related information is also included in the map format data or the collation data 37, the mark image M It is possible to browse, edit, etc. the target related information associated by selection.

Note that the mark image M itself may be deleted instead of only the selection of the mark image M to be displayed as a whole. The deletion of the mark image M is performed by a mark deletion operation different from the operation related to the mark image M (for example, a swipe operation of sliding while touching the mark image) such as the above-described mark addition operation. Alternatively, confirmation of deletion may be performed by pop-up or the like. In this case, the related information may be automatically deleted by erasing the mark image M, may be selectively erasable, and the related information may not be automatically erased. It can be erased by an erasing process).

The document creation step (process) of step S07 is a step of creating a document using the data stored in the related information database 30. For example, when the user selects the required target OB (for example, at least one of the mark images M displayed as the map 35), the document creating unit 22 displays the target relevant information associated with the mark image M. It is acquired from the related information database 30 and embedded in a predetermined format (for example, a report) to automatically create a document.

Further, the document creating unit 22 may be configured to be able to selectively specify a necessary item (target related information) for creating a document.

Further, the document creating means 22 may be provided with a plurality of formats of frequently used documents (for example, reports, quotations, purchase orders, etc.), and the user selects the formats of these documents and needs them. By selecting the target OB (for example, selecting at least one of the mark images M displayed as the map 35) and the necessary items (target related information) for creating the document as necessary, the document can be easily written. Can be created. Further, if necessary, the downloaded data may be processed or edited.

The procedure of the actual process (processing) is not limited to the order of the above description, and the necessary processing can be repeated in any order. In this embodiment, the position of the target OB is specified by the position specifying unit 23 and the state acquisition unit 17, and the state acquisition by the state acquisition unit 17 is performed based on the reference position.

It should be noted that the AR framework 27 recognizes the shape of the object and recognizes the unique name of the object (such as "wall", "chair", "desk") by, for example, collating the information of the object stored in the storage means. Alternatively, the image of the object displayed in the live view image LV of the real environment may be combined (superposed) with digital information (such as an object image related to the object) other than a horizontal plane or a vertical plane. ..

Further, the arrangement and the situation of the combined (superposed) object images may be recorded and stored. In this case, when the target area (real environment) is visited at a later timing (a few days, a few weeks, etc.) and the application program using the AR framework 27 is started, the previously placed object image is placed at the same position. Can be viewed and edited again. Also, an associating operation can be performed.

<Information management system>
Next, the information management system 100 of this embodiment will be described. FIG. 10 is a schematic diagram showing the overall configuration of the information management system 100.

As shown in the figure, the information management system 100 includes at least one information management device 10 (for example, information management devices 10A to 10C), a network 210, a server device 220 connected to the network 110, and a cloud server device 230. And so on.

Since each of the information management devices 10A to 10C has the same configuration as the above-described information management device 10 (tablet terminal 101), description thereof will be omitted.

The network 210 is an arbitrary communication network in which the respective devices can communicate with each other, and may be a wired communication network, a wireless communication network, or configured by both of them. You may do it. Further, the network 210 may be configured by one communication network or may be configured by a plurality of communication networks. For example, the Internet, a public telephone line network, a wide area communication network for wireless mobile units such as so-called 3G lines and 4G lines, a WAN (Wide Area Network), a wireless communication network for performing communication conforming to a LAN (Local Area Network) standard, Short-range wireless communication channels such as Bluetooth (registered trademark), Wi-Fi (registered trademark), NFC (Near Field Communication), infrared communication channels, HDMI (High-Definition Multimedia Interface) and USB The network 210 may include a communication network or communication path of any communication standard such as a wired communication network conforming to a standard such as (Universal Serial Bus).

As described above, each information management device 10 includes the information communication unit 16 (see FIG. 2), and is connected to another information management device 10, the server device 220, the cloud server device 230, and the terminal device 250 via the network 210. Then, the target related information stored in the related information database 30 can be transmitted and received, for example, in a data package unit (in mark identification number MID unit) or in a batch of a plurality of target related information.

As an example, the server device 220 and the cloud server device 230 respectively include storage means 221 and 231. The data package of the target related information generated by one or more information management apparatuses 10 is acquired via the network 210, and the storage means 221 and 231 are provided. You may make it memorize|store in 231. The certain information management device 10A acquires, for example, a data package of the target related information generated by the other information management device 10C and stored in the storage means 221 and 231 of the server device 220 or the cloud server device 230, The management program 20 can be activated to display, browse, edit, etc. the map 35 or the like on the display unit 13.

As described above, according to the information management system 100 of the present embodiment, for example, one information management device 10 is brought to the site, and the acquired target OB and the target related information relating thereto are acquired by a plurality of devices (others) via the network 210. Information management device 10, server device 220, cloud server device 230, terminal device 250, etc.). The method of sharing the data may be a configuration in which the data is attached to an email and transmitted/received.

<Information cooperation>
Next, cooperation (sharing) of the target related information in this embodiment will be described with reference to FIGS. 11 to 19.

FIG. 11 is a block diagram showing an example of the configuration of the information management device 10, FIG. 12(A) is a block diagram showing an outline of the cooperation means (cooperation processing), and FIG. 12(B) shows the origin marker SP. It is a figure which shows an example. Note that FIGS. 11 and 12 exemplify a configuration in which the cooperation unit 50 is provided in the information management device 10 (information management program 20). However, not limited to this, in the case of the information management system 100 including the information management device 10 and another device as shown in FIG. 10, the linking unit 50 can be shared by the information management device 10 and another device. The configuration may be provided (for example, in the server device 220 or the cloud server device 230).

With reference to FIG. 11, the information management device 10 of the present exemplary embodiment has a coordinating unit 50, and is capable of coordinating (sharing) target related information in different spaces, specifically, in data handled in different spaces. .. Hereinafter, cooperation (sharing) of the target related information by the cooperation unit 50 may be referred to as cooperation processing.

Here, the “space” is, in this example, an internal virtual space ARS in the information management device 10 (information management program 20) and a virtual space (virtual two-dimensional space, virtual three-dimensional space) different from the internal virtual space ARS. Say. That is, the different virtual space is, for example, another internal virtual space ARS or virtual two-dimensional space recognized and generated by the same information management device 10, or an external device or system (other information management device 10, terminal device). 250, another imaging device, a virtual two-dimensional space, another device having a coordinate system of a virtual three-dimensional space, and the like), such as a virtual three-dimensional space and a virtual two-dimensional space.

When the information management program 20 is activated, the information management device 10 of the present embodiment creates a virtual three-dimensional space that is a work target of the information management program 20 (for example, in response to a predetermined button operation or the like).

Specifically, when the information management program 20 is executed (started) and the environment information acquisition operation is performed, the image pickup means 15 is started and the live view video LV is displayed. The environment information acquisition operation may be, for example, a predetermined button operation by the user, a (control) processing start operation by the information management program 20, or a switching operation to another processing.

When the environment information acquisition operation is performed, the information management device 10 (for example, the environment information acquisition unit 293) waits for the reception of the virtual workspace generation operation, and when the virtual workspace generation operation is performed, the live view video LV. A virtual three-dimensional space corresponding to the real space recognized through is generated. Note that the virtual workspace creation operation may be performed at the same time when the environment information acquisition operation is performed.

Hereinafter, the virtual three-dimensional space corresponding to the physical space recognized through the live view image LV of the information management device 10 will be referred to as an internal virtual space ARS. The internal virtual space ARS is a work target area of the information management program 20. In the internal virtual space ARS, the mark image M is added (as data handled by the internal virtual space ARS) and target related information is associated, and the virtual 2 The map 35 is generated as data in the dimensional space or the virtual three-dimensional space.

The internal virtual space ARS disappears when the virtual work space ending operation or the information management program 20 ends, and is newly created when the next virtual work creating operation is performed. The virtual workspace ending operation may be, for example, an appropriate button operation by the user (for example, a dedicated button operation, a switching operation to another process, etc.), or, for example, an initial position (reference position) by the information management device 10. The control means 11 may automatically instruct the virtual workspace end operation based on the (re)acquisition of (1).

The environment information acquisition unit 293 waits for reception of a separate virtual workspace creating operation. The virtual work space generation operation may be, for example, an appropriate button operation by the user (for example, a dedicated button operation, a switching operation to another process, etc.), or a new environment information acquisition operation. For example, the control unit 11 may automatically instruct the virtual workspace generation operation based on the acquisition of the initial position (reference position) by the information management device 10.

With reference to FIG. 12A, the linking unit 50 can link (share) the target relevant information in the data handled in each of the virtual space 1 and the virtual space 2 different from the virtual space 1, for example. .. The virtual space 2 different from the virtual space 1 is a virtual space newly created by the virtual workspace creating operation after the virtual workspace ending operation of the virtual space 1, or a device in which the virtual space 1 is created (for example, information management). The virtual space created and acquired by another device different from the device 10) (for example, the other information management device 10 or the other devices 220, 230, 250 shown in FIG. 10).

The virtual space 1 is, specifically, one of the internal virtual space ARS, the map 35, and the environment data 36 (including those generated and acquired by other devices), and the virtual space 2 is the virtual space 1 and the virtual space 1. Is another virtual section, which is one of the internal virtual space ARS, the map 35 generated by the information management device 10, and the environment data 36 (including those generated/acquired by another device).

As a specific example of the cooperation, the cooperation unit 50 uses different (plural) internal virtual spaces ARS (virtual space 1) and internal virtual space ARS (virtual space 2) generated in a certain (one) information management device 10. ) Can share target related information. For example, the user sets the target related information in a certain internal virtual space ARS (virtual space 1) and is generated in the same information management device 10 at different timings (for example, in a certain internal virtual space ARS (virtual space 1)). After the virtual work space end operation, the registered target related information is linked to the internal virtual space ARS (virtual space 2) newly generated when the virtual work space generation operation is performed again (calling). It can be displayed again.

Alternatively, the linking unit 50 links the target related information registered in the internal virtual space ARS (virtual space 1) of a certain information management device 10 with the map 35 (virtual space 2) of the same information management device 10, The target related information can be displayed again (see FIG. 7). For example, when the information management device 10 performs a generation operation of the map 35 from the display state of the live view video LV (for example, a dedicated button operation by the user or a switching operation to another process), the virtual space 1 is displayed. The virtual space 2 is newly generated by ending the process.

Alternatively, the cooperation unit 50 can share the target related information among the plurality of different information management devices 10. Specifically, for example, the coordinating unit 50, for example, relates to the target related information registered in the virtual space 1 (internal virtual space ARS or map 35) of a certain information management device 10 in the virtual space 2 (internal virtual space) of another information management device 10. It is possible to output (display, etc.) in the other information management device 10 in association with the space ARS or the map 35).

Alternatively, the linking unit 50 can share the target related information between the information management device 10 and the other device 250 in the information management system 100 (see FIG. 10), for example. Specifically, the linking unit 50 is, for example, another device (for example, a terminal device shown in FIG. 10) that can handle (manage) the target related information regarding the target related information registered in the virtual space 1 of the information management device 10. Another space created in 250 (virtual space 2: virtual three-dimensional space and/or virtual two-dimensional space can be linked and shared with data handled in the other space. "Data handled in another space" Is, for example, data handled in a virtual space of a device different from the information management device 10 (for example, the terminal device 250 or the like), and for example, information of a space of a real environment separately acquired by the other device (described above). The environmental data 36) is, for example, three-dimensional image data, three-dimensional CAD data, 3DCG (computer graphics in three-dimensional space) data, point cloud data, a 360-degree image captured by a 360-degree camera. (Still image, moving image) data, 2D data includes 2D image data, 2D CAD data, 2DCG (computer graphics in 2D space) data, etc. Further, the environment data 36 includes the information management device 10. The image of the real environment (moving image, still image) captured in (3) is also included.

The linking unit 50 links the target related information registered in the virtual space 1 of the information management device 10 to the environment data 36 (virtual space 2) of another device (for example, the terminal device 250), so that the environment data 36 is displayed. Can be output to. Further, the coordinating unit 50 sets the target related information registered (set) in the environment data 36 (virtual space 2) of a device (for example, the terminal device 250 or the like) different from the information management device 10 to the information management device 10. In cooperation with the virtual space 1, it can be displayed on the live view image LV or displayed on the map 35.

Furthermore, the linking unit 50 can link target related information between data (environmental data 36 and the like) handled in different spaces (virtual spaces 1 and 2).

The details of the cooperation process will be described below. First, the mark image M of the present embodiment is an object image displayed on the live view image LV on the display unit 13 of the information management device 10, and more specifically, when the user gives a mark, the mark image M is a live image. It is set by the mark giving means 21 in the virtual three-dimensional space (internal virtual space) recognized through the view video LV. That is, each of the mark images M of this embodiment is an image having position coordinates (three-dimensional coordinates) in the virtual three-dimensional space.

Further, the target related information is stored in the storage unit 14 in association with the mark image M by the association unit 25 when the association start operation is performed by the user. In detail, as described above, the target related information is registered in the storage unit 14 as one information group (data unit) using the identification number (mark identification number MID) unique to the corresponding mark image M as a key. .. That is, it can be said that the mark image M of this embodiment has position coordinates (three-dimensional coordinates) in the virtual three-dimensional space and can hold the target related information. Further, it can be said that the target related information associated (held) with the mark image M also has position coordinates (three-dimensional coordinates) in the virtual three-dimensional space. Then, the linking unit 50 performs the linking process described above based on the three-dimensional coordinates of the mark image M.

In the present embodiment, the position specifying unit 23, the mark assigning unit 21, the associating unit 25, the overview display unit 29, the cooperation unit 50, and the like cooperate with the AR processing unit 27 to perform the position identifying process, the mark assigning process, and the associating process. , Overview display processing, cooperation processing, and the like.

The coordinating unit 50, together with the collating unit 292, the environment information acquiring unit 293, and the like, and one or more information management devices 10 and/or one or more other devices 250, target related information (that is associated with the target related information). The linked mark image M) is linked (shared). The collation unit 292 and/or the environment information acquisition unit 293 may be included in the cooperation unit 50, or may be separately provided from the cooperation unit 50. For example, in the example shown in FIG. 7, the overview display means 29 causes the cooperation means 50 (environmental information acquisition means 293, collation means 292) to collate the environment data 36 with the map 35, and superimposes them on the live view video LV. Although the map 35 of the corresponding space is displayed, the cooperation unit 50 (environmental information acquisition unit 293, collation unit 292) is provided in the other devices 220, 230, 250 of the information management system 100 shown in FIG. May be.

FIG. 12B is a schematic diagram of the information management device 10 displaying the live view image LV. The linking unit 50 links the target related information (and the mark image M with which it is associated) based on the position coordinates in the virtual space with the origin marker SP as a reference. Although the details of the origin marker SP will be described later, in the information management device 10, the user can set them on the internal virtual space ARS (on the live view image LV) and the map 35, and on the external device 250 (virtual). In a device having a coordinate system of a two-dimensional space or a virtual three-dimensional space), the user can set it on the environment data 36 or the like. The origin marker SP is means for specifying the starting point (origin) of the virtual space and the direction and size of the virtual space. The origin marker SP can be set by, for example, a point P1 (origin) arbitrarily designated by the user and a point P2 that specifies the direction and size (space vector) of the virtual space starting from the point P1. The reference point for collation described above corresponds to the origin marker SP.

For example, in the above example, the matching unit 292 matches the matching reference point (side) of the live view image LV (map 35) with the matching reference point (side) of the environmental data 36, and the information management device 10 The example in which the corresponding map 35 is displayed (on the upper surface thereof) by superimposing it on the environmental data 36 on the display means 13 or the like has been described. This is because the matching means 292 is set in the internal virtual space ARS (or in the map 35). An origin including the origin of the position coordinates of the target related information (and the mark image M with which it is associated) in the virtual space that matches the origin marker SP (set) and the origin marker SP set in the environment data 36. Match the markers).

The user performs an environment information acquisition operation (for example, an appropriate button operation) to start cooperation. Thereby, the environment information acquisition unit 293 (in response to the virtual work space generation operation) activates the image pickup unit 15, picks up the live view image LV and displays it on the display unit 13, and, for example, utilizes the AR processing unit 27. ) Generate a virtual three-dimensional space corresponding to the real space recognized through the live view video LV.

Then, the environment information acquisition unit 293 uses the virtual space 1 (for example, the information management device) to cooperate (share) the target related information in different virtual spaces (the virtual space 1 and the virtual space 2 illustrated in FIG. 12A). It waits for the reference marker (origin marker SP) in the internal virtual space ARS of 10. The origin marker SP is input by, for example, a user's reference reference point for matching operation. In this example, the user inputs two points (point P1 and point P2) on the live-view video LV in the operation of giving reference points for matching. The input of the point P1 (hereinafter referred to as the point P1 input operation) is an operation of specifying the start position (start point or origin) in the virtual three-dimensional space. The input of the point 2 (hereinafter referred to as the point P2 input operation) is an operation of specifying the direction and size (space vector) of the virtual three-dimensional space having the point P1 as the starting point. That is, the origin marker SP is a marker indicated by a line segment (linear) here, and is a space vector. The reference point (side) for collation described above corresponds to the origin marker SP.

The environment information acquisition unit 293 displays the origin marker SP on the live view image LV in accordance with the reference reference point giving operation by the user. When the user performs a mark giving operation and a mark giving position determining operation on the target OB visually recognized through the live view image LV, the mark giving means 21 gives a mark image M.

The position specifying unit 23 specifies the relative position of the mark image M in the internal virtual space ARS when the mark giving position determining operation is performed. Specifically, the relative position of the mark image M with respect to the origin marker SP of the internal virtual space ARS is specified. As described above, the mark image M has coordinates indicating the position in the virtual three-dimensional space, and the origin of the coordinates is the origin (point P1) of the origin marker SP.

As shown in FIG. 12A, the target related information registered in the virtual space 1 can be shared in the virtual space 2 newly created by the virtual workspace creating operation after the virtual workspace ending operation of the virtual space 1. Here, the virtual space 1 and the virtual space 2 are different virtual spaces corresponding to the same real environment E, and the real environment E includes at least the target OB corresponding to the target related information. ..

For example, at a construction site, a construction site, etc., different days and/or different workers may work on a certain target OB in the same real environment, and target related information (state, progress, etc.) about the target OB. Is preferably updated at any time.

In the present embodiment, the origin marker SP (SP1) is set in the virtual space 1, and in the virtual space 2, the position corresponding to the origin marker SP1 in the virtual space 1, that is, the origin marker SP1 in the real environment E (substantially) the same. The origin marker SP2 is set at the position (corresponding position). Accordingly, the origin marker SP1 of the virtual space 1 and the origin marker SP2 of the virtual space 2 can be collated (assumed to be at the same position) by the cooperation unit 50 (cooperation processing). That is, by displaying the position (coordinates) of the mark image M (which is associated with the target related information) in the virtual space 1 as it is in the virtual space 2, the (substantially) same position corresponding to a certain one real environment E. The mark image M can be displayed (shared) on the.

Here, although the user performs the operation of assigning the origin marker SP2, the information management device 10 is configured to be able to display the guide display TT with which the position of the origin marker SP1 in the virtual space 1 can be grasped. This guidance display TT is, for example, a display that allows the position of the origin marker SP1 in the real environment to be recognized. Specifically, the origin marker SP1 imaged (automatically or by user operation) when the origin marker SP1 is provided. The image data (list display) of the real environment with (display). Based on this guidance display TT, the user sets the origin marker SP2 in the virtual space 2 at a position corresponding to the origin marker SP1 in the virtual space 1.

Since the origins of the virtual three-dimensional coordinates of the virtual space 1 and the virtual space 2 coincide with each other by this cooperation processing, for example, the position (coordinates) of the target related information in the virtual space 1 from the origin marker SP1 is set to the origin marker SP2 in the virtual space 2. It can be reproduced as the position (coordinates) from. That is, the target related information can be output in the virtual space 2 (on the data handled in the virtual space 2), and the target related information can be shared.

That is, when there is the target related information (mark image M) registered in the internal virtual space ARS of the certain information management device 10, the origin marker SP2 is set at the position corresponding to the origin marker SP1 on the map 35 in the initial state. The mark image M can be displayed on the map 35 by performing the cooperation process by the cooperation unit 50. In addition, by setting another origin marker SP at a position corresponding to the origin marker SP on the environment data 36 in an external device or the like and performing the cooperation processing, the mark image M is further overlapped on the displayed map 35. It is possible to display the environmental data 36 (create collation data).

Further, by performing the cooperative processing of the origin marker SP1 of the internal virtual space ARS and the origin marker SP2 of the environment data 36, the environment data 36 can be displayed by being superimposed on the live view image LV of the information management device 10. Further, when the target related information is set on (only) the environment data 36, the target related information on the environment data 36 can be displayed and updated so as to be superimposed on the live view image LV of the information management device 10.

The cooperation process will be described with a specific example with reference to FIGS. 13 to 16. Here, the target related information set in the internal virtual space ARS1 (virtual space 1) of a certain information management device 10 is linked with another internal virtual space ARS2 (virtual space 2) of the same information management device 10 to create an internal virtual space. A case where the target related information is reproduced in the space ARS2 (for example, the target related information is displayed so as to be superimposed on the live view image LV of the information management device 10) will be exemplified. In the following description, the target related information is assumed to be associated with the mark image M. That is, the position (coordinates) of the mark image M is the holding position (coordinates) of the target related information.

13 and 14 are diagrams for explaining the work in the virtual space 1, for example, a case where work is performed on a wall scratch (target OB) in a certain room (real environment E). In both figures, the user operates the information management device 10 in a room where the target OB can be visually recognized, and in the virtual space 1 which can be recognized through the live view image LV, the mark image M is provided corresponding to the target OB. , Schematically shows how the target related information is associated with the mark image M.

Referring to FIG. 13A, when the user activates the information management device 10 (information management program 20) in the room (real environment E) as the work target, a virtual work space generation operation (environment information acquisition operation) is performed in response to the virtual work space generation operation. The information management device 10 activates the image pickup means 15, picks up the live view image LV, and displays it on the display means 13. Further, the position specifying means 23 resets the position of the own device (the above-mentioned reference position (initial position)) and acquires the position of the own device at that time (new reference position (initial position FS1). The information acquisition unit 293 uses the initial position FS1 as the initial origin (0, 0, 0) and corresponds to the space recognized through the live-view video LV (a virtual three-dimensional space (internal virtual space ARS1 (virtual space 1)). )) is generated, and the environment information acquisition unit 293 waits for reception of the origin marker SP1 in the internal virtual space ARS1.

Next, the user performs a reference point for collation operation (setting operation of the origin marker SP1) as shown in FIG. More specifically, in order to specify the start position in the internal virtual space ARS1, the user first specifies an arbitrary point P1 on the live view image LV (performs a point P1 input operation). It is desirable that the point P1 be, for example, a portion where the real environment E can be easily identified, and more desirably, for example, one point on a line segment that can be visually recognized on the live view image LV. In this example, the lower left corner of the door D is set as the point P1.

Based on the point P1 input operation, the environment information acquisition unit 293 regenerates the internal virtual space ARS1 with the position of the arbitrary point P1 designated by the user as the origin (0,0,0).

It should be noted that the position identifying means 23 has the information at the time of startup (initial position) even when the position of the arbitrary point P1 is set at the origin (0, 0, 0), and is its own device from the initial position. Can be traced. Therefore, by specifying the position of the mark image M (relative position from the origin (0,0,0)), the relative position of the target OB in the real space from the position of the own device can be grasped.

FIG. 14A is a state following FIG. 13B, and is a state in which the point P2 input operation is performed by the user. In order to specify the direction and size (space vector) of the internal virtual space ARS1, the user performs a point P2 input operation that specifies an arbitrary point P2 on the live view video LV. It is desirable that the point P2 be, for example, a portion where the real environment E can be easily identified. Further, for example, one point on the line segment that can be visually recognized as the same line segment as the point P1 on the live view image LV is more desirable. In this example, the lower right end of the door D is set as the point P2. The position (x1, y1, z1) of the point P2 is a relative position from the point P1 (0, 0, 0).

Based on the point P2 input operation, the environment information acquisition unit 293 acquires the origin marker SP1 and displays it on the live view image LV.

Further, the information processing device 10 (for example, the environment information acquisition unit 293) may be configured to store the guidance display TT so as to be callable. As shown in FIG. 14(A), the guidance display TT is information indicating which position of the arbitrary origin marker SP1 is set in the real environment E, and for example, the state where the origin marker SP1 is displayed. 3 is an image (screen shot, print screen) obtained by shooting the screen of the live view image LV of FIG. The guidance display TT is, for example, automatically photographed by a determination operation (button operation or the like) of the origin marker SP1 or by a user button operation or the like and stored in the storage unit 14. Further, before the reference point providing operation for collation is performed, if there is a stored guide display TT, the guide display TT is automatically called and displayed on the live view image LV (for example, thumbnail display). As a result, for example, when the origin marker SP2 is added in the virtual space 2, the origin marker is located at (substantially) the same position (corresponding position) as the origin marker SP1 in the real environment E based on the displayed guidance display TT. SP2 can be added, and the accuracy of cooperation processing (collation processing) can be improved.

FIG. 14B shows a state in which the mark giving position determining operation is performed by the user. The mark providing unit 21 displays the mark image M at the position designated by the user, and the position specifying unit 23 acquires and stores the position (coordinates (x2, y2, z2)) of the mark image M. The position (coordinates (x2, y2, z2)) of the mark image M may be stored in the storage unit 14 together with the target related information, or may be separately stored in the storage unit 14 as the position information (coordinates) of the mark image M. It may be stored.

The position of the mark image M is a relative position from the origin (0,0,0) in the internal virtual space ARS1. Further, the association unit 25 stores the target relevant information in the storage unit 14 in association with the mark image M when the association start operation is performed by the user. That is, the target related information is held in the folder corresponding to the mark image M having the position (coordinate) information in the internal virtual space ARS1. In other words, in the internal virtual space ARS1, the target related information having the position (coordinate) information in the internal virtual space ARS1 is registered.

Then, for example, a user different from the user who performed the work in the virtual space 1 refers to the target related information of the target OB of the same real environment E or updates the information, or determines the target OB of the same real environment E. When the target related information is newly associated with the target OB, the target related information of the virtual space 1 can be shared.

FIG. 15 and FIG. 16 are schematic diagrams showing a state of cooperation (sharing) of the target related information in the virtual space 2, for example, different from the user (previous user) who has worked on the internal virtual space ARS1. The user (the latter user) of (1) performs the work on the same target OB in the same real environment E. Note that the previous user and the subsequent user may be the same person, as long as they are processes in a virtual space 2 different from the virtual space 1.

Referring to FIG. 15A, a later user activates the information management device 10 (information management program 20) in the work target room (real environment E). The information management device 10 is, for example, the same device (tablet terminal) as that used by the previous user.

When the information management program 20 is activated, the information management device 10 activates the image capturing means 15 in response to a virtual work space generation operation (environmental information acquisition operation), captures the live view image LV, and displays it on the display means 13. To do. Further, the position specifying means 23 resets the position of the own device (the above-mentioned reference position (initial position)) and acquires the position of the own device at that time (a new reference position (initial position FS2). The information acquisition unit 293 uses the initial position FS2 as the initial origin (0, 0, 0) and corresponds to a space recognized through the live-view video LV (a virtual three-dimensional space (internal virtual space ARS2 (virtual space 2)). )) is generated, and the environment information acquisition unit 293 waits for reception of the origin marker SP2 in the internal virtual space ARS2.

Here, the initial position FS2 is a position at which the subsequent user has started the information management device 10, and even if the real environment E is the same, the position at which the information management device 10 is started by the previous user (see FIG. In many cases, the initial position FS1) does not completely match. In that case, the coordinates of the internal virtual space ARS2 with the initial position FS2 as the origin also do not match the coordinates of the internal virtual space ARS1.

As shown in FIG. 7B, the subsequent user performs a reference point for collation operation (setting operation of the origin marker SP2). More specifically, in order to specify the start position of the origin marker SP2 in the internal virtual space ARS2, the subsequent user first designates an arbitrary point P1 on the live view video LV (performs a point P1 input operation).

At this time, the information processing device 10 (for example, the environment information acquisition unit 293) calls one or a plurality of guide displays TT stored in the storage unit 14 and displays the guide display TT on the live view image LV (for example, thumbnail display). ..

The subsequent user refers to the guidance display TT and sets the point P1 on the live view image LV so that the origin marker SP2 is attached to the same position (corresponding position) as the origin marker SP1 set in the virtual space 1. Specify (tap operation, etc.). Specifically, the left lower end of the door D is set as the point P1 based on the guidance display TT.

Then, based on this point P1 input operation, the environment information acquisition unit 293 regenerates the internal virtual space ARS2 with the position of the point P1 designated by the user later as the origin (0, 0, 0). That is, the internal virtual space ARS2 is regenerated with the internal virtual space ARS1 and the origin (0,0,0) being matched (with the same position in the real environment E being the origin (0,0,0)). ..

FIG. 16A is a state following FIG. 15B. Subsequently, the subsequent user refers to the guidance display TT and performs a point P2 input operation so that the position is the same as the origin marker SP1 of the internal virtual space ARS1. As a result, the position of the point P2 is set (for example, the point P2 is set at the lower right end of the door D). Based on the point P2 input operation, the environment information acquisition unit 293 acquires the line segment defined by the points P1 and P2 as the origin marker SP2, and displays the origin marker SP2 on the live view image LV.

Note that the display mode (angle, length, etc.) of the origin markers SP1 and SP2 itself is different due to, for example, the position of the user when setting the origin markers SP1 and SP2 and the angle of view of the live view image LV. However, since the origin markers SP1 and SP2 are set as the space vectors of the virtual space 1 and the virtual space 2, respectively, and the origins (0, 0, 0) of the virtual space 1 and the virtual space 2 are the same, It can be said that there is no effect on the sharing of related information.

In this way, since the origin marker SP2 of the virtual space 2 can be attached to the same position (substantially) as the origin marker SP1 in the virtual space 1 by referring to the guidance display TT, the accuracy of the cooperative processing (matching processing) can be improved. Can be increased.

When the subsequent user sets the origin marker SP2 and performs a collation operation (for example, a confirmation operation of the origin marker SP2 or a collation start instruction operation, for example, an operation of a collation button (not shown)), the cooperation unit 50 (collation). The means 292) calls the mark image M and the target related information (stored by the previous user) stored in the storage means 14 of the information management device 10, and coordinates (x2, y2, z2) in the internal virtual space ARS2. The mark image M already stored is displayed at the position (the relative position of the origin marker SP2 from the point P1). That is, assuming that the position of the origin marker SP2 is the origin marker SP1 (matching with the origin marker SP1), the mark image M is displayed at the relative position (coordinates) from that position.

By doing so, it becomes possible to share the target related information for the same target OB even when the user and the date and time of the work are different. That is, even if the change in the progress or state of the work for the target OB is updated as the target related information at any time, the information can be unified.

In addition, for example, when a plurality of mark images M (and target related information) are added to the internal virtual space ARS1, all the mark images M (and target related information) that are added (stored) are displayed. To do. Therefore, when it is desired to extract (selectively) only a part of the mark images M (for example, to selectively display the mark images M given to only a part of the real environment E), It is preferable that the mark image M to be displayed be selectable by the user. Specifically, for example, the mark images M are grouped and stored in the storage unit 14, and at the time of display, the mark images M are selectively (for example, in a grouped state) an appropriate storage area (storage unit). It is advisable to have a configuration that can be prepared (stored) in part (14).

More specifically, when there are a plurality of real environments E to be worked, such as work in different buildings or work on different floors, the real environment E or the real environment E is further Corresponding data units (mark image M and target related information) may be stored in a grouped state for each predetermined area (hereinafter referred to as a work target area). In this case, for example, a group of work target areas (work target area group) can be set (designated) when the information management program 20 is started, and subsequent data units can be stored in the set work target group. Alternatively, the data units in the work target area group may be updated (added, deleted, moved, etc.) later.

When the data unit is stored for each work target area group, the information management device 10 (for example, the cooperation unit 50) displays the work target area group in a selectable manner at the start of the cooperation process. When the user selects a predetermined work target area, for example, the cooperation unit 50 prepares (in a predetermined storage area) all the data units included in the work target area group as reproducible data in the generated virtual space. To do. Then, when the origin marker SP is given by the user, the mark image M in the virtual space is set as a relative position from the origin (point P1) of the origin marker SP based on the coordinates (in the virtual space) of each of the mark images M. And target related information is displayed.

<Example of cooperation>
An example of cooperation in different virtual spaces 1 and 2 will be described with reference to FIGS. 17 and 18. 17A to 17D, as an example, an internal virtual space ARS1 (virtual space 1) of a certain information management apparatus 10A and an internal virtual space ARS2 (virtual space 2) of another information management apparatus 10B. This is an example of sharing target related information in. Since the information group (data unit) using the mark identification number MID as a key can be transmitted and received among a plurality of devices, the data unit stored in the information management device 10A can be separated by, for example, the communication means or the cooperation means 50. If the information management apparatus 10B of 1 is acquired, the data unit can be output in the information management apparatus 10B. In this example, the mark image M registered on the live view image LV of the information management apparatus 10A is displayed on the live view image LV of another information management apparatus 10B. Note that the target related information is associated with the mark image M (corresponding related information is stored in the folder of the mark image M) (the same applies in the following description).

As shown in FIG. 1A, a user who operates the information management device 10A in a certain real environment E (for example, the building A, N floor, room X) is on the live view image LV (internal virtual space ARS1). The origin marker SP1 is set, and mark images M1 and M2 are given to predetermined positions (coordinates (x1, y1, z1) and coordinates (x2, y2, z2)), respectively. The respective coordinates (x1, y1, z1) and coordinates (x2, y2, z2) of the mark images M1 and M2 are relative positions with respect to the origin (0, 0, 0) of the internal virtual space ARS1.

Further, in this example, as shown in FIG. 3B, in the information management device 10A, a plurality of mark images M are given to the plurality of work target areas 1 to 3. In such a case, in order to avoid displaying the unmarked mark image M, the mark image M can be grouped for each work target region and stored in the storage unit 14. In FIG. 1A, the user saves the mark images M1 and M2 in the work target area 1 group. Mark images M3 and M4 are stored in the work target area 2 group. Further, in the transmission/reception of the data unit, the information of the work target area group can also be transmitted/received.

As shown in FIG. 7C, the user who operates the information management device 10B in the same real environment E (for example, the building A, N floor, room X) first selects the corresponding work target area group. The information management apparatus 10B displays work target area groups 1 to 3 that can be selected when the information management program 20 is activated, and prompts the user to select them. The user selects the work target area 1 group. As a result, the information management apparatus 10B (cooperation unit 50) prepares the mark images M1 and M2 included in the work target area 1 group as shared data (in a predetermined storage area).

The user sets the origin marker SP2 on the live view image LV (internal virtual space ARS2). Here, the origin marker SP2 is set at the same (corresponding) position as the origin marker SP1 (the lower left end of the door D in this example) in the real environment E (work target area 1). The cooperation unit 50 collates the origin marker SP1 and the origin marker SP2, and specifies the positions of all the mark images M (here, the mark images M1 and M2) prepared as shared data.

As shown in FIG. 3D, the cooperation unit 50 has origins at the positions (coordinates (x1, y1, z1), coordinates (x2, y2, z2)) of the mark images M1 and M2, respectively. Mark images M1 and M2 are displayed at positions of coordinates (x1, y1, z1) and coordinates (x2, y2, z2) in the internal virtual space ARS2 assuming that they are at the origin (point P1) of the marker SP2.

As a result, the mark images M1 and M2 are displayed overlaid on the live view image LV of the information management apparatus 10B (in this example, the lower left corner of the door D), and it becomes possible to update the retained target related information. For example, it is possible to newly add the mark image M on the information management apparatus 10B side, and the mark image M added by being superimposed on the live view image LV on the information management apparatus 10A side by the processing of the cooperation unit 50 described above. Can be displayed.

The target related information held (associated) in the mark image M can be displayed by, for example, an operation (for example, tapping or clicking) on the mark image M, and addition/deletion/change of the target related information can be performed. It is possible (similar in the following examples).

17E to 17G, as an example, the internal virtual space ARS1 (virtual space 1) of a certain information management apparatus 10 and the map 35 (virtual space 2) of the (for example, the same) information management apparatus 10 are shown. This is an example of sharing target related information in.

In this example, the mark images M1 and M2 registered on the live view image LV of the information management device 10 are displayed on the map 35.

As shown in (E) of the figure, the user operating the information management device 10 sets the origin marker SP1 on the live view image LV (internal virtual space ARS1) and sets it at a predetermined position (coordinates (x1, y1, z1). ) And the coordinates (x2, y2, z2)) are provided with the mark images M1 and M2, respectively. The coordinates (x1, y1, z1) and the coordinates (x2, y2, z2) of the mark images M1 and M2 are relative positions based on the origin (0, 0, 0) of the internal virtual space ARS1.

When displaying the map 35, the user sets the origin marker SP2 on the map 35 (virtual space 2) as shown in FIG. Here, the origin marker SP2 is set at the same (corresponding) position as the origin marker SP1 in the real environment E (in this example, the end portion of the floor F on the inner side of the room).

As shown in (G) of the figure, the cooperation means 50 collates the origin marker SP1 and the origin marker SP2, and positions of the mark images M1 and M2 (coordinates (x1, y1, z1), coordinates (x2, y2, z2). )), the mark image M is displayed at the position of the coordinates (x, y) on the map 35, assuming that the origin is the origin marker SP2. As a result, the mark images M1 and M2 are displayed on the map 35, and it becomes possible to update the retained target related information.

For example, it is possible to newly add the mark image M on the map 35 side, and the mark image M added by being overlapped with the live view image LV on the information management device 10A side by the processing of the above-mentioned cooperation means 50 is displayed. be able to. In addition, for example, when the mark image M added on the side of the map 35 (two-dimensional virtual space 2) is superimposed on the live view image LV and displayed in the internal virtual space ARS (the virtual space 2 of three time periods), the mark image of the map 35 is displayed. The dimension of M is insufficient (for example, the z coordinate is insufficient). In such a case, the cooperation unit 50 gives the missing coordinates (for example, with a fixed value or a variable). For example, when the mark image M is given on the map 35, its coordinates are (x3, y3), but the linking means 50 adds the z coordinate and sets the coordinates of the mark image M to, for example, (x3, y3, 100) and the like are stored in the storage means 14. The user may be able to change the coordinates of the missing dimension. In the internal virtual space ARS (on the live view video LV), the mark image M is displayed based on the three-dimensional coordinates after the addition of the z coordinate.

18A to 18C, as an example, a system (for example, a system) in the internal virtual space ARS (virtual space 1) of the information management device 10 and another device (for example, the terminal device 250 shown in FIG. 10). , BIM (virtual space 2)), the target related information is shared by environment data (BIM data) 36. Since the information group (data unit) using the mark identification number MID as a key can be transmitted and received between a plurality of devices, for example, the data unit stored in the information management device 10 via the communication means or the cooperation means 50 is used as a terminal. When the device 250 acquires the data unit, the terminal device 250 can output the data unit.

In this example, the mark image M registered on the live view image LV of the information management device 10 is displayed on the environment data 36 handled by the terminal device 250 (system such as BIM).

As shown in (A) of the figure, the user operating the information management device 10 sets the origin marker SP1 on the live view image LV (internal virtual space ARS), and sets a predetermined position (coordinates (x, y, z )) is provided with the mark image M. The coordinates (x, y, z) of the mark image M are relative positions based on the origin (0, 0, 0) of the internal virtual space ARS1.

On the other hand, as shown in FIG. 3B, the user operating the terminal device 250 sets the origin marker SP2 in the environment data 36 on a system such as BIM. Here, the origin marker SP2 is set at the same (corresponding) position as the origin marker SP1 in the real environment E (lower left end of the window W in this example).

As shown in (C) of the figure, the linking means 50 collates the origin marker SP1 and the origin marker SP2, and the origin of the position (coordinates (x, y, z)) of the mark image M is at the origin marker SP2. , The mark image M is displayed at the position of the coordinates (x, y, z) in the environment data 36. As a result, the mark image M is displayed over the environmental data 36 (in this example, the lower left corner of the window W), and it is possible to update the retained target related information. For example, it is possible to newly add the mark image M on the information management apparatus 10B side, and the mark image M added by being superimposed on the live view image LV on the information management apparatus 10A side by the processing of the cooperation unit 50 described above. Can be displayed.

18D to 18F, as an example, the environment data (BIM data) 36 handled by the system (for example, BIM (virtual space 1)) in the terminal device 250 and the map 35 (of the information management device 10 ( This is an example of sharing target related information in the virtual space 1).

In this example, the mark image M is newly registered in the environment data 36 (for example, CAD data) generated on the system (for example, BIM) of the terminal device 250, and the mark image M is displayed on the map 35 of the information management device 10. The case is shown.

As shown in FIG. 3D, the user operating the system of the terminal device 250 sets the origin marker SP1 on the environment data (for example, point cloud data) 36 on the system and sets the predetermined position (coordinates). Mark images M1 and M2 are given to (x1, y1, z1) and coordinates (x2, y2, z2)), respectively. The coordinates (x1, y1, z1) and the coordinates (x2, y2, z2) of the mark images M1 and M2 are relative positions based on the origin (0, 0, 0) of the origin marker SP1 set in the environment data 36. is there.

On the other hand, the user who operates the information management device 10 as shown in FIG. 7E sets the origin marker SP2 on the map 35, for example. Here, the origin marker SP2 is set at the same (corresponding) position as the origin marker SP1 in the real environment E (lower left end of the wall surface WL in this example).

As shown in (F) of the figure, the cooperation unit 50 collates the origin marker SP1 and the origin marker SP2, and positions of the mark images M1 and M2 (coordinates (x1, y1, z1), coordinates (x2, y2, z2). )) is set to be the origin marker SP2 and the mark images M1 and M2 are displayed at the positions of the coordinates (x1, y1, z1) and the coordinates (x2, y2, z2) in the map 35. As a result, the mark images M1 and M2 are displayed in an overlapping manner on the map 35 (the lower left corner of the wall surface WL in this example), and the target related information that is held can be updated. In this way, the mark image M (target related information) newly set by the system (for example, BIM, CIM, etc.) such as another device (for example, the terminal device 250) is displayed in the virtual space 1 (map 35) of the information management device 10. It can also be displayed on the screen or on the live view image LV).

The origin marker SP is not limited to the one set by the user's operation, and point cloud data or “characteristic points” of the real environment E (a part of the environment data 36) may be used as the origin marker SP. For example, in the environmental data 36, the point group data corresponding to a door or a window whose position can be specified, or “characteristic points” corresponding to these points are set as the origin marker SP. Then, the environment data 36 in which the mark image M is set is read into the information management device 10, and is collated with the origin marker SP while displaying the live view image LV (the conformity of the shape is detected). When the shape that matches the origin marker SP (for example, the point cloud data corresponding to the door) is detected, the mark image M is displayed on the live view image LV.

FIG. 19 is a schematic diagram showing another example of the display in the live view image LV of the information management device 10. FIG. 19A is a schematic diagram of the real environment E, FIG. 19B, and FIG. Is a schematic view of the display on the live view video LV, FIG. 6D is a schematic view showing the environmental data 36, and FIG. 7E is a schematic view of the display on the live view video LV.

Since the angle of view of the live view image LV of the information management device 10 is limited, it may not be possible to display all of the plurality of mark images M in the work target area group. Specifically, as shown in FIG. 19A, it is assumed that the objects OB1 to OB3 are in the real environment E (work target area) and the mark images M1 to M3 are given to them. When the work target area is wide, only the mark image M1 is displayed at a certain angle of view of the live view image LV of the information management device 10 as shown in FIG. 7B, and the mark images M2 and M3 do not fit within the angle of view. There are cases. In such a case, for example, a means (for example, a display list guide DG) for suggesting the existence of the mark image M may be provided. An example of the display list guide DG may be a mark (arrow in the figure) indicating the position of the mark image M that is not displayed, as shown in the figure (B), or the figure (C). A list of the mark images M that are hidden as shown in (is desirable if the position of the mark image M is also suggested by the list display position).

Further, in FIGS. 2D and 2E, the mark image M (FIG. 2D) set on the environment data 36 (virtual space 1) is displayed as a live view image LV (virtual) of the information management device 10. An example is shown in which the space 2 is linked to FIG. As described above, the origin marker SP may be a "characteristic point" such as point cloud data, instead of a marker of a line segment. In this example, for example, the wall WL on the environment data 36 is used as the origin marker SP, as indicated by the broken line in FIG.

The cooperation unit 50 collates the origin marker SP (“characteristic point” of the point cloud data) in the virtual space 2. Specifically, a shape that matches the point cloud data is detected. When the origin marker SP (in this case, the wall WL) can be collated (the wall WL is detected on the live view image LV), the mark image M1 is displayed on the live view image LV. At this time, if the position of the mark image M1 displayed as a relative position from the origin marker SP (shown by a dashed line in FIG. 7E) is not within the angle of view of the live view image LV, for example, the origin marker SP The mark image M1 may be displayed at the origin X as point cloud data on the wall WL which is SP.

When a newly added mark image M in a certain virtual space 1 is displayed in another virtual space 2, the display mode of the mark image M is changed (for example, blinking display is performed) and the newly added mark image M is added. The mark image M may be configured to be suggested.

In addition, the cooperation unit 50 may perform scale reduction or rotation (in the coordinate system) of the virtual space as necessary in the cooperation processing.

Further, the origin marker SP may be any means that specifies the starting point (origin) of the virtual space and the three of the direction and the size. Therefore, an image (for example, a physical marker, a two-dimensional barcode, a photograph, etc.) may be used as the origin marker SP instead of the above-mentioned origin marker SP. In this case, for example, the origin marker SP may be one that defines the origin, direction, and size in one image (for example, the origin marker SP of 1 m square is set in front of a door having a width of 1 m). Further, the origin marker SP may be one in which the origin and the direction are designated by one image and the size is designated by a point, or the arbitrary points P1 and P2 designated by the user are respectively replaced by images. It may be one.

As described above, according to the present embodiment, when a wide-angle video (live view video LV) including the surrounding environment is displayed for the target OB existing in a certain space, the target OB existing in the space is displayed. Can be displayed in the mark image M, and the position of the target OB in the space can be automatically stored.

Further, the user can add the mark image M corresponding to the target OB simply by touching (tapping or the like) the position of the target OB while watching the live view image LV. Further, by selecting the mark image M, the data that can be associated with the mark image M is displayed in a list, so that it is possible to easily associate the mark image M (target OB) with the target image by a visual operation while on site. Information (including position information (relative position from the reference position) and shape) can be added.

In addition, by associating the target position information of the target OB and the attributes (“desk”, “chair”, etc.) of the target OB from the “characteristic points” of the target OB as target related information, the mark image M is Even in the information management device 10, even if the time and date are different, it is possible to display the same data on the display means 13 by going to the same space and executing the information management program 20.

In the conventional technique, for example, when the target OB is a scratch on a wall, even if the photographer takes a close-up photograph showing the state of the scratch OB, if the number of shots increases or the time elapses from the shooting. It is likely that the photo is unknown or unknown. In addition, in order to identify the location and convey the situation of the site to a third party, from the close-up of the scratch OB, while taking a retreat, step by step photographs of a wide angle of view including the surrounding environment are taken. Both detailed and detailed information is needed. However, it is necessary to manually associate (associate) them with each other later, and since the number of sheets becomes huge, it takes time to organize them, which causes confusion and becomes complicated.

Further, in order to manually organize these, it is necessary to leave a plurality of types of documents such as dividing into folders and writing in drawings, which is very cumbersome, and there is a problem that it takes time to work.

In addition, at the construction/construction site or the site of inspection/maintenance in facility management, it is necessary to leave not only the information of the target OB itself, but also the surrounding environment information related to the construction or work, or not directly related to it. As for the storage method, a method with increased flexibility in various data formats such as image data and document data has been desired.

According to the present embodiment, by performing a touch operation (tapping or the like) on the position of the target OB while visually checking the live view image LV of the space including the target OB, the mark image M is superimposed on the live view image LV in real time. Can be displayed. In addition, if the same information management program 20 is installed in the device (information management device 10), the mark image M is reproduced even when another device displays the live view image LV in the same space. ..

In addition, the mark image M automatically acquires and stores the positional relationship in the space only by touching the live view image VL.

In this way, the mark image M is easily attached to the target OB in a certain space by a visual operation, and the information of the target OB (mark image M) is shared among a plurality of users (a plurality of information management devices 10). Can be shared at.

Further, by directly operating (touching, tapping, etc.) the mark image M, the information (target related information) related to the mark image M can be easily linked. In other words, it is possible to considerably advance the association between the mark image M and the target related information on the spot. Alternatively, since the target-related information can be associated with each mark image M while displaying the positional relationship of the mark image M in the space on the map 35 in a bird's-eye view, even after leaving the site, Information can be linked and organized easily, and the work load can be greatly reduced.

Further, the information related to the target OB (target related information) can be managed (transmitted/received, downloaded, etc.) for each target OB as package data, so that it becomes easy for a plurality of people to share the data. Furthermore, since it can be embedded in documents etc. as package data, the efficiency of creating documents such as reports, quotations and purchase orders can be improved, and document mistakes such as adding wrong photographs can be avoided. ..

Furthermore, even when the virtual spaces are different, the handling related information can be shared. Specifically, for example, the target related information (images, memos, etc.) registered on the live view video LV in the information management device 10 (information management program 20) is transferred to, for example, an external BIM (Building Information Modeling) by the cooperation unit 50. In the information management device 10, target related information reproduced (shared, referenced) on data (CAD data, point cloud data, etc.) used in CIM (Construction Information Modeling) or the like or registered in external BIM/CIM or the like is used. Can be reproduced (shared, referenced).

That is, it is possible to bidirectionally share the target related information on the AR space of the information management device 10 executed in a real space such as a construction site or a construction site and on the data such as BIM/CIM in a remote office. It will be possible and will make a great contribution to Facility Management.

A part or all of the devices (hardware) configuring the information management device 10 and the information management system 100 described above may be replaced with software that realizes a function equivalent to the device (hardware).

The information management device, information management system, information management method, and information management program of the present invention are not limited to the above examples.

For example, in the above-described embodiment, when there are a plurality of target OBs, their positions are described as an example in the same room on the same floor, but the present invention is not limited to this. For example, the plurality of target OBs may be dispersed in rooms on different floors or rooms on different floors.

Further, for example, the plurality of objects OB1 and OB2 may be associated with each other so that a relationship (for example, a parent-child relationship) occurs. Specifically, since the position of the mark image M is specified by the position specifying means 23, wide-angle shooting is performed for the plurality of objects OB1 and OB2 so that the mark images M1 and M2 are included in the angle of view. At the time of wide-angle shooting of the image M2, the mark image M1 is shot (the positions of the objects OB1 and OB2 are specified) so as to fit within the angle of view. Thereby, the mark image M1 and the mark image M2 may be associated with each other. For example, in this case, the mark image M1 and the target-related information associated with it are child data, and the mark image M2 (the position of which is specified) photographed so as to include this and the target-related information associated with it May be associated so that is the parent's data. When the number of mark images M and the amount of target related information thereof become enormous, the parent-child relationship of each mark image M is recorded, so that data can be efficiently managed.

Further, the target OB may be recognized/specified by the object shape recognition means.

Further, the mark image M may be selectable and changeable. For example, the type of the mark image M may be different depending on the attribute of the associated target related information.

The matching means 292 may match the environment data 36 of the two-dimensional image with a part of the map 35 displayed as the three-dimensional image. For example, the collation data 37 may be created by superimposing it on the floor surface portion of the map 35 created for the target OB in the room and superimposing a two-dimensional image such as a design drawing of the room.

Further, in the above embodiment, the case where the information management device 10 is a tablet terminal type device has been described as an example, but the invention is not limited to this, and for example, a head mounted display type (or eyeglass type that is wirelessly worn on the head). ) Apparatus (spectacle-type AR glasses, a device for mixed reality (MR) platform (for example, a head-mounted display-type device that realizes holographic computing), or the like).

Further, although the terminal movement information in the above embodiment is described as the relative position information of the information management device 10 from the initial position, in addition to this, for example, the movement progress information of the terminal using motion tracking or the like ( The movement log and the action log) may be stored together in the storage unit 14 and may be reproducible. For example, when the information management device 10 is used at the site of equipment inspection or maintenance, the action log can be reproduced to trace/verify the inspection/maintenance execution order of the predecessor or expert. The efficiency of subsequent work can be improved.

Further, the information management device, the information management system, the information management method, and the information management program of the present invention are not limited to the above-mentioned embodiments, and various modifications can be made without departing from the gist of the present invention. Of course.

The information management device, the information management system, the information management method, and the information management program of the present invention can be used for a technique of adding information to an object existing in a real space and storing it as data.

10 information management device 11 control means 12 input means 13 display means (display)
14 storage means 15 imaging means 16 information communication means 17 state acquisition means 18 voice input means (microphone)
19 speaker 20 information management means (information management program)
21 mark giving means 22 document creating means 23 position specifying means 25 associating means 27 virtual visual processing means (virtual audiovisual processing means), AR processing means 29 overall view display means 30 related information database 35 map 50 cooperation means 250 terminal device ARS internal virtual Space E Real environment TT Guidance display 100 Information management system 101 Tablet terminal 110 Network 121 Physical button 122 Marking operation button 123 Shooting button 126 Operation button 131 Physical button 151 Lens 153 Image sensor 155 Image processor 210 Network 220 Server device 221, 231 Storage means 230 Cloud server device 291 Mapping means 292 Collating means 293 Environmental information acquisition means

Claims (16)

Imaging means capable of acquiring a live view image of a target area including a real target,
Display means capable of displaying the live view image,
Storage means,
Information management means,
A portable information management device having:
The information management means,
Position specifying means for specifying the position of the target,
Mark providing means for displaying on the display means a mark image corresponding to the target in an overlapping manner with the live view image,
Association information for storing information related to the target in the storage means in association with the mark image.
An information management device characterized by the above. The associating unit associates the position of the target specified by the position specifying unit and the information with the mark image,
The information management device according to claim 1, characterized in that. The mark giving means can give a plurality of the mark images.
The information management device according to claim 1 or 2, wherein There is a holistic display means for displaying the position of at least one of the mark images as a whole on the display means.
The information management apparatus according to any one of claims 1 to 3, wherein The information is at least one of image pickup data, character data and voice data,
The information management device according to any one of claims 1 to 4, characterized in that. At least information communication means capable of transmitting and receiving the information,
The information management device according to any one of claims 1 to 5, characterized in that. In a different space, it has a cooperation means for sharing the information,
The information management device according to any one of claims 1 to 6, wherein: An information management apparatus according to any one of claims 1 to 7,
And another device capable of managing the information,
An information management system characterized in that The other device is a server device,
At least transmitting and receiving the information between the information management device and the server device via a network,
The information management system according to claim 8, wherein: The other device is another terminal device,
At least transmitting and receiving the information between the information management device and the other terminal device via a network,
The information management system according to claim 8, wherein: An information management method for providing predetermined information to an existing object and storing the data as data that can be managed by an information management device,
An image capturing step of acquiring a live view image of a target region including a real target by an image capturing means;
A display step of displaying the live view image on display means,
A position specifying step of specifying the position of the target in a predetermined space,
A mark giving step of displaying a mark image corresponding to the target on the live view image on the display means,
An associating step of associating the information related to the target with the mark image and storing it in a storage means.
An information management method characterized by the following. An information communication step of transmitting and receiving the information between the information management device and another device;
The information management method according to claim 11, wherein: An imaging step of acquiring a live view image of a target area including a real target by an imaging means;
A display step of displaying the live view image on display means,
A position specifying step of specifying the position of the target in a predetermined space,
A mark providing step of displaying a mark image corresponding to the target on the live view image on the display means,
An associating step of storing the information related to the target in the storage means in association with the mark image;
An information management program that causes a computer to execute. The program according to claim 13,
An information communication step of transmitting and receiving the information between a plurality of information management devices,
An information management program characterized by that. The space is at least one of a virtual space recognized through the live view video and a space different from the virtual space,
The information management device according to claim 7, characterized in that. The information is associated with position coordinates in the virtual space,
The information management apparatus according to any one of claims 1 to 7 and 15, characterized in that.