JP7228311B1 - Information processing system, information processing method, program, server - Google Patents

Abstract

The present invention provides an information processing system, an information processing method, a program, and a server for linking image data to a drawing showing at least a portion of a structure. The present invention includes: a start point and end point setting unit that sets one or more sets of a start point position and an end point position on a drawing showing at least a portion of a structure; an association unit that associates one or more image data with the set; An information processing system comprising: a position linking unit that links image data corresponding to a predetermined position between a starting point position and an end point position corresponding to an associated set. [Selection diagram] Figure 1

Description

The present invention relates to an information processing system, an information processing method, a program, and a server.

In recent years, flying vehicles such as drones and unmanned aerial vehicles (UAVs) (hereinafter collectively referred to as “unmanned flying vehicles”) have begun to be used in industry. Under such circumstances, Japanese Patent Application Laid-Open No. 2002-200002 discloses a system in which an aircraft sequentially photographs an object to be photographed at a plurality of preset waypoints.

JP 2014-089160 A

Here, in the conventional system, a plurality of pieces of image data are organized so that the positions on the map can be visually recognized, for example, by associating the position information of the photographed image data with the position information of the map.

However, when inspecting the outer walls of buildings, for example, there is a need to organize image data captured on drawings that show at least a part of a structure, such as exterior drawings and blueprints. there was a need to

The present invention has been made in view of such a background, and in particular, provides an information processing system and information processing method, a program, and a server for linking image data to a drawing showing at least a part of a structure. aim.

The main invention of the present invention for solving the above problems is a start point and end point setting unit that sets one or more sets of start point positions and end point positions on a drawing showing at least a part of a structure, and one or more image data are set and a position linking unit that links image data corresponding to predetermined positions between a start point position and an end point position corresponding to the associated set.

According to the present invention, it is possible to provide an information processing system, an information processing method, a program, and a server for linking image data to a drawing showing at least part of a structure.

1 is a diagram showing the overall configuration of an information processing system according to an embodiment of the present invention; FIG. It is a figure which shows the system configuration|structure of the information processing system concerning embodiment of this invention. 3 is a block diagram showing the hardware configuration of the server of FIG. 2; FIG. 3 is a block diagram showing the hardware configuration of the terminal in FIG. 2; FIG. FIG. 3 is a block diagram showing the hardware configuration of the flying object of FIG. 2; FIG. 3 is a block diagram showing the functions of the server and terminals in FIG. 2; FIG. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. It is a figure of an example explaining processing of the information processing system concerning an embodiment of the invention. 1 is a flowchart of an information processing system according to an embodiment of the present invention;

The contents of the embodiments of the present invention are listed and explained. An information processing system, an information processing method, a program, and a server according to embodiments of the present invention have the following configurations.
[Item 1]
a start point/end point setting unit that sets one or more sets of start point positions and end point positions on a drawing showing at least a portion of a structure;
an associating unit that associates one or more pieces of image data with the set;
A position linking unit that links image data corresponding to a predetermined position between the start point position and the end point position corresponding to the associated set,
An information processing system characterized by:
[Item 2]
The start point and end point setting unit sets the set of the start point position and the end point position based on a user's setting operation on the drawing.
The information processing system according to item 1, characterized by:
[Item 3]
The start point and end point setting unit analyzes the drawing and sets a set of the start point position and the end point position for at least one surface of the structure.
The information processing system according to item 1, characterized by:
[Item 4]
The association unit associates the image data selected based on a selection operation by a user with the set.
The information processing system according to any one of items 1 to 3, characterized by:
[Item 5]
the associating unit associates the image data with the set based on at least one of time information and location information regarding the image data;
The information processing system according to any one of items 1 to 3, characterized by:
[Item 6]
the position linking unit calculates the predetermined position based on at least one of time information and position information regarding the image data, and links the image data corresponding to the predetermined position;
The information processing system according to any one of items 1 to 3, characterized by:
[Item 7]
The drawing is a drawing based on drawing information that does not include latitude and longitude information,
The information processing system according to any one of items 1 to 3, characterized by:
[Item 8]
A step of setting one or more sets of starting point positions and ending point positions on a drawing showing at least a part of a structure by a starting point and end point setting unit;
associating one or more image data with the set by an associator;
a step of linking image data corresponding to a predetermined position between a start point position and an end point position corresponding to the associated set by a position linking unit;
An information processing method characterized by:
[Item 9]
A step of setting one or more sets of starting point positions and ending point positions on a drawing showing at least a part of a structure by a starting point and end point setting unit;
associating one or more image data with the set by an associator;
causing the information processing system to perform a step of linking image data corresponding to a predetermined position between the start point position and the end point position corresponding to the associated set by the position linking unit;
A program characterized by
[Item 10]
a start point/end point setting unit that sets one or more sets of start point positions and end point positions on a drawing showing at least a portion of a structure;
an associating unit that associates one or more pieces of image data with the set;
A position linking unit that links image data corresponding to a predetermined position between the start point position and the end point position corresponding to the associated set,
A server characterized by:

<Details of Embodiment>
Hereinafter, embodiments of an information processing system, an information processing method, a program, and a server according to embodiments of the present invention will be described. In the accompanying drawings, the same or similar elements are denoted by the same or similar reference numerals and names, and duplicate descriptions of the same or similar elements may be omitted in the description of each embodiment. Also, the features shown in each embodiment can be applied to other embodiments as long as they are not mutually contradictory.

<Overall composition>
As shown in FIG. 1, the information processing system according to the present embodiment captures an image of at least a portion of a structure (for example, a wall surface) using an unmanned flying vehicle 4, and captures a drawing showing at least a portion of the structure. It manages image data in association with each other.

<System configuration>
As shown in FIG. 2, the information processing system in this embodiment has a server 1, a terminal 2, and an unmanned flying vehicle 4. The server 1, the terminal 2, and the unmanned air vehicle 4 may be communicably connected to each other via the network NW. Note that the illustrated configuration is only an example, and is not limited to this. For example, the unmanned air vehicle 4 may not be connected to the network NW. In that case, the operation of the unmanned flying object 4 is performed by a user-operated transmitter (so-called propo), or the image data acquired by the camera of the unmanned flying object 4 is connected to an auxiliary storage device (for example, A memory card such as an SD card, a USB memory, etc.) may be read by the user from the auxiliary storage device and stored in the server 1 or the terminal 2 afterwards. The unmanned air vehicle 4 may be connected to the network NW only for one of the purposes of storing the data.

<Server 1>
FIG. 3 is a diagram showing the hardware configuration of the server 1. As shown in FIG. Note that the illustrated configuration is an example, and other configurations may be employed.

As illustrated, a server 1 is connected to a terminal 2 and an unmanned air vehicle 4 and forms part of this system. The server 1 may be a general-purpose computer such as a workstation or personal computer, or may be logically implemented by cloud computing.

The server 1 includes at least a processor 10 , a memory 11 , a storage 12 , a transmission/reception section 13 , an input/output section 14 and the like, which are electrically connected to each other through a bus 15 .

The processor 10 is an arithmetic device that controls the overall operation of the server 1, controls transmission and reception of data between elements, executes applications, and performs information processing necessary for authentication processing. For example, the processor 10 is a CPU (Central Processing Unit) and/or a GPU (Graphics Processing Unit), and executes a program for this system stored in the storage 12 and developed in the memory 11 to perform each information process. .

The memory 11 includes a main memory composed of a volatile memory device such as a DRAM (Dynamic Random Access Memory), and an auxiliary memory composed of a non-volatile memory device such as a flash memory or a HDD (Hard Disc Drive). . The memory 11 is used as a work area or the like for the processor 10, and stores a BIOS (Basic Input/Output System) executed when the server 1 is started, various setting information, and the like.

The storage 12 stores various programs such as application programs. A database storing data used for each process may be constructed in the storage 12 . Further, each storage unit, which will be described later, may be provided in a part of the storage area.

The transmission/reception unit 13 connects the server 1 to the network NW. Note that the transmission/reception unit 13 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

The input/output unit 14 is an information input device such as a keyboard and a mouse, and an output device such as a display.

A bus 15 is commonly connected to the above elements and transmits, for example, address signals, data signals and various control signals.

<Terminal 2>
The terminal 2 shown in FIG. 4 also includes a processor 20 , a memory 21 , a storage 22 , a transmission/reception section 23 , an input/output section 24 and the like, which are electrically connected to each other through a bus 25 . Since the function of each element can be configured in the same manner as the server 1 described above, detailed description of each element will be omitted.

<Unmanned Aerial Vehicle 4>
FIG. 5 is a block diagram showing the hardware configuration of the unmanned flying object 4. As shown in FIG. Flight controller 41 may have one or more processors, such as programmable processors (eg, central processing units (CPUs)).

The flight controller 41 also has a memory 411 and can access the memory. Memory 411 stores logic, code, and/or program instructions executable by the flight controller to perform one or more steps. The flight controller 41 may also include sensors 412 such as an inertial sensor (acceleration sensor, gyro sensor), GPS sensor, proximity sensor (for example, rider).

Memory 411 may include, for example, separable media or external storage devices such as SD cards and random access memory (RAM). Data acquired from cameras/sensors 42 may be communicated directly to memory 411 and stored. For example, still image/moving image data captured by a camera or the like may be recorded in an internal memory or an external memory. 2 may be recorded. A camera 42 is installed on the unmanned air vehicle 4 via a gimbal 43 .

Flight controller 41 includes a control module (not shown) configured to control the state of unmanned air vehicle 4 . For example, the control module adjusts the spatial orientation, velocity, and/or acceleration of unmanned air vehicle 4, which has six degrees of freedom (translational motions x, y, and z, and rotational motions _θx , _θy , and _θz ). Therefore, the propulsion mechanism (motor 45, etc.) of the unmanned air vehicle 4 is controlled via an ESC 44 (Electric Speed Controller). A motor 45 powered by a battery 48 rotates a propeller 46 to generate lift of the unmanned air vehicle 4 . The control module can control one or more of the states of the mount, sensors.

Flight controller 41 is configured to transmit and/or receive data from one or more external devices (e.g., transceiver (propo) 49, terminal, display, or other remote controller). It is possible to communicate with the unit 47 . Transceiver 49 may use any suitable means of communication, such as wired or wireless communication.

For example, the transceiver 47 utilizes one or more of a local area network (LAN), a wide area network (WAN), infrared, wireless, WiFi, point-to-point (P2P) networks, telecommunications networks, cloud communications, and the like. can do.

The transmitting/receiving unit 47 transmits and/or receives one or more of data obtained by the sensors 42, processing results generated by the flight controller 41, predetermined control data, user commands from a terminal or a remote controller, and the like. be able to.

Sensors 42 according to this embodiment may include inertial sensors (acceleration sensors, gyro sensors), GPS sensors, proximity sensors (eg lidar), or vision/image sensors (eg cameras).

<Server functions>
FIG. 6 is a block diagram illustrating functions implemented in the server 1 and terminal 2. As shown in FIG. In the present embodiment, the server 1 includes a communication unit 110, a start/end point setting unit 120, an association unit 130, an image linking unit 140, an information output unit 150, a drawing-related data storage unit 160, an aircraft-related information storage unit 170. It has Various functional units are illustrated as functional units in the processor 10 of the server 1, but some or all of the various functional units may be the processor 10 of the server 1, the processor 20 of the terminal 2, or the controller of the unmanned air vehicle 4. 41 may be implemented in any configuration of the processor 10 or the processor 20 and the controller 41 according to the capabilities of the controller 41.

The communication unit 110 communicates with the terminal 2 and the unmanned air vehicle 4 via the network NW. The communication unit 110 also functions as a reception unit that receives various requests, data, and the like from the terminal 2, the unmanned air vehicle 4, and the like.

The start point/end point setting unit 120 sets one or more sets of start point positions and end point positions on a drawing showing at least a portion of a structure (for example, a wall surface).

As a specific example, as exemplified in FIG. 24, and the first axis (hereinafter also referred to as "X-axis") and the second axis (hereinafter also referred to as "Y-axis") orthogonal to the first axis with respect to the drawing are displayed. set. For example, the start point and end point setting unit 120 may automatically assign the first axis and the second axis according to the vertical and horizontal directions of the read drawing data, or may set the first axis according to the user's operation, and set the first axis. For example, a second axis orthogonal to the axis may be set.

Then, as illustrated in FIGS. 8 and 9, the start point and end point setting unit 120 performs a position specifying operation (setting operation) for specifying the start point position and the end point position on the drawing by the user performed on the terminal 2. may set a set of start and end positions for the drawing. Alternatively, the start point and end point setting unit 120 performs image analysis using a learning model that has learned in advance the drawing of the drawing data, and as illustrated in FIG. A set of positions may be set automatically. Then, the start point/end point setting unit 120 associates the set of the start point position and the end point position with the drawing data and stores them in the drawing-related data storage unit 160 . In addition, as exemplified in FIG. 8, the display may be made different so that the start point position and the end point position can be distinguished (for example, the shape, color, size, etc. of the point), or the set of the start point position and the end point position may be different. An arrow from the start point to the end point may be displayed for easy recognition.

Further, as illustrated in FIG. 9, the start point/end point setting unit 120 may set a plurality of sets of the start point position and the end point position on the drawing. 9, the start point/end point setting unit 120 sets the direction from the start point position to the end point position along at least one of the first axis and the second axis (for example, the direction along the first axis). is the horizontal direction in the drawing, and the second axis is the vertical direction in the vertical direction, etc.). Simplify. Furthermore, the position where the start point position and the end point position can be set by the start point and end point setting unit 120 may be any position on the drawing. The position of the intersection of both axes (so-called XY coordinates), or the position of the central point of a rectangular area partitioned into a grid may be specified.

In embodiments of the present invention, structures may be, for example, buildings such as houses and buildings, dams, embankments, bridges, power generation facilities, power transmission facilities, tanks, vehicles, buildings, and the like.

In the embodiment of the present invention, the drawing is a drawing showing at least part of the structure, and may be, for example, a plan view such as an external view of the structure from one direction or a design drawing. , 3D model data (for example, BIM data, CAD data, etc.) of a structure may be an image that is stored from one direction, or an image of an actual structure may be captured. The drawing may also be a drawing showing the appearance of the side or top surface of the structure in particular. Furthermore, the superiority of the present invention becomes particularly clear when the drawing data is drawing data that does not include position information (for example, latitude/longitude/height information).

The associating unit 130 associates one or more pieces of image data with the set of the start point position and the end point position set by the start point end point setting unit 120 . As a specific example, as illustrated in FIG. 10, a predetermined set is selected by the user's selection operation (in the example, the selected arrow is dotted), and as illustrated in FIG. , one or more pieces of image data are associated with the selection set by selecting image data to be associated with the selection set by a user's selection operation. Alternatively, when a set of the start point position and the end point position is set along the flight path of the unmanned air vehicle 4 by user operation, the time information or position information included in each image data is compared with each other, and each set Image data according to the relative distance (especially the length on the drawing) between the start point position and the end point position in , may be associated with each set in order of time or position. As a more specific example, for example, when two sets of start point position and end point position having arrows of the same length are set, image data for 10 timings grasped by time information are stored in each set. On the other hand, it is also possible to automatically associate each image data for five timings in chronological order. The associating unit 130 may store in the drawing-related data storage unit 160 correspondence information indicating the correspondence between sets and image data obtained by such associating processing. The image data may be image data captured by a camera provided on the unmanned air vehicle 4 , may be stored in the air vehicle-related information storage unit 170 , and may be read by the association unit 130 .

The image linking unit 140 links the image data corresponding to each set linked by the linking unit 130 to predetermined positions from the start point position to the end point position.

As a specific example, the image linking unit 140 may set the direction from the start point position to the end point position along the height direction of the structure (particularly, when the Y axis is the height of the structure). direction, and the direction from the start point position to the end point position is set along the Y-axis direction), the height information included in the image data (that is, the image captured by the camera of the unmanned air vehicle 4). The position on the drawing (for example, the position between the start point position and the end point position) may be set based on the acquired altitude information). As a more specific example, the length of the arrow indicating the selection set illustrated in FIG. In the case of 6m, 8m, and 10m, five pieces of image data may be linked to positions 0cm, 2cm, 6cm, 8cm, and 10cm from the starting point of the arrow as illustrated in FIG. For example, if the length of the arrow indicating the selection set is 5 cm on the drawing, the five image data are 0 cm, 1 cm, 3 cm, 4 cm, It may be tied at a position of 5 cm. At this time, if the minimum value of the height information of the associated multiple image data is not 0, the range from 0 to the maximum value is the distance from the start point position to the end point position of the corresponding selection set (that is, (length of the arrow selected in the case of ), and each image data may be linked from the position corresponding to the minimum value of the height information of the associated multiple image data. Alternatively, instead of the range from 0 to the maximum value, a position corresponding to at least one of the lower limit value and the upper limit value of the height information of a plurality of associated image data is set based on the user's operation on the terminal 2. Alternatively, it may be automatically set as the start point position or end point position of the corresponding selection set according to the height information of the image data.

Further, in the same manner as described above, for example, the image linking unit 140 may set the direction from the start point position to the end point position along the horizontal direction of the structure (particularly, when the X axis is along the horizontal direction of the structure). and the direction from the start point position to the end point position is set along the X-axis direction), the latitude and longitude information included in the image data (that is, when the image was acquired by the camera of the unmanned air vehicle 4 The position on the drawing may be set based on the latitude and longitude information). At this time, it may be configured such that it can be set based on at least one of latitude information and longitude information to associate the position of the image data based on the user's operation on the terminal 2 .

As another specific example, the image linking unit 140 equally divides the distance from the start point position to the end point position by the number of associated image data along the direction from the start point position to the end point position. You may make it string|string respectively to a position equally. As a more specific example, if the length of the arrow indicating the selection set illustrated in FIG. It may be linked to positions 0 cm, 2 cm, 4 cm, 6 cm, 8 cm, and 10 cm from the starting point. For example, if the length of the arrow indicating the selection set is 5 cm on the drawing, the six image data are linked to positions 0 cm, 1 cm, 2 cm, 3 cm, 4 cm, and 5 cm from the starting point of the arrow. may be

The information output unit 150 outputs drawing data stored in the drawing-related data storage unit 160, image data associated with the drawing data, aircraft path information and self-position information stored in the aircraft-related information storage unit 170, and unmanned flight data. It generates output information to be transmitted in response to a request from the terminal 2 or the unmanned air vehicle 4 based on image data captured by the body 4 or the like.

As a specific example, the information output unit 150 selects tag information linked to the drawing data on the terminal 2 (for example, the tag information is information indicating the position of the structure, and in FIG. 13, the tag information is " The corresponding drawing data may be included in the output information and generated accordingly.

In addition, in FIG. 13, the position of the image data linked on the drawing may be visualized based on the visualization information such as a symbol. In this case, the information output unit 150 may generate the output information including the corresponding image data. At that time, the output information may also include image-related information related to the image data included in the output information.

The image-related information may be, for example, information about a structure, such as location information of the structure (for example, latitude and longitude information), component type information on the walls of the structure (for example, tile, brick, concrete, wood, etc.). etc.) may be included.

The image-related information may include, for example, position information (for example, latitude/longitude/height information) obtained by acquiring an image with the unmanned air vehicle 4 and information indicating the deterioration state of the structure. Information indicating the deterioration state of the structure (hereinafter also referred to as "deterioration state information") may be deterioration state information associated with image data via an input operation on the terminal 2, or may be deterioration state information associated with image data. may be deterioration state information obtained by image analysis. The deterioration state information obtained by image analysis of the image data is the result information obtained by the information output unit 150 inputting the image data into a learning model provided inside or outside the system and analyzing it. It may be the result information analyzed using a learning model for estimating the deterioration state of the. The deterioration state information includes, for example, deterioration type information (for example, cracks, chips, stains, etc.) as shown in FIG. level divided into 2, levels divided into predetermined ranges of the size of the chipped area, etc.). Furthermore, the information output unit 150 may limit the image data to be visualized on the read drawing to visualizing the image data estimated to include the degraded portion using the deterioration state information.

Based on the output information generated by the information output unit 150, the terminal 2 displays the information in a format as illustrated in FIG. 13 in correspondence with a predetermined user interface on the application executed on the terminal 2. FIG.

<Example of inspection method>
An inspection method by the information processing system according to the present embodiment will be described with reference to FIG.

First, the server 1 sets one or more sets of start point positions and end point positions on a drawing showing at least a part of a structure (for example, a wall surface) by the start point and end point setting unit 120 (SQ101).

Next, the server 1 associates one or more pieces of image data with the set of start point position and end point position set by the start point/end point setting unit 120 by the association unit 130 .

Next, the server 1 causes the image linking unit 140 to link the image data corresponding to each set linked by the linking unit 130 to predetermined positions from the start point position to the end point position (SQ103).

Next, the server 1 generates output information to be transmitted to the terminal 2 based on the drawing data stored in the drawing-related data storage unit 160, the image data associated with the drawing data, and various other information (SQ104). .

Next, based on the output information generated by the information output unit 150, the terminal 2 displays various information in a predetermined format in correspondence with a predetermined user interface on the application executed on the terminal 2 (SQ105).

Thus, the present invention can provide an information processing system, an information processing method, a program, and a server for linking image data to a drawing showing at least a part of a structure. In such a case, it becomes easy to link image data obtained by photographing a structure described in a drawing that does not include latitude and longitude information.

The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit and interpret the present invention. It goes without saying that the present invention can be modified and improved without departing from its spirit, and that equivalents thereof are included in the present invention.

1 server 2 user terminal 4 unmanned air vehicle

Claims (8)

a start point/end point setting unit that sets one or more sets of start point positions and end point positions on a drawing showing at least a portion of a structure;
an associating unit that associates one or more pieces of image data selected based on a user's selection operation with the set;
A position linking unit that links image data corresponding to a predetermined position between the start point position and the end point position corresponding to the associated set,
An information processing system characterized by: The start point and end point setting unit sets the set of the start point position and the end point position based on a user's setting operation on the drawing.
The information processing system according to claim 1, characterized by: The start point and end point setting unit analyzes the drawing and sets a set of the start point position and the end point position for at least one surface of the structure.
The information processing system according to claim 1, characterized by: the position linking unit calculates the predetermined position based on at least one of time information and position information regarding the image data, and links the image data corresponding to the predetermined position;
4. The information processing system according to any one of claims 1 to 3, characterized by: The drawing is a drawing based on drawing information that does not include latitude and longitude information,
4. The information processing system according to any one of claims 1 to 3, characterized by: A step of setting one or more sets of starting point positions and ending point positions on a drawing showing at least a part of a structure by a starting point and end point setting unit;
a step of associating one or more pieces of image data selected based on a selection operation by a user with the set by an associating unit;
a step of linking image data corresponding to a predetermined position between a start point position and an end point position corresponding to the associated set by a position linking unit;
An information processing method characterized by: A step of setting one or more sets of starting point positions and ending point positions on a drawing showing at least a part of a structure by a starting point and end point setting unit;
a step of associating one or more pieces of image data selected based on a selection operation by a user with the set by an associating unit;
causing the information processing system to perform a step of linking image data corresponding to a predetermined position between the start point position and the end point position corresponding to the associated set by the position linking unit;
A program characterized by a start point/end point setting unit that sets one or more sets of start point positions and end point positions on a drawing showing at least a portion of a structure;
an associating unit that associates one or more pieces of image data selected based on a user's selection operation with the set;
A position linking unit that links image data corresponding to a predetermined position between the start point position and the end point position corresponding to the associated set,
A server characterized by:

Images