JP2022067096A - Device, system and method for monitoring and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method which accurately identifies an imaging position of an image taken for a maintenance or a monitoring of a traffic infrastructure including a railway and the like.

SOLUTION: A monitoring device to monitor a maintained object on a travel route of a vehicle has a control unit to store a first image taken about a maintained object and a second image taken about a periphery of the imaging position of the first image in an information storing unit in association with each other. The control unit associates positioning information obtained by positioning the imaging position of the first image with the first image and the second image. The positioning information includes positioning accuracy indicating accuracy of the positioning. The control unit identifies the imaging position of the first image based on the positioning information when the positioning accuracy is equal to or better than a prescribed threshold value and identifies the imaging position of the first image based on the second image when the positioning accuracy is under the threshold value.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present disclosure relates to monitoring devices, monitoring systems, monitoring methods, and computer programs.

Conventionally, a camera is attached to a maintenance vehicle or the like of a railway, a rail is photographed by the camera while the maintenance vehicle or the like is running, and the photographed image is used for maintenance of the rail.

In Patent Document 1, an infrared camera mounted on a maintenance vehicle or the like captures a rail to generate an infrared image, and the position where the infrared image is captured is specified from the mileage of the maintenance vehicle or the like, and the specified position is specified. It is disclosed to generate data in which an infrared image is associated with an infrared image.

Japanese Unexamined Patent Publication No. 10-73413

However, in order to specify the position from the mileage as in Patent Document 1, it is necessary to register in advance information indicating the departure position and the travel route of the maintenance vehicle or the like. Further, since the mileage is calculated by using the number of rotations of the wheels in the mileage meter provided in the maintenance vehicle or the like, a large error may be included due to wear and / or slip of the wheels.

It is an object of the present disclosure to provide a technique capable of accurately specifying the shooting position of an image taken for maintenance or monitoring of a transportation infrastructure including a railway.

The monitoring device according to one aspect of the present disclosure is a monitoring device that monitors a maintenance object on the movement path of the vehicle, and is a monitoring device for photographing the maintenance object and the periphery of the photographing position of the first image and the first image. The control unit is provided with a control unit that stores the image in the information storage unit in association with the second image obtained by the image. The positioning information is associated with the positioning information, and the positioning information includes the positioning accuracy indicating the accuracy of the positioning. When the positioning accuracy is equal to or higher than a predetermined threshold value, the control unit uses the positioning information as the first image. If the positioning accuracy is less than the threshold value, the shooting position of the first image is specified based on the second image.

The monitoring device according to one aspect of the present disclosure is a monitoring device that monitors a maintenance object on the movement path of the vehicle, and is a monitoring device for photographing the maintenance object and the periphery of the photographing position of the first image and the first image. The control unit is provided with a control unit that stores the image in the information storage unit in association with the second image obtained by the image. The positioning information is associated with the positioning information, and the positioning information includes the positioning accuracy indicating the accuracy of the positioning. When the positioning accuracy is equal to or higher than a predetermined threshold value, the control unit uses the positioning information as the first image. If the positioning accuracy is less than the threshold value, the second image is displayed and the user accepts the input of the shooting position of the second image.

The monitoring system according to one aspect of the present disclosure is a monitoring system that monitors a maintenance object on a vehicle's movement path, and includes a first camera that photographs the maintenance object and generates a first image, and the above-mentioned. A second camera that captures the periphery of the shooting position of the first image to generate a second image, the first image that captures the maintenance object, and a second camera that captures the periphery of the shooting position of the first image. When the positioning information for positioning the shooting position of the first image including the positioning accuracy indicating the positioning accuracy is associated with the two images and stored in the information storage unit, and the positioning accuracy is equal to or higher than a predetermined threshold value. A monitoring device for specifying the shooting position of the first image based on the positioning information and specifying the shooting position of the first image based on the second image when the positioning accuracy is less than the threshold value is provided. ..

The monitoring system according to one aspect of the present disclosure is a monitoring system that monitors a maintenance object on a vehicle's movement path, and includes a first camera that photographs the maintenance object and generates a first image, and the above-mentioned. A second camera that captures the periphery of the shooting position of the first image to generate a second image, the first image that captures the maintenance object, and a second camera that captures the periphery of the shooting position of the first image. When the positioning information for positioning the shooting position of the first image including the positioning accuracy indicating the positioning accuracy is associated with the two images and stored in the information storage unit, and the positioning accuracy is equal to or higher than a predetermined threshold value. The shooting position of the first image is specified based on the positioning information, and when the positioning accuracy is less than the threshold value, the second image is displayed and the user inputs the shooting position of the second image. It is equipped with a monitoring device that accepts images.

The monitoring method according to one aspect of the present disclosure is a monitoring method for monitoring a maintenance object on the movement path of the vehicle, the first image in which the maintenance object is photographed, and the periphery of the imaging position of the first image. The first image is stored in the information storage unit in association with the second image in which the image was taken, and in addition to the second image, the positioning information obtained by positioning the shooting position of the first image is associated with the positioning. The information includes positioning accuracy indicating the accuracy of the positioning, and when the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information, and the positioning accuracy is less than the threshold value. In this case, the shooting position of the first image is specified based on the second image.

The monitoring method according to one aspect of the present disclosure is a monitoring method for monitoring a maintenance object on the movement path of the vehicle, the first image in which the maintenance object is photographed, and the periphery of the imaging position of the first image. The first image is stored in the information storage unit in association with the second image in which the image was taken, and in addition to the second image, the positioning information obtained by positioning the shooting position of the first image is associated with the positioning. The information includes positioning accuracy indicating the accuracy of the positioning, and when the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information, and the positioning accuracy is less than the threshold value. In this case, the second image is displayed, and the user accepts the input of the shooting position of the second image.

The computer program according to one aspect of the present disclosure is a computer program for monitoring a maintenance object on a moving path of a vehicle, and is a first image in which the maintenance object is photographed and a photographing position of the first image. The first image is stored in the information storage unit in association with the second image obtained by photographing the periphery of the above image, and in addition to the second image, the positioning information obtained by positioning the photographing position of the first image is associated with the first image. The positioning information includes a positioning accuracy indicating the accuracy of the positioning, and when the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information, and the positioning accuracy is the threshold value. If it is less than, the computer is made to specify the shooting position of the first image based on the second image.

The computer program according to one aspect of the present disclosure is a computer program for monitoring a maintenance object on a moving path of a vehicle, and is a first image in which the maintenance object is photographed and a photographing position of the first image. The first image is stored in the information storage unit in association with the second image obtained by photographing the periphery of the above image, and in addition to the second image, the positioning information obtained by positioning the photographing position of the first image is associated with the first image. The positioning information includes positioning accuracy indicating the accuracy of the positioning, and when the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information, and the positioning accuracy is the threshold value. If it is less than, the second image is displayed and the computer is made to accept the input of the shooting position of the second image from the user.

It should be noted that these comprehensive or specific embodiments may be realized by a system, an apparatus, a method, an integrated circuit, a computer program or a recording medium, and any of the system, an apparatus, a method, an integrated circuit, a computer program and a recording medium. It may be realized by various combinations.

According to the present disclosure, it is possible to accurately identify the shooting position of an image taken for maintenance or monitoring of transportation infrastructure including railways and the like.

Block diagram showing a configuration example of the monitoring system according to this embodiment Functional block diagram showing a configuration example of the monitoring device according to this embodiment The figure which shows the structural example of the monitoring information which concerns on this embodiment A diagram for explaining an example of a method of associating a monitored image with positioning information. A diagram for explaining an example of a method for estimating positioning information. A flowchart showing an example of the process of specifying the shooting position of the surveillance image. The figure which shows the display example of the maintenance UI (User Interface) which concerns on this embodiment. The figure which shows the hardware configuration of the computer which realizes the functional block of the monitoring apparatus which concerns on this embodiment by a computer program.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

(Implementation)
<Configuration of monitoring system>
FIG. 1 is a block diagram showing a configuration example of a monitoring system according to the present embodiment.

The monitoring system 2 is a system for monitoring railway maintenance objects. Examples of railway maintenance objects include rails, rail fastening devices, sleepers, trackbeds, roadbeds, overhead lines, overhead wire poles, insulators, balancers, traffic lights, tunnel inner walls, bridges, slopes, and the like. The "railway" according to the present disclosure is not limited to a track on which a general train or a train travels, but also includes a track on which a tram travels, a track on which a monorail travels, a track on which a Shinkansen travels, and the like.

The monitoring system 2 is mounted on a railway maintenance vehicle or the like (hereinafter referred to as "vehicle") 1. The vehicle 1 may be a powered vehicle or a vehicle (bogie) towed by the powered vehicle. A part of the monitoring system 2 may be installed in a place different from that of the vehicle 1.

The monitoring system 2 includes cameras 10A, 10B, 10C, a video recorder 20, a global positioning satellite (hereinafter referred to as GNSS (Global Navigation Satellite System)) receiving device 30, an RTK (Real Time Kinematic) receiving device 40, and monitoring. The device 100 is provided.

The cameras 10A, 10B, and 10C are compatible with PoE (Power over Ethernet) and may be connected to the video recorder 20 by an Ethernet cable. However, the cameras 10A, 10B, and 10C may be connected to the video recorder 20 by a wired cable (for example, a USB (Universal Serial Bus) cable) different from the Ethernet cable or a wireless method (for example, IEEE802.11 or Bluetooth). Further, the connection method between the cameras 10A, 10B and 10C and the video recorder 20 is not limited to PoE. For example, the cameras 10A, 10B, and 10C may be supplied with auxiliary power by another connection method.

The video recorder 20 is connected to the monitoring device 100 by, for example, an Ethernet cable. However, the video recorder 20 may be connected to the monitoring device 100 by a wired cable (for example, a USB cable) different from the Ethernet cable or a wireless method (for example, IEEE802.11 or Bluetooth).

The GNSS receiving device 30 is connected to the monitoring device 100 by, for example, a USB cable. However, the GNSS receiving device 30 may be connected to the monitoring device 100 by a wired cable or a wireless method different from the USB cable.

The RTK receiving device 40 is connected to the monitoring device 100 by, for example, a USB cable. However, the RTK receiving device 40 may be connected to the monitoring device 100 by a wired cable or a wireless method different from the USB cable.

The camera 10A is installed in the vehicle 1 so that the object to be maintained can be photographed. The number of cameras 10A installed is not limited to one, and may be two or more. The moving image data generated by the camera 10A taken from the moving vehicle 1 is referred to as a surveillance moving image. As an example of the format of the surveillance video, MPEG2, MPEG4, H.M. 264, H. 265, JPEG, uncompressed data and the like.

The camera 10B is installed in the vehicle 1 so that a distance marker existing around the track can be photographed. The number of cameras 10B installed is not limited to one, and may be two or more. The moving image data generated by the camera 10B taken from the moving vehicle 1 is referred to as a distance marker moving image. The format of the distance marker video may be the same as that of the surveillance video. A distance marker is a sign indicating the distance from a certain starting point on a railway. The distance marker often has a character indicating the distance, but the character may not be indicated. The "character" of the distance marker may include numbers, symbols and the like. Further, the distance marker is not limited to a sign indicating a distance from a certain starting point, and may be a thing indicating a distance, a distance interval, or a specific place.

The camera 10C is installed in the vehicle 1 so that the scenery seen from the vehicle 1 can be photographed. The number of cameras 10C installed is not limited to one, and may be two or more. The moving image data generated by the camera 10C taken from the moving vehicle 1 is referred to as a landscape moving image. The format of the landscape video may be the same as that of the surveillance video. The camera 10C may be installed so as to capture a landscape behind, in front of, or on the side of the vehicle 1. The camera 10C may be installed inside the vehicle 1 or may be installed outside the vehicle 1 (for example, on the roof of the vehicle 1).

The camera 10A, the camera 10B, and the camera 10C may be installed in the vehicle 1 so as to be able to shoot in different directions from each other. As a result, as described above, different objects such as maintenance objects, distance markers, and landscapes can be photographed. Alternatively, if one camera can shoot both the distance marker and the landscape, the camera 10B may shoot a moving image including both the distance marker and the landscape without installing the camera 10C.

The video recorder 20 is a device that receives and records a surveillance video, a distance marker video, and a landscape video from the cameras 10A, 10B, and 10C, respectively.

The GNSS receiving device 30 periodically receives a GNSS signal, which is a positioning signal emitted from the GNSS satellite 3, and generates GNSS information 201 (see FIG. 2) from the received GNSS signal. The GNSS information 201 includes the reception time of the GNSS signal, the latitude and longitude of the GNSS receiving device 30 at the reception time, and the positioning accuracy of the latitude and longitude. The GNSS receiving device 30 transmits the generated GNSS information 201 to the monitoring device 100. Hereinafter, the measured latitude and longitude are referred to as positioning coordinates.

The RTK receiving device 40 periodically receives an RTK signal, which is a positioning signal transmitted from the radio base station 4, and generates correction information 202 (see FIG. 2) from the received RTK signal. The correction information 202 includes information for correcting the positioning coordinates indicated by the GNSS signal. The positioning coordinates corrected by using the RTK signal indicate a more accurate position than the positioning coordinates before the correction. The RTK receiving device 40 transmits the generated correction information 202 to the monitoring device 100.

The monitoring device 100 receives the GNSS information 201 and the correction information 202 from the GNSS receiving device 30 and the RTK receiving device 40, respectively. The monitoring device 100 is based on the received GNSS information 201 and correction information 202, and the information indicating the positioning coordinates of the camera 10A at the reception time included in the GNSS information 201 and the correction information 202 (hereinafter referred to as “positioning information”) 206. To generate. For example, by installing the GNSS receiving device 30 and the camera 10A so as to be aligned in a straight line in the height direction, the positioning coordinates of the camera 10A can be obtained from the GNSS information 201. Alternatively, the positioning coordinates of the camera 10A can be obtained by registering the positional deviation between the GNSS receiving device 30 and the camera 10A in advance and correcting the positioning coordinates indicated by the GNSS information 201 with the registered positional deviations. be able to.

The monitoring device 100 is inside the monitoring device 100 based on at least one of the time information included in the GNSS information 201 and the time information received from the time server 5 (for example, an NTP (Network Time Protocol) server). Set the clock. As a result, the internal clock of the monitoring device 100 is synchronized with the correct time. In addition, the monitoring device 100 synchronizes the video recorder 20 and the internal clocks of the cameras 10A, 10B, and 10C with the internal clock of the monitoring device 100. As a result, the internal clocks of the cameras 10A, 10B, 10C, the video recorder 20, and the monitoring device 100 are synchronized with the correct time.

From the video recorder 20, the monitoring device 100 includes each image frame (hereinafter referred to as “surveillance image”) 203 (see FIG. 2) constituting the surveillance moving image and each image frame (hereinafter referred to as “distance marker image”) constituting the distance marker moving image. 204 (see FIG. 2) and each image frame (hereinafter referred to as “landscape image”) 205 (see FIG. 2) constituting the landscape moving image are acquired and stored.

The monitoring device 100 identifies the shooting position of the monitoring image 203 by using the positioning information 206 positioned at the same time as the shooting time of the monitoring image 203. As a result, the shooting position of the surveillance image 203 can be accurately specified. However, while the vehicle 1 is traveling in a mountainous area, an urban area, a tunnel, or the like, the positioning information 206 cannot be used because the GNSS signal cannot be received or the positioning accuracy of the GNSS signal is insufficient. In some cases. In such a case, the monitoring device 100 specifies the shooting position of the monitoring image 203 by using at least one of the distance marker image 204 and the landscape image 205 taken at the same time as the shooting time of the monitoring image 203. do. As a result, even when the positioning information 206 cannot be used, the shooting position of the monitoring image 203 can be accurately specified. In the description of the present disclosure, the expression "same time as the shooting time" does not mean exactly the same time, but represents a time included in a predetermined period including the shooting time. Next, the above-mentioned contents will be described in detail.

<Configuration of monitoring device>
FIG. 2 is a functional block diagram showing a configuration example of the monitoring device 100 according to the present embodiment.

The monitoring device 100 includes an information storage unit 101, a GNSS information receiving unit 102, a correction information receiving unit 103, a clock synchronization unit 104, an image acquisition unit 105, a positioning information generation unit 106, a monitoring information generation unit 107, and a shooting position specifying unit 108. It also has a UI processing unit 109.

The information storage unit 101 may be realized by the memory 1002, the storage 1003, or a combination thereof shown in FIG. The information storage unit 101 stores GNSS information 201, correction information 202, monitoring image 203, distance marker image 204, landscape image 205, positioning information 206, and monitoring information 207.

The GNSS information receiving unit 102, the correction information receiving unit 103, the clock synchronization unit 104, the image acquisition unit 105, the positioning information generation unit 106, the monitoring information generation unit 107, the shooting position specifying unit 108, and the UI processing unit 109 are shown in FIG. The processor 1001 shown in the above may be realized by cooperating with the memory 1002 and the like. Therefore, the process described mainly of these functional blocks 102 to 109 can be read as the process mainly composed of the processor 1001.

The GNSS information receiving unit 102 periodically receives the GNSS information 201 from the GNSS receiving device 30 and stores it in the information storage unit 101.

The correction information receiving unit 103 periodically receives the correction information 202 from the RTK receiving device 40 and stores it in the information storage unit 101.

The clock synchronization unit 104 sets the internal clock of the monitoring device 100 based on at least one of the time information included in the GNSS information 201 and the correction information 202 and the time information received from the time server 5. For example, when the time information is obtained from at least one of the GNSS information 201 and the correction information 202, the clock synchronization unit 104 sets the internal clock of the monitoring device 100 using the time information. When the time information cannot be obtained from either the GNSS information 201 or the correction information 202, the clock synchronization unit 104 acquires the time information from the time server 5 and sets the internal clock of the monitoring device 100 using the time information. do.

In addition, the clock synchronization unit 104 synchronizes the internal clocks of the video recorder 20 and the cameras 10A, 10B, and 10C with the internal clocks of the monitoring device 100. As a result, the internal clocks of the cameras 10A, 10B, 10C, the video recorder 20, and the monitoring device 100 are synchronized with the correct time. Therefore, the shooting times of the monitoring image 203, the distance marker image 204, and the landscape image 205 all represent accurate times.

The image acquisition unit 105 acquires the monitoring image 203, the distance marker image 204, and the landscape image 205 from the video recorder 20 and stores them in the information storage unit 101.

The positioning information generation unit 106 generates positioning information 206 indicating the positioning coordinates (latitude and longitude) of the camera 10A at the reception time included in the GNSS information 201 and the correction information 202 based on the GNSS information 201 and the correction information 202. Then, it is stored in the information storage unit 101.

The monitoring information generation unit 107 includes the monitoring image 203, the distance marker image 204 and the landscape image 205 taken at the same time as the shooting time of the monitoring image 203, and the positioning information 206 positioned at the same time as the shooting time. Is associated with each other to generate monitoring information 207. The details of the monitoring information 207 will be described later (see FIG. 3).

The shooting position specifying unit 108 uses at least one of the positioning information 206, the distance marker image 204, and the landscape image 205 associated with the monitoring image 203 in the monitoring information 207 to determine the shooting position of the monitoring image 203. Identify. The shooting position of the surveillance image 203 may be expressed as a business kilometer indicating a distance from a predetermined starting point on a railway line.

When the positioning accuracy of the positioning information 206 associated with the monitoring image 203 in the monitoring information 207 is sufficient (for example, the positioning accuracy is equal to or higher than a predetermined threshold value), the photographing position specifying unit 108 positions the positioning information 206. The business kilometer corresponding to the coordinates may be the shooting position of the monitoring image 203. For example, the photographing position specifying unit 108 converts the positioning coordinates of the positioning information 206 into business kilometers by using a predetermined database that converts the positioning coordinates on the orbit into business kilometers.

When the positioning information 206 is not associated with the monitoring image 203 in the monitoring information 207, or the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is insufficient (for example, the shooting position specifying unit 108). If the positioning accuracy is less than the predetermined threshold value), the following processing may be performed. That is, the shooting position specifying unit 108 uses at least one of the distance marker image 204 and the landscape image 205 associated with the monitoring image 203 in the monitoring information 207, and uses the business kilometer of the shooting position of the monitoring image 203. May be specified. The details of this specific method will be described later.

The UI processing unit 109 displays the maintenance UI 400 (see FIG. 7) based on the monitoring information 207, and accepts operations for the maintenance UI 400. For example, the UI processing unit 109 displays the maintenance UI 400 on a display which is an example of the output device 1005 shown in FIG. Then, the UI processing unit 109 receives an operation on the maintenance UI 400 through a keyboard and a mouse, which are examples of the input device 1004 shown in FIG. The details of the maintenance UI 400 will be described later (see FIG. 7).

<Structure of monitoring information>
FIG. 3 is a diagram showing a configuration example of the monitoring information 207 according to the present embodiment.

The monitoring information 207 has, as items, a monitoring image, a distance marker image, a landscape image, a shooting time, positioning information, a route name, a business kilometer, a maintenance necessity flag, a maintenance factor, a shooting means, a photographer, and a memo. That is, it can be said that the monitoring information 207 is information indicating the correspondence between these items.

The ID (Identification) of the surveillance image 203 is registered in the item of the surveillance image. An example of the ID of the surveillance image 203 is the file name of the surveillance image 203.

The ID of the distance marker image 204 is registered in the item of the distance marker image. An example of the ID of the distance marker image 204 is the file name of the distance marker image 204.

The ID of the landscape image 205 is registered in the landscape image item. An example of the ID of the landscape image 205 is the file name of the landscape image 205.

The shooting time of the surveillance image 203 is registered in the shooting time item. The shooting time item may include not only the time but also the date. Further, the shooting time can be displayed in milliseconds or microseconds depending on the shooting fps, so that the shooting timing of each image can be confirmed in detail. As a result, even if the value of fps becomes large, it is possible to prevent the images taken at different timings from being recognized as being taken at the same time. For example, for an image shot every millisecond, if the shooting time is displayed only in seconds, it is not possible to grasp the shooting order of each image, but by displaying it in milliseconds, each image is shot. You can grasp the order.

In the item of positioning information, the positioning coordinates measured at the same time as the shooting time of the monitoring image 203 and the positioning accuracy of the positioning coordinates are registered. The positioning accuracy may be expressed by a numerical value indicating the high accuracy. Alternatively, the positioning accuracy may be expressed in three stages: "good" indicating that the accuracy is sufficient, "poor" indicating that the accuracy is insufficient, and "impossible" indicating that positioning could not be performed. good.

In the item of the route name, the name of the route on which the monitoring image 203 was taken is registered.

In the item of business kilometers, the business kilometers of the shooting position of the surveillance image 203 are registered.

In the maintenance necessity flag item, a flag indicating whether or not maintenance work is required is registered for the maintenance target object shown in the monitoring image 203. The maintenance worker may see the monitoring image 203 displayed on the maintenance UI 400 described later and determine whether or not maintenance work is necessary for the maintenance target object in the monitoring image 203. Alternatively, the monitoring device 100 may analyze the monitoring image 203 and automatically determine whether or not maintenance work is required for the maintenance target in the monitoring image 203. A flag indicating the result of the determination may be registered in the maintenance necessity flag item. For example, "necessary" indicating that maintenance work is required or "unnecessary" indicating that maintenance work is not required may be registered in the maintenance necessity flag.

In the maintenance factor item, when the maintenance necessity flag is "necessary", the factor that requires maintenance work is registered. For example, when the equipment of the maintenance target is abnormal, "equipment abnormality" may be registered in the item of maintenance factor. For example, when the maintenance object is missing, "missing" may be registered in the maintenance factor item.

The photographing means of the surveillance image 203 is registered in the item of the photographing means. For example, when the monitoring image 203 is taken by the camera 10A mounted on the maintenance vehicle or the like, "maintenance vehicle or the like" may be registered in the item of the photographing means. For example, when the surveillance image 203 is taken by a terminal with a camera owned by a maintenance worker, the "hand terminal" may be registered in the item of the shooting means.

The photographer of the surveillance image 203 is registered in the photographer item. For example, when the surveillance image 203 is taken by a maintenance worker named "Pana Hanako", "Pana Hanako" is registered in the photographer's item. For example, when the surveillance image 203 is taken by a maintenance worker named "Taro Matsushita", "Taro Matsushita" is registered in the item of the photographer.

In the memo item, the content of the memo for the monitoring image 203 is registered. For example, when a maintenance worker inputs a memo by looking at the monitoring image 203 displayed on the maintenance UI 400 described later, the content of the input memo is registered in the memo item.

<Association of surveillance image and positioning information>
FIG. 4 is a diagram for explaining an example of a method of associating the monitoring image 203 with the positioning information 206.

As shown in FIG. 3, the monitoring information generation unit 107 generates monitoring information 207 by associating the monitoring image 203 with the positioning information 206 positioned at the same time as the shooting time of the monitoring image 203. Thereby, the shooting position of the surveillance image 203 can be specified from the positioning information 206 associated with the surveillance image 203.

However, the frame rate of the surveillance video and the reception frequency of the GNSS signal may differ. For example, it is assumed that the frame rate of the surveillance video is 30 fps and the reception frequency of the GNSS signal is 10 Hz. In this case, since the frame period of the surveillance video is (1/30) seconds and the positioning cycle by the GNSS signal is (1/10) seconds, the positioning information 206 is provided one-to-one for all the surveillance images 203. It cannot be associated.

Therefore, when the positioning information 206 positioned at the same time as the shooting time of the monitoring image 203 exists, the shooting position specifying unit 108 associates the positioning information 206 with the monitoring image 203 and sets the shooting time of the monitoring image 203. If the positioning information 206 positioned at the same time does not exist, the positioning information 206 corresponding to the monitoring image 203 is estimated based on the positioning information 206 positioned at another time.

For example, as shown in FIG. 4, when the monitoring image 203 is captured at each of the time t1, t2, t3, and t4, and the positioning information 206 is positioned at the time t1 and the time t4, the imaging position specifying unit 108 , Performs the following processing. That is, the shooting position specifying unit 108 associates the monitoring image 203 shot at time t1 with the positioning information 206 positioned at time t1, and positions the monitoring image 203 shot at time t4 at time t4. The positioning information 206 is associated with it. In addition, the photographing position specifying unit 108 estimates the positioning information 206 at the time t2 and the time t3 based on the positioning information 206 at the time t1 and the time t4. Next, the estimation method will be described.

FIG. 5 is a diagram for explaining an example of a method for estimating positioning information.

For example, as shown in FIG. 5, the photographing position specifying unit 108 divides the line segment L connecting the position P1 indicated by the positioning information 206 at time t1 and the position P4 indicated by the positioning information 206 at time t4 into three division points. Obtain P2 and P3. The number of divisions may be calculated based on a value obtained by dividing the positioning cycle by the cycle of the monitoring image 203. Further, the shape of the line segment L may be along the alignment of the rail.

The shooting position specifying unit 108 estimates the division point P2 as the positioning information 206 at the time t2, and the division point P3 as the positioning information 206 at the time t3. Then, in the monitoring information 207, the shooting position specifying unit 108 associates the estimated time t2 positioning information 206 with the time t2 monitoring image 203, and associates the estimated time t2 positioning information 206 with the time t3 monitoring image 203. Correspond to information 206.

As a result, the positioning information 206 can be associated with each surveillance image 203 even when the frame cycle of the surveillance video and the positioning cycle based on the GNSS signal are different.

<When positioning accuracy is insufficient>
Since the GNSS signal cannot be received while the vehicle 1 is traveling in a tunnel, underground, or the like, the positioning information generation unit 106 cannot generate the positioning information 206 during that time. Therefore, in the monitoring information 207, the positioning information 206 may not be associated with the monitoring image 203 taken while the vehicle 1 is traveling in a tunnel, underground, or the like. In this case, the shooting position of the monitoring image 203 cannot be specified from the positioning information 206.

Further, while the vehicle 1 is traveling in an urban area or a mountainous area, the GNSS signal can be received but the RTK signal cannot be received, or the reception accuracy of the GNSS signal is insufficient. The positioning accuracy of information 207 may be insufficient. Therefore, in the monitoring information 207, the positioning information 206 associated with the monitoring image 203 taken while the vehicle 1 is traveling in an urban area, a mountainous area, or the like may have insufficient positioning accuracy. .. In this case, the shooting position of the monitoring image 203 specified from the positioning information 206 may be significantly deviated from the actual shooting position.

Therefore, the photographing position specifying unit 108 is associated with the monitoring image 203 in which the positioning information 206 is not associated with the monitoring information 207, or the positioning information 206 with insufficient positioning accuracy (less than a predetermined threshold value) is associated with the monitoring information 207. The shooting position of the surveillance image 203 is specified by using at least one of the distance marker image 204 and the landscape image 205 taken at the same time as the shooting time of the surveillance image 203.

For example, the shooting position specifying unit 108 specifies the shooting position from the distance marker image 204 as follows. That is, the shooting position specifying unit 108 performs character recognition on the distance marker image 204 shot at the same time as the shooting time of the monitoring image 203, and reads the characters of the distance marker. The shooting position specifying unit 108 identifies the business kilometer indicated by the distance marker from the characters of the read distance marker. In the monitoring information 207, the shooting position specifying unit 108 associates the specified business kilometer with the monitoring image 203 shot at the same time as the shooting time of the distance marker image 204.

For example, the shooting position specifying unit 108 specifies the shooting position from the landscape image 205 as follows. That is, the shooting position specifying unit 108 recognizes an object on the landscape image 205 shot at the same time as the shooting time of the monitoring image 203, and recognizes the object. The photographing position specifying unit 108 inquires of a predetermined database having a correspondence relationship between the recognized object and the position where the object exists, and obtains the position where the recognized object exists. In the monitoring information 207, the photographing position specifying unit 108 associates the operating kilometer at the position where the object exists with the monitoring image 203 photographed at the same time as the photographing time of the landscape image 205. Examples of objects that can be photographed and located in landscape images include tunnels, bridges, station buildings, stores, signs, signs, bridges, and landmarks.

For example, the shooting position specifying unit 108 specifies the shooting position from the landscape image 205 as follows in cooperation with the UI processing unit 109. That is, the UI processing unit 109 displays the landscape image 205 taken at the same time as the shooting time of the monitoring image 203, and receives the input of the position indicated by the landscape image 205 from the maintenance worker. The maintenance worker looks at the displayed landscape image 205 and inputs the position where the landscape image 205 was taken. In the monitoring information 207, the shooting position specifying unit 108 associates the business kilometer of the input position with the monitoring image 203 shot at the same time as the shooting time of the landscape image 205.

<Process to specify the shooting position of the surveillance image>
FIG. 6 is a flowchart showing an example of the process of specifying the shooting position of the surveillance image 203.

The shooting position specifying unit 108 selects one of the monitoring images 203 from the plurality of monitoring images 203 included in the monitoring information 207 (S101).

The shooting position specifying unit 108 acquires the positioning information 206 associated with the monitoring image 203 selected in S101 in the monitoring information 207 (that is, measured at the same time as the shooting time of the monitoring image 203) (S102). ).

The monitoring information generation unit 107 determines whether or not the positioning accuracy included in the positioning information 206 acquired in S102 is equal to or higher than a predetermined threshold value (S103).

When the positioning accuracy is equal to or higher than the threshold value (S103: YES), the shooting position specifying unit 108 specifies the business kilometer in the positioning coordinates of the positioning information 206 acquired in S102 (S104). Then, the process of S112 is executed.

When the positioning accuracy is less than the threshold value, or when the positioning information 206 is not associated with the monitoring image 203 selected in S101 (S103: NO), the photographing position specifying unit 108 performs the following processing. That is, the shooting position specifying unit 108 acquires the distance marker image 204 associated with the monitoring image 203 selected in S101 in the monitoring information 207 (that is, taken at the same time as the shooting time of the monitoring image 203). (S105).

The shooting position specifying unit 108 performs character recognition of the distance marker for the distance marker image 204 selected in S105 (S106). The shooting position specifying unit 108 determines whether or not the character recognition of the distance marker is successful in S106 (S107).

When the character recognition of the distance marker is successful (S107: YES), the shooting position specifying unit 108 specifies the business kilometer indicated by the recognized character of the distance marker (S108). Then, the process of S112 is executed.

When the character recognition of the distance marker fails (S107: NO), the shooting position specifying unit 108 cooperates with the UI processing unit 109 to perform the following processing. That is, the UI processing unit 109 is associated with the monitoring image 203 selected in S101 in the monitoring information 207 as the maintenance UI 400 (that is, the landscape image taken at the same time as the shooting time of the monitoring image 203). 205 is displayed, and the input of the position indicated by the landscape image 205 is accepted (S109).

The maintenance worker inputs the position information indicated by the displayed landscape image 205 (S110). The shooting position specifying unit 108 specifies the business kilometer of the position input in S110 (S111). Then, the process of S112 is executed.

In the monitoring information 207, the shooting position specifying unit 108 associates the monitoring image 203 selected in S101 with the business kilometer specified in S104, S108, or S111 (S112). The monitoring device 100 executes the above-described processing for each monitoring image 203 included in the monitoring information 207, and then ends this processing.

According to the above processing, when the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is sufficient, the shooting position specifying unit 108 uses the positioning information 206 to operate the shooting position of the monitoring image 203. To identify. Further, when the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is insufficient, or when the positioning information 206 is not associated with the monitoring image 203, the photographing position specifying unit 108 monitors the monitoring image 203. The business kilometer of the shooting position of the monitoring image 203 is specified by using the distance marker image 204 or the landscape image 205 associated with the above. Thereby, regardless of the place where the surveillance image 203 is photographed, it is possible to specify the business kilometer of the imaging position of the surveillance image 203.

<Display of maintenance UI>
FIG. 7 is a diagram showing a display example of the maintenance UI 400 according to the present embodiment.

The UI processing unit 109 displays an image of the maintenance UI 400 as shown in FIG. 7 on a display which is an example of the output device 1005. The maintenance UI 400 includes a monitoring image area 401, a distance marker image area 402, a landscape image area 403, an information area 404, a map area 405, a front button 406, a next button 407, a maintenance-required button 408, and a memo button 409.

In the surveillance image area 401, one of the surveillance images 203 included in the surveillance information 207 is displayed.

In the distance marker image area 402, the distance marker image 204 associated with the monitoring image 203 displayed in the monitoring image area 401 in the monitoring information 207 is displayed.

In the landscape image area 403, the landscape image 205 associated with the monitoring image 203 displayed in the monitoring image area 401 in the monitoring information 207 is displayed.

In the information area 404, information about the monitoring image 203 displayed in the monitoring image area 401 is displayed. For example, the file name of the surveillance image 203, the shooting date, the shooting time, the photographer, the memo, the business kilometer, and the latitude and longitude indicating the shooting position are displayed. The UI processing unit 109 may display the information in the information area 404 with reference to the monitoring information 207.

In the map area 405, a map around the shooting position of the monitoring image 203 displayed in the monitoring image area 401 and a pin 410 indicating the shooting position are displayed.

The UI for accepting the input of the position indicated by the landscape image 205 described in S110 of FIG. 6 may be the same as the maintenance UI 400. In this case, the maintenance worker can input the position information indicated by the landscape image 205 by looking at the landscape image 205 displayed in the landscape image area 403 and hitting the pin 410 at the position on the map indicated by the landscape.

When the front button 406 is pressed, the UI processing unit 109 displays the monitoring image 203, the distance marker image 204, and the landscape image 205 associated with the previous shooting time in the monitoring information 207 in the monitoring image area. It is displayed in 401, a distance marker image area 402, and a landscape image area 403, respectively. Further, the UI processing unit 109 displays a map around the shooting position of the monitoring image 203 associated with the previous shooting time and a pin 410 indicating the shooting position in the map area 405.

When the next button 407 is pressed, the UI processing unit 109 displays the monitoring image 203, the distance marker image 204, and the landscape image 205 associated with the next shooting time in the monitoring information 207 in the monitoring image area. It is displayed in 401, a distance marker image area 402, and a landscape image area 403, respectively. Further, the UI processing unit 109 displays a map around the shooting position of the monitoring image 203 associated with the next shooting time and a pin 410 indicating the shooting position in the map area 405.

When the maintenance-required button 408 is pressed, the UI processing unit 109 changes the maintenance-required flag associated with the displayed monitoring image 203 to "necessary" in the monitoring information 207.

When the memo button 409 is pressed, the UI processing unit 109 displays a memo input UI (not shown). In addition, the UI processing unit 109 registers the content input to the memo input UI in the memo item associated with the monitored image 203 being displayed in the monitoring information 207.

For example, the maintenance worker presses the next button 407 to switch the monitoring image 203 to be displayed, and confirms whether or not there is a problem with the maintenance object. When there is a problem with the maintenance object, the maintenance worker presses the maintenance-required button 408. Further, when the maintenance worker wants to add a comment to the maintenance object, he / she presses the memo button 409 and inputs a memo.

The UI processing unit 109 may provide a search function. For example, the UI processing unit 109 may search for the monitoring image 203 to which the business kilometer is not associated from the monitoring information 207 and display it on the maintenance UI 400. As a result, the maintenance worker can efficiently input the position indicated by the landscape image 205. Further, for example, the UI processing unit 109 may search the monitoring image 203 associated with the maintenance necessity flag "necessary" from the monitoring information 207 and display it on the maintenance UI 400. As a result, the maintenance worker can efficiently confirm the monitoring image 203 of the maintenance target object that requires maintenance.

<Modifications and supplements>
In the information storage unit 101, in addition to the surveillance image 203, the distance marker image 204, and the landscape image 205 taken from the cameras 10A, 10B, and 10C provided in the vehicle 1, the surveillance image 203 taken from the camera owned by the maintenance worker , Distance marker image 204, landscape image 205 may also be stored. In this case, the monitoring information generation unit 107 may also add the monitoring image 203, the distance marker image 204, and the landscape image 205 taken by the maintenance worker to the monitoring information 207. The camera possessed by the maintenance worker may be a smartphone with a camera or a tablet terminal. Alternatively, the camera possessed by the maintenance worker may be attached to the helmet worn by the maintenance worker.

In the above description, the RTK signal is also used for positioning, but positioning may be performed only by the GNSS signal without using the RTK signal. Alternatively, the monitoring device 100 may specify the shooting position of the monitoring image 203 by using the distance marker image 204 or the landscape image 205 without using either the GNSS signal or the RTK signal.

In the above, the case where the maintenance target is a railway has been described, but the maintenance target is not limited to the railway. The above-mentioned contents can be applied to a place where a vehicle travels, such as a general road or a highway. For example, by applying the above-mentioned contents to a highway, it is possible to accurately specify the shooting position of an image taken for maintenance such as a tunnel, a shoulder or a collapse prevention wall on the highway.

Further, in the above description, the case of specifying the position by using GNSS or RTK has been described, but the position can also be specified based on the information of the route name and the business kilometer. According to this configuration, the position can be accurately specified from the business kilometers recognized from the distance marker and the route name. The route name may be registered in advance so that the business kilometer can be recognized from the distance marker.

<Hardware configuration>
The functional block of the monitoring device 100 described above may be realized by a computer program. FIG. 8 is a diagram showing a hardware configuration of a computer that realizes a functional block of the monitoring device 100 according to the present embodiment by a computer program. The computer may be read as an information processing device.

The computer 1000 includes a processor 1001, a memory 1002, a storage 1003, an input device 1004, an output device 1005, a communication device 1006, a GPU (Graphics Processing Unit) 1007, a reading device 1008, and a bus 1009.

Each of the devices 1001 to 1008 is connected to the bus 1009 and can transmit and receive data in both directions via the bus 1009.

The processor 1001 is a device that executes a computer program stored in the memory 1002 and realizes the above-mentioned functional block. The processor 1001 may be read as a control unit. As an example of the processor 1001, CPU (Central Processing Unit), MPU (Micro Processing Unit), controller, LSI (large scale integration), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field-Programmable Gate) Array).

The memory 1002 is a device for storing computer programs and data handled by the computer 1000. The memory 1002 may be read as a storage unit. The memory 1002 may include a ROM (Real-Only Memory) and a RAM (Random Access Memory). Examples of RAM include DRAM (Dynamic Random Access Memory), MRAM (Magnetoresistive Random Access Memory), and FeRAM (Ferroelectric Random Access Memory).

The storage 1003 is a device composed of a non-volatile storage medium and storing computer programs and data handled by the computer 1000. Examples of the storage 1003 include HDD (Hard Disk Drive) and SSD (Solid State Drive).

The input device 1004 is a device that receives data to be input to the processor 1001. Examples of the input device 1004 include a keyboard, a mouse, a touch pad, and a microphone.

The output device 1005 is a device that outputs data generated by the processor 1001. Examples of the output device 1005 include a display and a speaker.

The communication device 1006 is a device that transmits / receives data to / from another device represented by a server via a communication network. The communication device 1006 may include a transmitting unit for transmitting data and a receiving unit for receiving data. The communication device 1006 may support either wired communication or wireless communication. An example of wired communication is Ethernet®. Examples of wireless communication include 802.11, Bluetooth, LTE, 4G, and 5G.

The GPU 1007 is a device that processes image depiction at high speed. The GPU 1007 may be used for processing related to AI (Artificial Intelligence) (for example, deep learning) and processing related to character recognition for a distance marker or the like.

The reading device 1008 is a device that reads data from an external recording medium such as a DVD-ROM (Digital Versatile Disk Read Only Memory) or a USB memory.

The functional block of the monitoring device 100 may be realized as an LSI which is an integrated circuit. These functional blocks may be individually integrated into one chip, or may be integrated into one chip so as to include a part or all of them. Although it is referred to as LSI here, it may be referred to as IC, system LSI, super LSI, or ultra LSI depending on the degree of integration. Furthermore, if an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology or another technology derived from it, it is naturally possible to integrate functional blocks using that technology.

The video recorder 20 of the present embodiment receives the surveillance video, the distance marker video, and the landscape video from the cameras 10A, 10B, and 10C, respectively, but the video recorder 20 receives the surveillance images from the cameras 10A, 10B, and 10C. , Distance marker image, and landscape image may be received respectively. This configuration eliminates the need for processing to extract images from moving images.

Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications, modifications, substitutions, additions, deletions, and even examples within the scope of the claims. It is understood that it belongs to the technical scope of the present disclosure. Further, each component in the above-described embodiment may be arbitrarily combined as long as the gist of the invention is not deviated.

The techniques disclosed in the present disclosure are useful for monitoring and maintaining transportation infrastructure in general, including railways.

1 Vehicle 2 Surveillance system 3 GNSS satellite 4 Radio base station 5 Time server 10A, 10B, 10C Camera 20 Video recorder 30 GNSS receiver 40 RTK receiver 100 Monitoring device 101 Information storage 102 GNSS information receiver 103 Correction information receiver 104 Clock synchronization unit 105 Image acquisition unit 106 Positioning information generation unit 107 Monitoring information generation unit 108 Shooting position specification unit 109 UI processing unit 201 GNSS information 202 Correction information 203 Monitoring image 204 Distance marker image 205 Landscape image 206 Positioning information 207 Monitoring information 400 Maintenance UI for
401 Surveillance image area 402 Distance target image area 403 Landscape image area 404 Information area 405 Map area 406 Front button 407 Next button 408 Maintenance required button 409 Memo button 410 pin 1000 Computer 1001 Processor 1002 Memory 1003 Storage 1004 Input device 1005 Output device 1006 Communication Equipment 1007 GPU
1008 Reader 1009 Bus

Claims (8)

A monitoring device that monitors maintenance objects on the vehicle's travel path.
A control unit for storing the first image of the maintenance object and the second image of the periphery of the shooting position of the first image in the information storage unit in association with each other is provided.
The control unit associates the first image with the positioning information obtained by positioning the shooting position of the first image in addition to the second image.
The positioning information includes positioning accuracy indicating the accuracy of the positioning.
The control unit
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the shooting position of the first image is specified based on the second image.
Monitoring device. A monitoring device that monitors maintenance objects on the vehicle's travel path.
A control unit for storing the first image of the maintenance object and the second image of the periphery of the shooting position of the first image in the information storage unit in association with each other is provided.
The control unit associates the first image with the positioning information obtained by positioning the shooting position of the first image in addition to the second image.
The positioning information includes positioning accuracy indicating the accuracy of the positioning.
The control unit
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the second image is displayed and the user accepts the input of the shooting position of the second image.
Monitoring device. A monitoring system that monitors maintenance objects on the vehicle's travel path.
A first camera that captures the maintenance object and generates a first image,
A second camera that captures the periphery of the shooting position of the first image to generate a second image, and
The first image in which the maintenance object is photographed and the second image in which the periphery of the photographing position of the first image is photographed are associated and stored in the information storage unit.
Corresponding the positioning information for positioning the shooting position of the first image including the positioning accuracy indicating the positioning accuracy,
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, a monitoring device for specifying the shooting position of the first image based on the second image is provided.
Monitoring system. A monitoring system that monitors maintenance objects on the vehicle's travel path.
A first camera that captures the maintenance object and generates a first image,
A second camera that captures the periphery of the shooting position of the first image to generate a second image, and
The first image in which the maintenance object is photographed and the second image in which the periphery of the photographing position of the first image is photographed are associated and stored in the information storage unit.
Corresponding the positioning information for positioning the shooting position of the first image including the positioning accuracy indicating the positioning accuracy,
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the second image is provided with a monitoring device that displays the second image and accepts an input of a shooting position of the second image from a user.
Monitoring system. It is a monitoring method that monitors maintenance objects on the vehicle's movement path.
The first image in which the maintenance object is photographed and the second image in which the periphery of the photographing position of the first image is photographed are associated and stored in the information storage unit.
In addition to the first image and the second image, positioning information for positioning the shooting position of the first image is associated with each other.
The positioning information includes positioning accuracy indicating the accuracy of the positioning.
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the shooting position of the first image is specified based on the second image.
Monitoring method. It is a monitoring method that monitors maintenance objects on the vehicle's movement path.
The first image in which the maintenance object is photographed and the second image in which the periphery of the photographing position of the first image is photographed are associated and stored in the information storage unit.
In addition to the first image and the second image, positioning information for positioning the shooting position of the first image is associated with each other.
The positioning information includes positioning accuracy indicating the accuracy of the positioning.
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the second image is displayed and the user accepts the input of the shooting position of the second image.
Monitoring method. A computer program for monitoring maintenance objects on the vehicle's travel path.
The first image in which the maintenance object is photographed and the second image in which the periphery of the photographing position of the first image is photographed are associated and stored in the information storage unit.
In addition to the first image and the second image, positioning information for positioning the shooting position of the first image is associated with each other.
The positioning information includes positioning accuracy indicating the accuracy of the positioning.
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the shooting position of the first image is specified based on the second image.
Let the computer do that,
Computer program. A computer program for monitoring maintenance objects on the vehicle's travel path.
The first image in which the maintenance object is photographed and the second image in which the periphery of the photographing position of the first image is photographed are associated and stored in the information storage unit.
In addition to the first image and the second image, positioning information for positioning the shooting position of the first image is associated with each other.
The positioning information includes positioning accuracy indicating the accuracy of the positioning.
When the positioning accuracy is equal to or higher than a predetermined threshold value, the shooting position of the first image is specified based on the positioning information.
When the positioning accuracy is less than the threshold value, the second image is displayed and the user accepts the input of the shooting position of the second image.
Let the computer do that,
Computer program.

Images