JP7117645B2 - MONITORING DEVICE, MONITORING SYSTEM, MONITORING METHOD, AND COMPUTER PROGRAM - Google Patents

Description

The present disclosure relates to a monitoring device, a monitoring system, a monitoring method, and a computer program.

2. Description of the Related Art Conventionally, a camera is attached to a railroad maintenance vehicle or the like, a rail is photographed by the camera while the maintenance vehicle or the like is running, and the photographed image is used for rail maintenance.

In Patent Document 1, an infrared camera attached to a maintenance vehicle or the like captures a rail to generate an infrared image, specifies the position where the infrared image was captured from the travel distance of the maintenance vehicle, etc., and specifies the position and infrared images.

JP-A-10-73413

However, in order to specify the position from the travel distance as in Patent Document 1, it is necessary to register in advance information indicating the departure position and travel route of the maintenance vehicle or the like. In addition, since the traveled distance is calculated using the number of rotations of the wheels in the odometer provided in the maintenance vehicle or the like, it may include a large error due to wear and/or slippage of the wheels.

An object of the present disclosure is to provide a technology capable of accurately specifying the photographing position of an image photographed for maintenance or monitoring of transportation infrastructure including railways.

A monitoring device according to an aspect of the present disclosure is a monitoring device that monitors a maintenance target on a vehicle movement route, and includes a first image obtained by capturing the maintenance target, and at the capturing time of the first image. a control unit that associates a second image obtained by photographing an object indicating a distance or a location, wherein the control unit displays the first image and the second image associated with the first image.

A monitoring device according to an aspect of the present disclosure is a monitoring device that monitors a maintenance target on a vehicle movement route, and includes a first image obtained by capturing the maintenance target, and at the capturing time of the first image. a control unit that associates a second image obtained by photographing an object indicating a distance or a location, wherein the control unit associates the photographing position of the first image with the second image associated with the first image. Identify based on

A monitoring device according to an aspect of the present disclosure is a monitoring device that monitors a maintenance target on a vehicle movement route, and includes a first image obtained by capturing the maintenance target, and at the capturing time of the first image. a controller that associates the landscape image captured around the shooting position of the first image with the controller, wherein the controller displays the first image and the landscape image associated with the first image. .

A monitoring system according to one aspect of the present disclosure is a monitoring system for monitoring a maintenance target on a vehicle movement route, comprising: a first camera that captures the maintenance target to generate a first image; a second camera that captures an object indicating the distance or location at the capturing time of the first image to generate a second image; the first image capturing the maintenance object; and the capturing time of the first image. and a monitoring device that associates a second image obtained by photographing an object indicating a distance or a location with and displays the first image and the second image associated with the first image.

A monitoring system according to one aspect of the present disclosure is a monitoring system for monitoring a maintenance target on a vehicle movement route, comprising: a first camera that captures the maintenance target to generate a first image; a second camera that captures an object indicating the distance or location at the capturing time of the first image to generate a second image; the first image capturing the maintenance object; and the capturing time of the first image. and a monitoring device that associates a second image obtained by photographing an object indicating a distance or location with a monitoring device that specifies the photographing position of the first image based on the second image associated with the first image; , provided.

A monitoring system according to one aspect of the present disclosure is a monitoring system for monitoring a maintenance target on a vehicle movement route, comprising: a first camera that captures the maintenance target to generate a first image; a second camera that captures the surroundings of the capturing position of the first image at the capturing time of the first image to generate a landscape image; the first image that captures the maintenance object; a monitoring device that associates a scenery image captured around the shooting position of the first image at a shooting time, and displays the first image and the scenery image associated with the first image; Prepare.

A monitoring method according to an aspect of the present disclosure is a monitoring method for monitoring a maintenance target on a moving route of a vehicle. A second image obtained by photographing an object indicating a distance or a place is associated with the first image and the second image associated with the first image are displayed.

A monitoring method according to an aspect of the present disclosure is a monitoring method for monitoring a maintenance target on a moving route of a vehicle. A second image obtained by photographing an object indicating the distance or location is associated, and the photographing position of the first image is specified based on the second image associated with the first image.

A monitoring method according to an aspect of the present disclosure is a monitoring method for monitoring a maintenance target on a moving route of a vehicle. A scenery image photographed around the shooting position of the first image is associated with the first image, and the scenery image associated with the first image is displayed.

A computer program according to an aspect of the present disclosure is a computer program for monitoring a maintenance target on a vehicle travel route, and includes a first image obtained by capturing the maintenance target and a capturing time of the first image. and a second image obtained by photographing an object indicating the distance or location, and causing the computer to display the first image and the second image associated with the first image.

A computer program according to an aspect of the present disclosure is a computer program for monitoring a maintenance target on a vehicle travel route, and includes a first image obtained by capturing the maintenance target and a capturing time of the first image. is associated with a second image obtained by photographing an object indicating the distance or location, and the photographing position of the first image is specified based on the second image associated with the first image. let the computer do it.

A computer program according to an aspect of the present disclosure is a computer program for monitoring a maintenance target on a vehicle travel route, and includes a first image obtained by capturing the maintenance target and a capturing time of the first image. causing a computer to perform the following: .

In addition, these generic or specific aspects may be realized by a system, device, method, integrated circuit, computer program or recording medium, and any of the system, device, method, integrated circuit, computer program and recording medium may be implemented. may be implemented in any combination.

Advantageous Effects of Invention According to the present disclosure, it is possible to accurately identify the photographing position of an image photographed for maintenance or monitoring of transportation infrastructure including railways.

FIG. 1 is a block diagram showing a configuration example of a monitoring system according to this embodiment; FIG. 1 is a functional block diagram showing a configuration example of a monitoring device according to this embodiment; A diagram showing a configuration example of monitoring information according to the present embodiment A diagram for explaining an example of a method of associating a monitoring image with positioning information. Diagram for explaining an example of a method of estimating positioning information Flowchart showing an example of a process for specifying a shooting position of a surveillance image A diagram showing a display example of a maintenance UI (User Interface) according to the present embodiment. FIG. 3 is a diagram showing the hardware configuration of a computer that realizes the functional blocks of the monitoring device according to the present embodiment by means of a computer program;

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art. It should be noted that the accompanying drawings and the following description are provided to allow those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

(this embodiment)
<Structure of monitoring system>
FIG. 1 is a block diagram showing a configuration example of a monitoring system according to this embodiment.

The monitoring system 2 is a system for monitoring railway maintenance objects. Examples of railway maintenance objects include rails, rail fasteners, sleepers, roadbeds, roadbeds, overhead wires, overhead wire utility poles, insulators, balancers, traffic signals, inner walls of tunnels, bridges, slopes, and the like. In addition, the “railway” according to the present disclosure is not limited to tracks on which general electric trains or trains run, but also includes tracks on which trams run, tracks on which monorails run, tracks on which bullet trains run, and the like.

The monitoring system 2 is mounted on a railroad maintenance vehicle or the like (hereinafter referred to as “vehicle”) 1 . The vehicle 1 may be a motorized vehicle or a vehicle (bogie) towed by a motorized vehicle. Note that part of the monitoring system 2 may be installed at a location different from the vehicle 1 .

The monitoring system 2 includes cameras 10A, 10B, 10C, a video recorder 20, a global positioning satellite (hereinafter referred to as GNSS (Global Navigation Satellite System)) receiver 30, an RTK (Real Time Kinematic) receiver 40, and a monitor A device 100 is provided.

The cameras 10A, 10B, and 10C are compatible with PoE (Power over Ethernet) and may be connected to the video recorder 20 via Ethernet cables. However, the cameras 10A, 10B, and 10C may be connected to the video recorder 20 by a wired cable (eg, USB (Universal Serial Bus) cable) different from the Ethernet cable or by a wireless system (eg, IEEE802.11 or Bluetooth). Also, the connection method between the cameras 10A, 10B, and 10C and the video recorder 20 is not limited to PoE. For example, the cameras 10A, 10B, 10C may be supplemented with auxiliary power through other connection schemes.

The video recorder 20 is connected to the monitoring device 100 by, for example, an Ethernet cable. However, the video recorder 20 may be connected to the monitoring device 100 by a wired cable (eg, USB cable) different from the Ethernet cable or by a wireless system (eg, IEEE802.11 or Bluetooth).

The GNSS receiver 30 is connected to the monitoring device 100 by, for example, a USB cable. However, the GNSS receiver 30 may be connected to the monitoring device 100 by a wired cable or a wireless method different from the USB cable.

The RTK receiving device 40 is connected to the monitoring device 100 by, for example, a USB cable. However, the RTK receiving device 40 may be connected to the monitoring device 100 by a wired cable or a wireless system different from the USB cable.

The camera 10A is installed in the vehicle 1 so as to photograph the maintenance object. The number of installed cameras 10A is not limited to one, and may be two or more. The moving image data generated by the camera 10A capturing images from the running vehicle 1 is referred to as a monitoring moving image. Examples of surveillance video formats include MPEG2, MPEG4, H. 264, H. H.265, JPEG, and uncompressed data.

The camera 10B is installed on the vehicle 1 so as to photograph distance markers existing around the track. The number of installed cameras 10B is not limited to one, and may be two or more. The moving image data generated by the camera 10B shooting the moving vehicle 1 will be referred to as a distance marker moving image. The format of the distance marker video may be the same as the surveillance video. A distance marker is a sign that indicates the distance from a certain starting point on a railway. The distance marker often has characters indicating the distance, but the characters may not be indicated. Note that the "characters" of the distance marker may include numbers, symbols, and the like. Further, the distance marker is not limited to a marker that indicates a distance from a certain starting point, and may be a marker that indicates a distance, an interval of distances, or a specific location.

The camera 10C is installed on the vehicle 1 so as to photograph the scenery seen from the vehicle 1. - 特許庁The number of installed cameras 10C is not limited to one, and may be two or more. The moving image data generated by the camera 10C shooting the moving vehicle 1 is referred to as scenery moving image. The format of the scenery video may be the same as that of the surveillance video. The camera 10</b>C may be installed so as to photograph the scenery behind, in front of, or to the side of the vehicle 1 . The camera 10C may be installed inside the vehicle 1 or outside the vehicle 1 (for example, on the roof of the vehicle 1).

The camera 10A, the camera 10B, and the camera 10C may be installed on the vehicle 1 so as to capture images in different directions. As a result, as described above, it is possible to photograph different things such as maintenance objects, distance markers, and landscapes. Alternatively, if one camera can capture both the distance marker and the scenery, the camera 10B may capture a moving image including both the distance marker and the scenery without installing the camera 10C.

The video recorder 20 is a device that receives and records a monitoring video, a distance marker video, and a scenery video from the cameras 10A, 10B, and 10C.

The GNSS receiver 30 periodically receives GNSS signals, which are positioning signals emitted from the GNSS satellites 3, and generates GNSS information 201 (see FIG. 2) from the received GNSS signals. The GNSS information 201 includes the reception time of the GNSS signal, the latitude and longitude of the GNSS receiver 30 at the reception time, and the positioning accuracy of the latitude and longitude in association with each other. The GNSS receiver 30 transmits the generated GNSS information 201 to the monitoring device 100 . The measured latitude and longitude are hereinafter referred to as positioning coordinates.

The RTK receiver 40 periodically receives an RTK signal, which is a positioning signal transmitted from the radio base station 4, and generates correction information 202 (see FIG. 2) from the received RTK signal. The correction information 202 includes information for correcting the positioning coordinates indicated by the GNSS signal. The positioning coordinates corrected using the RTK signal indicate a more accurate position than the positioning coordinates before correction. The RTK receiving device 40 transmits the generated correction information 202 to the monitoring device 100 .

The monitoring device 100 receives GNSS information 201 and correction information 202 from the GNSS receiver 30 and the RTK receiver 40, respectively. Based on the received GNSS information 201 and correction information 202, the monitoring device 100 generates information (hereinafter referred to as “positioning information”) 206 indicating the positioning coordinates of the camera 10A at the reception time included in the GNSS information 201 and the correction information 202. to generate For example, the positioning coordinates of the camera 10A can be obtained from the GNSS information 201 by installing the GNSS receiver 30 and the camera 10A so as to be aligned in the height direction. Alternatively, the positional deviation between the GNSS receiver 30 and the camera 10A is registered in advance, and the positioning coordinates of the camera 10A are obtained by correcting the positioning coordinates indicated by the GNSS information 201 with the registered positional deviation. be able to.

Based on at least one of the time information included in the GNSS information 201 and the time information received from the time server 5 (for example, an NTP (Network Time Protocol) server), the monitoring device 100 detects the internal set the clock; Thereby, the internal clock of the monitoring device 100 is synchronized with the correct time. In addition, the monitoring device 100 synchronizes the internal clocks of the video recorder 20 and the cameras 10A, 10B, and 10C with the internal clocks of the monitoring device 100 . Thereby, the internal clocks of the cameras 10A, 10B, 10C, the video recorder 20, and the monitoring device 100 are synchronized with the correct time.

The monitoring apparatus 100 receives from the video recorder 20 each image frame (hereinafter referred to as "monitoring image") 203 (see FIG. 2) constituting the monitoring video and each image frame (hereinafter "distance mark image") constituting the distance mark moving image. ) 204 (refer to FIG. 2) and each image frame (hereinafter referred to as “landscape image”) 205 (refer to FIG. 2) constituting the landscape moving image are acquired and stored.

The monitoring apparatus 100 identifies the captured position of the monitoring image 203 using the positioning information 206 obtained at the same time as the monitoring image 203 was captured. As a result, it is possible to accurately identify the imaging position of the monitoring image 203 . However, when the vehicle 1 is traveling in a mountainous area, an urban area, or in a tunnel, the positioning information 206 cannot be used because the GNSS signal cannot be received or the positioning accuracy of the GNSS signal is insufficient. Sometimes. In such a case, the monitoring apparatus 100 uses at least one of the distance marker image 204 and the landscape image 205 captured at the same time as the monitoring image 203 is captured to specify the capturing position of the monitoring image 203. do. As a result, even if the positioning information 206 cannot be used, the imaging position of the monitoring image 203 can be specified with high accuracy. Note that in the description of the present disclosure, the expression “same time as the shooting time” does not strictly represent the same time, but represents a time included within a predetermined period including the shooting time. Next, the above contents will be described in detail.

<Structure of monitoring device>
FIG. 2 is a functional block diagram showing a configuration example of the monitoring device 100 according to this embodiment.

The monitoring device 100 includes an information storage unit 101, a GNSS information reception unit 102, a correction information reception unit 103, a clock synchronization unit 104, an image acquisition unit 105, a positioning information generation unit 106, a monitoring information generation unit 107, an imaging position identification unit 108, and a UI processing unit 109 .

The information storage unit 101 may be implemented by the memory 1002, the storage 1003, or a combination thereof shown in FIG. The information storage unit 101 stores GNSS information 201 , correction information 202 , monitoring images 203 , distance marker images 204 , scenery images 205 , positioning information 206 and monitoring information 207 .

The GNSS information receiving unit 102, the correction information receiving unit 103, the clock synchronization unit 104, the image acquisition unit 105, the positioning information generation unit 106, the monitoring information generation unit 107, the shooting position identification unit 108, and the UI processing unit 109 are shown in FIG. may be realized by cooperation of the processor 1001 shown in FIG. Therefore, the processing mainly performed by these functional blocks 102 to 109 can be read as the processing mainly performed by the processor 1001 .

The GNSS information receiver 102 periodically receives GNSS information 201 from the GNSS receiver 30 and stores it in the information storage 101 .

The correction information reception unit 103 periodically receives the correction information 202 from the RTK reception device 40 and stores it in the information storage unit 101 .

The clock synchronization unit 104 sets the internal clock of the monitoring device 100 based on at least one of the time information included in the GNSS information 201 and the correction information 202 and the time information received from the time server 5 . For example, when time information is obtained from at least one of the GNSS information 201 and the correction information 202, the clock synchronization unit 104 sets the internal clock of the monitoring device 100 using the time information. If the time information is not obtained from either the GNSS information 201 or the correction information 202, the clock synchronization unit 104 obtains the time information from the time server 5, and uses the time information to set the internal clock of the monitoring device 100. do.

Additionally, the clock synchronization unit 104 synchronizes the internal clocks of the video recorder 20 and the cameras 10A, 10B, and 10C with the internal clock of the monitoring device 100 . Thereby, the internal clocks of the cameras 10A, 10B, 10C, the video recorder 20, and the monitoring device 100 are synchronized with the correct time. Therefore, the photographing times of the monitoring image 203, the distance marker image 204, and the landscape image 205 all represent accurate times.

The image acquisition unit 105 acquires the monitoring image 203 , the distance marker image 204 and the landscape image 205 from the video recorder 20 and stores them in the information storage unit 101 .

Based on the GNSS information 201 and the correction information 202, the positioning information generation unit 106 generates the positioning information 206 indicating the positioning coordinates (latitude and longitude) of the camera 10A at the reception time included in the GNSS information 201 and the correction information 202. and stored in the information storage unit 101 .

The monitoring information generation unit 107 generates a monitoring image 203, a distance marker image 204 and a landscape image 205 captured at the same time as the monitoring image 203 was captured, and positioning information 206 positioned at the same time as the capturing time. are associated to generate monitoring information 207 . Details of the monitoring information 207 will be described later (see FIG. 3).

The shooting position specifying unit 108 uses at least one of the positioning information 206, the distance marker image 204, and the scenery image 205, which are associated with the monitoring image 203 in the monitoring information 207, to determine the shooting position of the monitoring image 203. Identify. The photographing position of the surveillance image 203 may be expressed as a business kilometer indicating the distance from a predetermined starting point on the railway line.

If the positioning accuracy of the positioning information 206 associated with the monitoring image 203 in the monitoring information 207 is sufficient (for example, the positioning accuracy is equal to or greater than a predetermined threshold value), the imaging position specifying unit 108 determines the positioning of the positioning information 206. A business kilometer corresponding to the coordinates may be used as the photographing position of the monitoring image 203 . For example, the shooting position specifying unit 108 converts the positioning coordinates of the positioning information 206 into operating kilometers using a predetermined database that converts the positioning coordinates on the orbit into operating kilometers.

The imaging position specifying unit 108 detects that the positioning information 206 is not associated with the monitoring image 203 in the monitoring information 207, or that the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is insufficient (for example, If the positioning accuracy is less than a predetermined threshold), the following processing may be performed. That is, the photographing position specifying unit 108 uses at least one of the distance marker image 204 and the scenery image 205 associated with the surveillance image 203 in the monitoring information 207 to determine the operating kilometer of the photographing position of the surveillance image 203 . may be specified. The details of this specifying method will be described later.

The UI processing unit 109 displays the maintenance UI 400 (see FIG. 7) based on the monitoring information 207, and receives operations on the maintenance UI 400. FIG. For example, the UI processing unit 109 displays a maintenance UI 400 on a display, which is an example of the output device 1005 shown in FIG. The UI processing unit 109 receives operations on the maintenance UI 400 through the keyboard and mouse, which are examples of the input device 1004 shown in FIG. Details of the maintenance UI 400 will be described later (see FIG. 7).

<Structure of monitoring information>
FIG. 3 is a diagram showing a configuration example of the monitoring information 207 according to this embodiment.

The monitoring information 207 has items of monitoring image, distance marker image, landscape image, photographing time, positioning information, route name, operating kilometer, maintenance necessity flag, maintenance factor, photographing means, photographer, and memo. In other words, the monitoring information 207 can be said to be information indicating the correspondence between these items.

The ID (Identification) of the monitoring image 203 is registered in the monitoring image item. An example of the ID of the monitoring image 203 is the file name of the monitoring image 203 .

The ID of the distance mark image 204 is registered in the item of the distance mark image. An example of the ID of the distance mark image 204 is the file name of the distance mark image 204 .

The ID of the scenery image 205 is registered in the item of scenery image. An example of the ID of the landscape image 205 is the file name of the landscape image 205 .

The shooting time of the monitoring image 203 is registered in the item of shooting time. Note that the shooting time item may include not only the time but also the date. In addition, by displaying the shooting time in milliseconds or microseconds according to the shooting fps, the shooting timing of each image can be confirmed in detail. As a result, even if the fps value increases, it is possible to prevent images shot at different timings from being recognized as having been shot at the same time. For example, for images shot every 1 millisecond, if the shooting time is displayed only in seconds, it is impossible to know the shooting order of each image. You can figure out the order.

In the item of positioning information, the positioning coordinates measured at the same time as the imaging time of the monitoring image 203 and the positioning accuracy of the positioning coordinates are registered. Positioning accuracy may be represented by a numerical value that indicates the degree of accuracy. Alternatively, the positioning accuracy can be expressed in three levels: "good" indicating sufficient accuracy, "poor" indicating insufficient accuracy, and "improper" indicating that positioning was not possible. good.

The name of the route on which the monitoring image 203 was captured is registered in the line name item.

The business km of the shooting position of the monitoring image 203 is registered in the business km item.

A flag indicating whether maintenance work is required for the maintenance object shown in the monitoring image 203 is registered in the maintenance necessity flag item. The maintenance worker may look at the monitoring image 203 displayed on the maintenance UI 400, which will be described later, and determine whether maintenance work is required for the maintenance object in the monitoring image 203. FIG. Alternatively, the monitoring apparatus 100 may analyze the monitoring image 203 and automatically determine whether maintenance work is required for the maintenance target in the monitoring image 203 . A flag indicating the result of the determination may be registered in the maintenance necessity flag item. For example, the maintenance necessity flag may be registered with "necessary" indicating that maintenance work is required or "unnecessary" indicating that maintenance work is not required.

In the item of maintenance factor, when the maintenance necessity flag is "required", the factor that maintenance work is required is registered. For example, when the equipment of the maintenance object is abnormal, "equipment abnormality" may be registered in the maintenance factor item. For example, if the maintenance target is missing, "missing" may be registered in the maintenance factor item.

The shooting means of the monitoring image 203 is registered in the shooting means item. For example, when the monitoring image 203 is captured by the camera 10A mounted on a maintenance vehicle or the like, "maintenance vehicle or the like" may be registered in the imaging means item. For example, if the monitoring image 203 is captured by a camera-equipped terminal possessed by a maintenance worker, "hand terminal" may be registered in the photographing means item.

The photographer of the monitoring image 203 is registered in the item of photographer. For example, if the monitoring image 203 was taken by a maintenance worker whose name is "Hanako Pana", "Hanako Pana" is registered in the item of photographer. For example, if the monitoring image 203 was taken by a maintenance worker whose name is "Taro Matsushita", "Taro Matsushita" is registered in the item of photographer.

The content of the memo for the monitoring image 203 is registered in the memo item. For example, when a maintenance worker views a monitoring image 203 displayed on the maintenance UI 400 (to be described later) and inputs a memo, the content of the input memo is registered in the memo item.

<Association of monitoring images with positioning information>
FIG. 4 is a diagram for explaining an example of a method of associating the monitoring image 203 and the positioning information 206. As shown in FIG.

As shown in FIG. 3 , the monitoring information generation unit 107 generates monitoring information 207 by associating a monitoring image 203 with positioning information 206 positioned at the same time as when the monitoring image 203 was captured. Thereby, the shooting position of the monitoring image 203 can be specified from the positioning information 206 associated with the monitoring image 203 .

However, the frame rate of the surveillance video and the reception frequency of the GNSS signals may differ. For example, it is assumed that the frame rate of the monitoring video is 30 fps and the reception frequency of the GNSS signal is 10 Hz. In this case, the frame cycle of the monitoring video is (1/30) second, and the positioning cycle by the GNSS signal is (1/10) second. cannot be matched.

Therefore, if there is positioning information 206 that was positioned at the same time as the time when the monitoring image 203 was captured, the capturing position specifying unit 108 associates the monitoring image 203 with the positioning information 206, If the positioning information 206 positioned at the same time does not exist, the positioning information 206 corresponding to the monitoring image 203 is estimated based on the positioning information 206 positioned at another time.

For example, as shown in FIG. 4, when the monitoring image 203 is captured at times t1, t2, t3, and t4, respectively, and the positioning information 206 is measured at times t1 and t4, the capturing position specifying unit 108 , do the following: That is, the photographing position specifying unit 108 associates the monitoring image 203 photographed at time t1 with the positioning information 206 positioned at time t1, and associates the surveillance image 203 photographed at time t4 with the positioning information 206 positioned at time t4. Positioning information 206 is associated. In addition, the shooting position specifying unit 108 estimates the positioning information 206 at time t2 and time t3 based on the positioning information 206 at time t1 and time t4. Next, the estimation method will be described.

FIG. 5 is a diagram for explaining an example of a positioning information estimation method.

For example, as shown in FIG. 5, the photographing position specifying unit 108 divides a line segment L connecting a position P1 indicated by the positioning information 206 at time t1 and a position P4 indicated by the positioning information 206 at time t4 into three segments. Get P2 and P3. Note that the number of divisions may be calculated based on a value obtained by dividing the positioning period by the period of the monitoring image 203 . Also, the shape of the line segment L may follow the linear shape of the rail.

The shooting position specifying unit 108 estimates the division point P2 as the positioning information 206 at time t2, and the division point P3 as the positioning information 206 at time t3. Then, in the monitoring information 207, the imaging position specifying unit 108 associates the monitoring image 203 at time t2 with the estimated positioning information 206 at time t2, and associates the monitoring image 203 at time t3 with the estimated positioning information at time t3. Information 206 is associated.

Thereby, even if the frame cycle of the monitoring video and the positioning cycle by the GNSS signal are different, the positioning information 206 can be associated with each monitoring image 203 .

<When positioning accuracy is insufficient>
Since the GNSS signal cannot be received while the vehicle 1 is traveling in a tunnel, underground, or the like, the positioning information generation unit 106 cannot generate the positioning information 206 during that time. Therefore, in the monitoring information 207, the positioning information 206 may not be associated with the monitoring image 203 captured while the vehicle 1 is traveling in a tunnel, underground, or the like. In this case, the imaging position of the monitoring image 203 cannot be specified from the positioning information 206. FIG.

In addition, while the vehicle 1 is traveling in an urban area, a mountainous area, or the like, the GNSS signal can be received but the RTK signal cannot be received, or the reception accuracy of the GNSS signal is insufficient. The positioning accuracy of the information 207 may be insufficient. Therefore, in the monitoring information 207, the positioning information 206 associated with the monitoring image 203 captured while the vehicle 1 is traveling in an urban area, a mountainous area, or the like may have insufficient positioning accuracy. . In this case, the shooting position of the monitoring image 203 specified from the positioning information 206 may deviate greatly from the actual shooting position.

Therefore, the imaging position specifying unit 108 determines whether the monitoring image 203 associated with the monitoring information 207 is not associated with the positioning information 206 or is associated with the positioning information 206 with insufficient positioning accuracy (less than a predetermined threshold). , the photographing position of the surveillance image 203 is specified using at least one of the range marker image 204 and the scenery image 205 photographed at the same time as the surveillance image 203 was photographed.

For example, the shooting position specifying unit 108 specifies the shooting position from the distance marker image 204 as follows. That is, the photographing position specifying unit 108 performs character recognition on the distance post image 204 photographed at the same time as the monitoring image 203 is photographed, and reads the characters of the distance post. The photographing position specifying unit 108 specifies the operating kilometer indicated by the read distance marker from the read character of the distance marker. In the monitoring information 207, the photographing position specifying unit 108 associates the specified operating kilometers with the surveillance image 203 photographed at the same time as the distance mark image 204 was photographed.

For example, the shooting position specifying unit 108 specifies the shooting position from the scenery image 205 as follows. That is, the photographing position specifying unit 108 performs object recognition on the scenery image 205 photographed at the same time as the monitoring image 203 is photographed, and recognizes the object. The photographing position specifying unit 108 obtains the position where the recognized object exists by querying a predetermined database having a correspondence relationship between the recognized object and the position where the object exists. In the surveillance information 207, the photographing position specifying unit 108 associates the surveillance image 203 photographed at the same time as the landscape image 205 with the operating kilometer of the position where the object exists. Examples of identifiable objects captured in landscape images include tunnels, bridges, station buildings, shops, billboards, signs, bridges, and landmarks.

For example, the shooting position specifying unit 108 specifies the shooting position from the landscape image 205 in cooperation with the UI processing unit 109 as follows. That is, the UI processing unit 109 displays a landscape image 205 captured at the same time as the monitoring image 203 was captured, and receives an input of a position indicated by the landscape image 205 from the maintenance worker. The maintenance worker looks at the displayed scenery image 205 and inputs the position where the scenery image 205 was photographed. In the monitoring information 207, the photographing position specifying unit 108 associates the surveillance image 203 photographed at the same time as the landscape image 205 with the operating kilometers of the input position.

<Processing for Identifying Shooting Position of Monitoring Image>
FIG. 6 is a flow chart showing an example of processing for specifying the shooting position of the monitoring image 203. As shown in FIG.

The shooting position specifying unit 108 selects one monitoring image 203 from among the plurality of monitoring images 203 included in the monitoring information 207 (S101).

The imaging position specifying unit 108 acquires the positioning information 206 associated with the monitoring image 203 selected in S101 in the monitoring information 207 (that is, measured at the same time as the imaging time of the monitoring image 203) (S102). ).

The monitoring information generation unit 107 determines whether or not the positioning accuracy included in the positioning information 206 acquired in S102 is equal to or greater than a predetermined threshold (S103).

If the positioning accuracy is equal to or greater than the threshold (S103: YES), the shooting position specifying unit 108 specifies operating kilometers in the positioning coordinates of the positioning information 206 acquired in S102 (S104). Then, the process of S112 is executed.

If the positioning accuracy is less than the threshold value, or if the monitoring image 203 selected in S101 is not associated with the positioning information 206 (S103: NO), the imaging position specifying unit 108 performs the following processing. That is, the imaging position specifying unit 108 acquires the distance marker image 204 associated with the monitoring image 203 selected in S101 in the monitoring information 207 (that is, captured at the same time as the monitoring image 203 was captured). (S105).

The photographing position specifying unit 108 performs character recognition of the distance mark on the distance mark image 204 selected in S105 (S106). The shooting position specifying unit 108 determines whether or not character recognition of the distance marker has succeeded in S106 (S107).

If character recognition of the distance marker is successful (S107: YES), the photographing position identification unit 108 identifies operating kilometers indicated by the characters of the recognized distance marker (S108). Then, the process of S112 is executed.

When the character recognition of the distance marker fails (S107: NO), the shooting position specifying unit 108 cooperates with the UI processing unit 109 to perform the following processing. That is, the UI processing unit 109 uses the landscape image associated with the monitoring image 203 selected in S101 in the monitoring information 207 (that is, captured at the same time as the monitoring image 203) as the maintenance UI 400. 205 is displayed, and the input of the position indicated by the landscape image 205 is accepted (S109).

The maintenance worker inputs position information indicated by the displayed landscape image 205 (S110). The photographing position specifying unit 108 specifies the operating kilometers of the position input in S110 (S111). Then, the process of S112 is executed.

In the monitoring information 207, the imaging position specifying unit 108 associates the operating kilometers specified in S104, S108, or S111 with the monitoring image 203 selected in S101 (S112). After the monitoring apparatus 100 executes the above-described processing for each monitoring image 203 included in the monitoring information 207, the processing ends.

According to the above processing, if the positioning information 206 associated with the monitoring image 203 has sufficient positioning accuracy, the shooting position specifying unit 108 uses the positioning information 206 to determine the operating kilometer of the shooting position of the monitoring image 203 . identify. In addition, when the positioning accuracy of the positioning information 206 associated with the monitoring image 203 is insufficient, or when the positioning information 206 is not associated with the monitoring image 203, the imaging position specifying unit 108 Using the distance marker image 204 or landscape image 205 associated with , the operating kilometers of the shooting position of the surveillance image 203 are specified. As a result, it is possible to specify the operating kilometer of the shooting position of the surveillance image 203 regardless of where the surveillance image 203 was shot.

<Display of maintenance UI>
FIG. 7 is a diagram showing a display example of the maintenance UI 400 according to this embodiment.

The UI processing unit 109 displays an image of the maintenance UI 400 as shown in FIG. 7 on a display, which is an example of the output device 1005 . The maintenance UI 400 includes a monitoring image area 401 , a distance marker image area 402 , a landscape image area 403 , an information area 404 , a map area 405 , a previous button 406 , a next button 407 , a maintenance required button 408 and a memo button 409 .

One of the monitoring images 203 included in the monitoring information 207 is displayed in the monitoring image area 401 .

A distance mark image 204 associated with the monitoring image 203 displayed in the monitoring image area 401 in the monitoring information 207 is displayed in the distance mark image area 402 .

A landscape image 205 associated with the monitoring image 203 displayed in the monitoring image area 401 in the monitoring information 207 is displayed in the landscape image area 403 .

Information about the monitoring image 203 displayed in the monitoring image area 401 is displayed in the information area 404 . For example, the file name, shooting date, shooting time, photographer, memo, business kilometers, and latitude and longitude indicating the shooting position of the surveillance image 203 are displayed. The UI processing unit 109 may refer to the monitoring information 207 and display this information in the information area 404 .

In the map area 405, a peripheral map of the imaging position of the monitoring image 203 displayed in the monitoring image area 401 and a pin 410 indicating the imaging position are displayed.

Note that the UI for receiving the input of the position indicated by the landscape image 205, described in S110 of FIG. 6, may be the same as the UI for maintenance 400. FIG. In this case, the maintenance worker can input the position information indicated by the landscape image 205 by looking at the landscape image 205 displayed in the landscape image area 403 and placing a pin 410 on the position on the map indicated by the landscape.

When the previous button 406 is pressed, the UI processing unit 109 moves the monitoring image 203, the distance marker image 204, and the scenery image 205, which are associated with the immediately previous shooting time in the monitoring information 207, into the monitoring image area. 401, distance mark image area 402, and scenery image area 403, respectively. In addition, the UI processing unit 109 displays in the map area 405 a peripheral map of the shooting position of the monitoring image 203 associated with the previous shooting time and a pin 410 indicating the shooting position.

When the next button 407 is pressed, the UI processing unit 109 moves the monitoring image 203, the distance marker image 204, and the scenery image 205, which are associated with the next shooting time in the monitoring information 207, into the monitoring image area. 401, distance mark image area 402, and scenery image area 403, respectively. In addition, the UI processing unit 109 displays in the map area 405 a peripheral map of the shooting position of the monitoring image 203 associated with the next shooting time and a pin 410 indicating the shooting position.

When the maintenance required button 408 is pressed, the UI processing unit 109 changes the maintenance necessity flag associated with the displayed monitoring image 203 in the monitoring information 207 to "required".

When the memo button 409 is pressed, the UI processing unit 109 displays a memo input UI (not shown). In addition, the UI processing unit 109 registers the content input to the memo input UI in the memo item associated with the monitoring image 203 being displayed in the monitoring information 207 .

For example, the maintenance worker presses the next button 407 to switch the displayed monitoring images 203 and confirms whether or not there is a problem with the maintenance object. The maintenance worker presses the maintenance required button 408 when there is a problem with the maintenance object. Also, if the maintenance worker wants to add a comment to the maintenance object, he or she presses the memo button 409 to enter a memo.

The UI processing unit 109 may provide a search function. For example, the UI processing unit 109 may search the monitoring information 207 for a monitoring image 203 not associated with a business kilometer and display it on the maintenance UI 400 . Thereby, the maintenance worker can efficiently input the position indicated by the scenic image 205 . Further, for example, the UI processing unit 109 may search the monitoring image 203 associated with the maintenance necessity flag “required” from the monitoring information 207 and display it on the maintenance UI 400 . As a result, the maintenance worker can efficiently check the monitoring image 203 of the maintenance object requiring maintenance.

<Modification and Supplement>
The information storage unit 101 stores monitoring images 203 captured by the cameras 10A, 10B, and 10C provided in the vehicle 1, distance marker images 204, and landscape images 205, as well as monitoring images 203 captured by cameras possessed by maintenance workers. , a distance marker image 204 and a landscape image 205 may also be stored. In this case, the monitoring information generation unit 107 may add to the monitoring information 207 the monitoring image 203, the distance marker image 204, and the landscape image 205 captured by the maintenance worker. The camera possessed by the maintenance worker may be a camera-equipped smartphone or tablet terminal. Alternatively, the camera carried by the maintenance worker may be attached to the helmet worn by the maintenance worker.

Although the RTK signal is also used for positioning in the above description, positioning may be performed using only the GNSS signal without using the RTK signal. Alternatively, the monitoring device 100 may identify the capturing position of the monitoring image 203 using the range marker image 204 or the landscape image 205 without using either the GNSS signal or the RTK signal.

In the above description, the case where the maintenance object is the railway has been described, but the maintenance object is not limited to the railway. What has been described above can be applied to locations where the vehicle travels, such as general roads or highways. For example, by applying the above-described content to an expressway, it is possible to accurately specify the photographing positions of images photographed for maintenance, such as tunnels, road shoulders, or collapsing prevention walls on the expressway.

Moreover, although the case where the position is specified using GNSS or RTK has been described above, the position can also be specified based on the information on the route name and operating kilometers. According to this configuration, it is possible to accurately specify the position from the operating kilometers and the route name recognized from the milepost or the like. It should be noted that the route name may be registered in advance, and the operating kilometers may be recognized from the distance marker.

<Hardware configuration>
The functional blocks of the monitoring device 100 described above may be realized by a computer program. FIG. 8 is a diagram showing the hardware configuration of a computer that implements the functional blocks of monitoring apparatus 100 according to this embodiment by means of a computer program. Note that the computer may be read as an information processing device.

Computer 1000 includes processor 1001 , memory 1002 , storage 1003 , input device 1004 , output device 1005 , communication device 1006 , GPU (Graphics Processing Unit) 1007 , reader 1008 , and bus 1009 .

Each device 1001-1008 is connected to a bus 1009 and can transmit and receive data bi-directionally over the bus 1009. FIG.

A processor 1001 is a device that executes a computer program stored in a memory 1002 and implements the functional blocks described above. The processor 1001 may be read as a control unit. Examples of the processor 1001 include CPU (Central Processing Unit), MPU (Micro Processing Unit), controller, LSI (large scale integration), ASIC (Application Specific Integrated Circuit), PLD (Programmable Logic Device), FPGA (Field-Programmable Gate Array).

Memory 1002 is a device that stores computer programs and data handled by computer 1000 . The memory 1002 may be read as a storage unit. The memory 1002 may include ROM (Real-Only Memory) and RAM (Random Access Memory). Examples of RAM include DRAM (Dynamic Random Access Memory), MRAM (Magnetoresistive Random Access Memory), and FeRAM (Ferroelectric Random Access Memory).

The storage 1003 is configured by a non-volatile storage medium and is a device that stores computer programs and data handled by the computer 1000 . Examples of the storage 1003 include HDDs (Hard Disk Drives) and SSDs (Solid State Drives).

The input device 1004 is a device that receives data to be input to the processor 1001 . Examples of input devices 1004 include keyboards, mice, touchpads, and microphones.

The output device 1005 is a device that outputs data generated by the processor 1001 . Examples of the output device 1005 include a display and speakers.

The communication device 1006 is a device that transmits and receives data to and from another device represented by a server via a communication network. Communication device 1006 may include a transmitter for transmitting data and a receiver for receiving data. The communication device 1006 may support both wired communication and wireless communication. An example of wired communication is Ethernet (registered trademark). Examples of wireless communication include IEEE802.11, Bluetooth, LTE, 4G, 5G.

The GPU 1007 is a device that processes image rendering at high speed. Note that the GPU 1007 may be used for processing related to AI (Artificial Intelligence) (for example, deep learning) and processing related to character recognition for distance markers and the like.

The reading device 1008 is a device that reads data from an external recording medium such as a DVD-ROM (Digital Versatile Disk Read Only Memory) or USB memory.

Note that the functional blocks of the monitoring device 100 may be implemented as an LSI, which is an integrated circuit. These functional blocks may be made into one chip individually, or may be made into one chip so as to include part or all of them. Although LSI is used here, it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration. Furthermore, if an integration technology that replaces the LSI appears due to advances in semiconductor technology or another technology derived from it, that technology may naturally be used to integrate the functional blocks.

Note that the video recorder 20 of the present embodiment receives the monitoring video, the distance marker video, and the scenery video from the cameras 10A, 10B, and 10C, respectively, but the video recorder 20 receives the monitoring video from the cameras 10A, 10B, and 10C. , a range marker image, and a landscape image, respectively. This configuration eliminates the need for processing for extracting images from moving images.

Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is obvious that a person skilled in the art can conceive of various modifications, modifications, substitutions, additions, deletions, and equivalents within the scope of the claims. It is understood that it belongs to the technical scope of the present disclosure. Also, the components in the above-described embodiments may be combined arbitrarily without departing from the spirit of the invention.

The technology of the present disclosure is useful for monitoring and maintaining transportation infrastructure in general, including railways.

1 Vehicle 2 Monitoring System 3 GNSS Satellite 4 Radio Base Station 5 Time Server 10A, 10B, 10C Camera 20 Video Recorder 30 GNSS Receiver 40 RTK Receiver 100 Monitoring Device 101 Information Storage Unit 102 GNSS Information Receiver 103 Correction Information Receiver 104 Clock synchronization unit 105 Image acquisition unit 106 Positioning information generation unit 107 Monitoring information generation unit 108 Shooting position identification unit 109 UI processing unit 201 GNSS information 202 Correction information 203 Monitoring image 204 Distance mark image 205 Scenery image 206 Positioning information 207 Monitoring information 400 Maintenance UI for
401 Surveillance image area 402 Range marker image area 403 Landscape image area 404 Information area 405 Map area 406 Previous button 407 Next button 408 Maintenance required button 409 Memo button 410 Pin 1000 Computer 1001 Processor 1002 Memory 1003 Storage 1004 Input device 1005 Output device 1006 Communication Device 1007 GPU
1008 reader 1009 bus

Claims (7)

A monitoring device for monitoring a maintenance object on a moving route of a vehicle,
a control unit for storing in an information storage unit a first image obtained by photographing the maintenance object and a second image obtained by photographing the object indicating the distance or location at the photographing time of the first image in association with each other;
The control unit associates, in addition to the first image and the second image, positioning information obtained by measuring the photographing position of the first image,
The positioning information includes positioning accuracy indicating the positioning accuracy,
The control unit
If the positioning accuracy is equal to or greater than a predetermined threshold, specifying the shooting position of the first image based on the positioning information;
If the positioning accuracy is less than the threshold, specifying the shooting position of the first image based on the second image;
surveillance equipment. The control unit
In addition to the first image, the second image, and the positioning information, associating a scenery image captured around the shooting position of the first image at the shooting time of the first image,
displaying the scenery image associated with the first image when the positioning information is less than the threshold and the shooting position of the first image cannot be specified based on the second image;
A monitoring device according to claim 1 . When the positioning information is less than the threshold value and the photographing position of the first image cannot be specified based on the second image, the controller determines the scenic image associated with the first image. display and receive an input of the shooting position of the landscape image;
3. A monitoring device according to claim 2. The control unit calculates an operating kilometer corresponding to the specified shooting position of the first image.
4. A monitoring device according to any one of claims 1, 2 and 3. A monitoring system for monitoring a maintenance object on a moving route of a vehicle,
a first camera that captures the maintenance object to generate a first image;
a second camera that captures an object indicating the distance or location at the capture time of the first image to generate a second image;
storing in an information storage unit the first image obtained by photographing the maintenance object and a second image obtained by photographing the object indicating the distance or location at the photographing time of the first image in association with each other;
Associating the first image with positioning information obtained by positioning the shooting position of the first image in addition to the second image,
The positioning information includes positioning accuracy indicating the accuracy of the positioning,
If the positioning accuracy is equal to or greater than a predetermined threshold, specifying the shooting position of the first image based on the positioning information;
a monitoring device that specifies the shooting position of the first image based on the second image when the positioning accuracy is less than the threshold;
Monitoring system. A monitoring method for monitoring a maintenance target on a moving route of a vehicle,
storing in an information storage unit a first image obtained by photographing the maintenance object and a second image obtained by photographing the object indicating the distance or location at the photographing time of the first image in association with each other;
Associating the first image with positioning information obtained by positioning the shooting position of the first image in addition to the second image,
The positioning information includes positioning accuracy indicating the accuracy of the positioning,
If the positioning accuracy is equal to or greater than a predetermined threshold, specifying the shooting position of the first image based on the positioning information;
If the positioning accuracy is less than the threshold, specifying the shooting position of the first image based on the second image;
Monitoring method. A computer program for monitoring a maintenance object on a moving route of a vehicle,
storing in an information storage unit a first image obtained by photographing the maintenance object and a second image obtained by photographing the object indicating the distance or location at the photographing time of the first image in association with each other;
Associating the first image with positioning information obtained by positioning the shooting position of the first image in addition to the second image,
The positioning information includes positioning accuracy indicating the accuracy of the positioning,
If the positioning accuracy is equal to or greater than a predetermined threshold, specifying the shooting position of the first image based on the positioning information;
If the positioning accuracy is less than the threshold, specifying the shooting position of the first image based on the second image;
make the computer do
computer program.

Images