JP2017001638A - Train position detection system using image processing, and train position and environmental change detection system using image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a train position detection system using image processing, capable of the travel position of a train with high accuracy and at low cost.SOLUTION: The train position detection system has an image feature database for storing a SURF feature extracted from each frame image of a base video image recorded by photographing a space ahead in a traveling direction while causing a train with a video camera set to travel on a track and position information associated with the SURF feature. A travel position of the train is detected on the basis of similarity calculated by comparing the SURF feature of a designated frame image of a reference video image recorded by photographing a space ahead in a traveling direction by the video camera while causing the train to travel on the track on date and time different from the date and time when the base video image is recorded with the SURF feature of each frame image of the base video image stored in the image feature database by frame matching processing.SELECTED DRAWING: Figure 12

Description

  The present invention relates to a train position detection system using image processing, and a train position and environment change detection system using image processing, and detects the position of a train that runs on a railroad track, and the time-series environment around the track. It is suitable for use in grasping changes.

  In railways, it is important to detect changes in the time-series environment around the track on which the train travels in order to find obstacles such as buildings and vegetation that may interfere with train travel at an early stage. It is. As a method to detect time-series environmental changes around the track, compare the video images recorded by shooting the periphery of the track with a video camera installed on the train while running the train on the same route at different dates and times It is possible to do. When comparing these video images recorded at different dates and times, it is desirable that the frames of the video images to be compared were taken at the same point. For this purpose, it is necessary to accurately detect the traveling position of the train.

  Conventionally, the following techniques are known as techniques for detecting the traveling position of a train. One is to detect the train position by comparing the TG position (the train travel position calculated from the train travel distance and route information) based on the signal from the speed generator and the GPS position from the GPS receiver. At the same time, the on-board device extracts the characteristics of the detected train position from both the vertical vibration signal of the train output from the acceleration sensor and the acoustic signal output from the sound collector, and the on-board device is connected to the rail joints, branches, etc. This is a technique for correcting with a singular point passage signal when passing through the singular point (see Patent Document 1). It is. The other is to apply DGPS (Differential Earth Positioning System) to a mobile traveling system, create a track information database by traveling for train information measurement, and during operation traveling, the information processing unit DGPS distance is used when the signal is high, and when the reception reliability is medium, the DGPS distance is used as an initial value to calculate the line curvature equivalent value during operation, and the distance measure is compared with the known line curvature equivalent value. When the reception reliability is low, the distance is specified based on the axle distance based on the axle rotation speed (see Patent Document 2).

JP 2011-225188 A JP 2004-271255 A JP 2015-23370 A

[Search July 8, 2013], Internet <http://www.m-system.co.jp/mstoday/plan/mame/b_network/9812/index.html>

  However, according to the study by the present inventors, the technique for detecting the position of the train proposed in Patent Documents 1 and 2 is used to detect a time-series environmental change around the track on which the train travels. There are drawbacks in that the required position detection accuracy cannot be obtained and the cost is high.

  Therefore, the problem to be solved by the present invention is to provide a train position detection system using image processing that can detect the traveling position of a train with high accuracy and low cost.

  Another problem to be solved by the present invention is that the traveling position of the train can be detected with high accuracy and at low cost, thereby easily changing the time-series environment around the track on which the train travels. It is another object of the present invention to provide a train position and environment change detection system using image processing that can be accurately detected.

In order to solve the above problems, the present invention provides:
The SURF feature extracted from each frame image of the first moving image recorded by shooting the space ahead of the traveling direction with the camera while traveling on a train on which a camera capable of shooting the moving image is installed, and Having an image feature database storing location information associated with the SURF feature;
The designated frame image of the second video recorded by photographing the space ahead of the traveling direction with the camera while running the train on the track at a date and time different from the date and time when the first video was recorded. The train's travel position is detected based on the similarity obtained by comparing the SURF feature of each frame image of the first moving image stored in the image feature database with the frame matching process. It is a train position detection system using image processing characterized by.

In addition, this invention
The SURF feature extracted from each frame image of the first moving image recorded by shooting the space ahead of the traveling direction with the camera while traveling on a train on which a camera capable of shooting the moving image is installed, and Having an image feature database storing location information associated with the SURF feature;
The designated frame image of the second video recorded by photographing the space ahead of the traveling direction with the camera while running the train on the track at a date and time different from the date and time when the first video was recorded. The train's travel position is detected based on the similarity obtained by comparing the SURF feature of each frame image of the first moving image stored in the image feature database with the frame matching process. Is a train position and environment change detection system using image processing characterized by.

  In this train position detection system or train position and environment change detection system, the position information associated with the SURF feature extracted from each frame image of the first moving image corresponds to, for example, each frame image of the first moving image. The distance information is obtained by integrating the speed from the attached kilometer information or the speed information associated with each frame image of the first moving image, but is not limited thereto. The speed information is acquired by, for example, a speed generator or a GPS receiver installed on the train, but is not limited thereto.

  The similarity obtained by comparing the SURF feature of the specified frame image of the second moving image with the SURF feature of each frame image of the first moving image stored in the image feature database by frame matching processing is preferably Is the number of SURF feature points corresponding to the number of SURF feature points extracted from the designated frame image of the second movie between the frame image of the first movie and the designated frame image of the second movie. Ratio. In this case, a pair of SURF feature points having corresponding points between the frame image of the first moving image and the designated frame image of the second moving image is typically the Euclidean distance of the feature description vector of the SURF feature. Is the shortest pair. Preferably, the position of the train is detected by reflecting the above similarity as a weight in the estimation result of the position of the train based only on the speed information associated with each frame image of the second moving image. By doing so, the position of the train can be detected with high accuracy.

  In order to prevent a decrease in detection accuracy due to a periodically provided structure such as a sleeper unique to a railway track, preferably, SURF features are obtained from the frame images of the first moving image and the second moving image. At the time of extraction, for example, the lower center trajectory portion and the central remote landscape portion of the frame image are masked with a mask image.

  The kilometer corresponding to the frame image of the second moving image can also be obtained as follows. That is, a longitudinal accelerometer and a GPS receiver are installed on the train. Thus, the longitudinal acceleration measured by the first moving image and the longitudinal accelerometer is obtained by photographing the space ahead of the traveling direction with the camera while running on the track on the train on which the longitudinal accelerometer and the GPS receiver are installed. Or the velocity information acquired by the GPS receiver is recorded, and an image database is created by manually allocating about a kilometer to each frame image of the recorded first moving image. Then, the second moving image recorded by photographing the space ahead in the traveling direction with the camera while traveling on the train at different dates after the date and time when the first moving image was recorded and stored in the above image database. The frame image of the second moving image is determined by determining which frame image of the first moving image corresponds to the designated frame image of the second moving image by comparing with the first moving image that has been made. Determine the kilometer.

  In the train position and environment change detection system described above, preferably, a space in front of the traveling direction is photographed by the camera while the vehicle on which the camera is installed is traveling on the track, and the positions are different from each other on the track. Around the trajectory by comparing two images taken at a position where the distance between them calculated using the speed information associated with each frame image of the second video is the selected distance Detects time-series environmental changes. Here, the distance between the positions that are different from each other on the trajectory and calculated using the velocity information can be rephrased as the sampling interval or the distance between the two frames. The selected distance is typically constant and predetermined, but is not limited to this. Since the two images taken at different positions on the orbit and the distance between them calculated using the speed information becomes the selected distance are images with different viewpoints, the two images By comparing these images, in other words, by performing monocular stereo vision using these two images, it is possible to quantitatively evaluate the time-series environmental changes around the orbit. Preferably, the virtual obstacle frames are set in multiple stages (or multiples) in the direction of the plane orthogonal to the traveling direction of the train.

  Cameras capable of shooting videos are installed so that the space ahead of the traveling direction of the train can be taken, and are typically installed inside the cab of the vehicle, for example, inside the window glass in front of the cab. The Any camera may be used as long as it can shoot a moving image, and it may be a digital camera or an analog camera, and may or may not have a recording function. This camera is, for example, a video camera or an industrial camera. As the video camera, a digital video camera is preferably used, and in particular, a high-resolution digital video camera such as a high-definition digital video camera is used. As the video camera, a thermographic video camera using far-infrared rays may be used. This makes it possible to use this train position detection system or the train position and environment change detection system at night. By synchronizing the image photographed by the camera and the vehicle speed information by the image synchronizer, the image photographed by the camera and the vehicle speed information can be associated one-to-one. The image synchronizer is typically installed in a train vehicle together with a camera. As the image synchronization device, basically any device can be used as long as image synchronization is possible, and a conventionally known device can be used. The image synchronization apparatus may be a dedicated image synchronization apparatus, or may be a computer, a data logger, or the like provided with an image synchronization function by software. As a dedicated image synchronization apparatus, preferably, a data index of speed information obtained at each selected time interval and assigned with a data index (or synchronization number) is sounded at each selected time interval. An image synchronization apparatus having a digital / analog conversion circuit for converting to a level signal is used (see Patent Document 3). This image synchronizer typically has an output terminal to which an audio input terminal of a camera having an audio input function is connected, and an audio level signal output from the digital / analog conversion circuit is output from the output terminal. Is output. The data index is typically sent to the input terminal of the image synchronizer as a serial signal of serial communication standard. The serial communication standard is, for example, RS485, RS422A, or RS232C, and is selected as necessary from among these, among which RS485 is preferable. In one typical example, the data index is converted into a frequency signal of 1200 Hz or 2200 Hz, and “1” is 1200 Hz and “0” is 2200 Hz. In a typical example, the image synchronization apparatus has a TTL level conversion circuit that converts a data index sent to the input terminal of the image synchronization apparatus as a serial signal of the serial communication standard into a TTL level signal. The TTL level conversion circuit Are input to the digital / analog conversion circuit. The time interval for acquiring the speed information or other sensor data in addition to this and the time interval for converting to the audio level signal are selected as necessary. The number of bits of the data index is selected as necessary, and is, for example, 6 bits or more and 10 bits or less.

  The train position detection system or the train position and environment change detection system further includes, for example, a vehicle motion measurement device installed in the vehicle, in addition to the camera and the image synchronization device. This vehicle vibration measuring device has, for example, at least one selected from the group consisting of an acceleration sensor, an angular velocity sensor, and a tilt sensor. Among these, the acceleration sensor is a three-axis acceleration sensor that detects the acceleration in the front-rear direction, the left-right direction, and the up-down direction of a train vehicle traveling on the track.

  According to this invention, the SURF feature of the designated frame image of the second moving image is obtained by comparing the SURF feature of each frame image of the first moving image stored in the image feature database by the frame matching process. Since the traveling position of the train is detected based on the similarity, the traveling position of the train can be detected with high accuracy and at low cost. In addition, the train position and environment change detection system can detect the travel position of the train with high accuracy and low cost, thereby facilitating time-series environmental changes around the track on which the train travels. And can be detected accurately.

1 is a schematic diagram showing an outline of a train position detection system according to a first embodiment of the present invention. It is a circuit diagram which shows an example of the circuit structure of the image synchronizer of the train position detection system by 1st Embodiment of this invention. It is a basic diagram for demonstrating the method to convert a data index into an audio | voice level signal in the image synchronizer of the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the specific structural example of the image synchronizer of the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the structural example of the vehicle fluctuation measuring apparatus of the train position detection system by 1st Embodiment of this invention. It is the top view, front view, and side view which show the structural example of the housing | casing of the vehicle fluctuation measuring apparatus of the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows an example of the signal waveform of each part of the vehicle oscillation measuring apparatus of the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows an example of the signal waveform of each part of the image synchronizer of the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows an example of the screen of the camera attitude | position adjustment program of the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows an example of the screen of the interlocking display program of the train position detection system by 1st Embodiment of this invention. It is a basic diagram for demonstrating the concept of the frame matching process used in the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the relationship between the production | generation process of the image feature database used in the train position detection system by 1st Embodiment of this invention, and a frame matching process. It is a basic diagram which shows the production | generation processing flow of the image feature database used in the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows an example of the corresponding point search by the SURF characteristic used in the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows an example of the template matching by the SURF characteristic used in the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the example of extraction of the SURF characteristic used in the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the flow of the frame matching process used in the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the search flow of the frame image of the base video image corresponding to the start frame of the reference video image used in the train position detection system by 1st Embodiment of this invention. It is a basic diagram for demonstrating the feature description vector of a SIFT feature. It is a drawing substitute photograph which shows an example of the extraction result of the SURF characteristic with respect to the start frame of the reference video image in the train position detection system by 1st Embodiment of this invention. 6 is a drawing-substituting photograph showing an example of matching between the SURF feature of the start frame of the reference video image and the SURF feature of the base video image stored in the image feature database in the train position detection system according to the first embodiment of the present invention. . It is a drawing substitute photograph which shows the example of the start frame of the reference video image in the train position detection system by 1st Embodiment of this invention, and the frame of the base video image matched with this start frame. It is a drawing substitute photograph which shows an example of the probability density distribution in the start position of the reference video image in the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows an example of the mode of update of probability density distribution by the speed information in the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the correction method of the speed accumulation error by matching of the SURF characteristic in the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows an example of the mask image used at the time of extraction of SURF feature in the train position detection system by 1st Embodiment of this invention. It is a basic diagram for demonstrating the precision of the frame matching process used in the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Kusatsu Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Kusatsu Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Kusatsu Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Kusatsu Line performed using the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the frame matching result in JR Kusatsu Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Nara Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Nara Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Nara Line performed using the train position detection system by 1st Embodiment of this invention. It is a drawing substitute photograph which shows the frame matching result in JR Nara Line performed using the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows the frame matching result in JR Kusatsu Line performed using the train position detection system by 1st Embodiment of this invention. It is a basic diagram which shows an example of the detection screen of the building limit trouble detection program by the building limit trouble detection technique used as the foundation of environmental change extraction in the train position and environment change detection system by 2nd Embodiment of this invention. It is a basic diagram which shows an example of the building limit trouble detection program setting screen by the building limit trouble detection technique used as the basis of environmental change extraction in the train position and environment change detection system by 2nd Embodiment of this invention. Outline lines showing the definition of the track coordinate system used in the construction limit trouble detection program setting screen by the construction limit trouble detection technology which is the basis of the environment change extraction in the train position and environment change detection system according to the second embodiment of the present invention. FIG. It is a basic diagram which shows an example of the virtual building limit frame setting screen by the building limit trouble detection technique used as the basis of an environmental change extraction in the train position and environmental change detection system by 2nd Embodiment of this invention. The abbreviation which shows the setting method of the building limit frame width used by the virtual building limit frame setting screen by the building limit trouble detection technique used as the basis of an environmental change extraction in the train position and environment change detection system by 2nd Embodiment of this invention. FIG. The abbreviation which shows an example of the relationship between the virtual building limit frame and the pixel on a video camera image by the building limit trouble detection technique used as the basis of environment change extraction in the train position and environment change detection system according to the second embodiment of the present invention FIG. FIG. 7 is a diagram for explaining an example of a change in appearance when a digital video camera moves forward in a construction limit obstacle detection technology that is a basis of environment change extraction in a train position and environment change detection system according to a second embodiment of the present invention; It is a basic diagram. It is a basic diagram which shows an example of the image of the digital video camera which advances in the construction limit obstacle detection technique used as the foundation of environmental change extraction in the train position and environmental change detection system by 2nd Embodiment of this invention. An example of the vanishing point and the moving direction of the image on the detection screen of the building limit trouble detection program based on the building limit trouble detection technology that is the basis of the environment change extraction in the train position and environment change detection system according to the second embodiment of the present invention. FIG. The virtual building limit before and after the advancement of the digital video camera in the building limit trouble detection program by the building limit trouble detection technique which is the basis of the environment change extraction in the train position and environment change detection system according to the second embodiment of the present invention It is a basic diagram which shows an example of a frame. Outline line showing an example of tracking method on epipolar line in building limit trouble detection program by building limit trouble detection technology which is the basis of environment change extraction in train position and environment change detection system according to second embodiment of this invention FIG. An example of the similarity profile by the tracking on the epipolar line in the building limit trouble detection program by the building limit trouble detection technique which is the basis of the environment change extraction in the train position and environment change detection system according to the second embodiment of the present invention is shown. It is a basic diagram. Examination for stabilizing tracking in a test line photographing test performed using a construction limit obstacle detection technique which is the basis of environmental change extraction in the train position and environmental change detection system according to the second embodiment of the present invention. It is a basic diagram for demonstrating a result. Synthesis of similarity profiles between different points in a test line photography test performed using a construction limit obstacle detection technique that is the basis for environmental change extraction in the train position and environmental change detection system according to the second embodiment of the present invention. It is a basic diagram which shows an example of a method. It is a basic diagram which shows an example of the trouble limit frame used for an environmental change extraction in the train position and environmental change detection system by 2nd Embodiment of this invention. It is a basic diagram which shows an example of the multiple obstacle limit frame used for an environmental change extraction in the train position and environmental change detection system by 2nd Embodiment of this invention. It is a basic diagram which shows the result of having calculated | required the number of construction limit frame troubles about the utility pole of a trackside by the test run performed on the actual line using the train position and environmental change detection system by 2nd Embodiment of this invention. . A pair of frames corresponding to a base video image and a reference video image in an analysis of a result obtained by a test run performed on an actual route using the train position and environment change detection system according to the second embodiment of the present invention It is a basic diagram which shows an example which calculated the difference only by. It is a drawing substitute photograph which shows the mode of the process target area of the base video image obtained by the test driving | running | working performed on the actual route using the train position and environment change detection system by 2nd Embodiment of this invention. It is a drawing substitute photograph which shows the mode of the process target area of the reference video image obtained by the test driving | running | working performed on the actual route using the train position and environment change detection system by 2nd Embodiment of this invention. It is a drawing substitute photograph which shows the trouble detection result by the virtual building limit frame in the base video image obtained by the test run performed on the actual route using the train position and environment change detection system according to the second embodiment of the present invention. is there. It is a drawing substitute photograph which shows the trouble detection result by the virtual building limit frame in the reference video image obtained by the test run performed on the actual route using the train position and environment change detection system according to the second embodiment of the present invention. is there. The number of troubled points and the difference detection result of the attention location in the base video image and the reference video image obtained by the test running performed on the actual route using the train position and environment change detection system according to the second embodiment of the present invention FIG. It is a drawing substitute photograph which shows the mode of the process target area of the base video image obtained by the test driving | running | working performed on the actual route using the train position and environment change detection system by 2nd Embodiment of this invention. It is a drawing substitute photograph which shows the mode of the process target area of the reference video image obtained by the test driving | running | working performed on the actual route using the train position and environment change detection system by 2nd Embodiment of this invention. The number of trouble points and the difference detection of the attention points of the base video image and the reference video image obtained by the test run performed on the actual line using the train position and environment change detection system according to the second embodiment of the present invention. It is a basic diagram which shows a result. The abbreviation which shows the detection result of the trouble of the utility pole by the virtual building limit frame in the reference video image obtained by the test run performed on the actual route using the train position and environment change detection system according to the second embodiment of the present invention. FIG. It is a basic diagram which shows the expanded virtual building limit frame used in the train position and environmental change detection system by 2nd Embodiment of this invention. Abbreviation for explaining an operation method of an application having a function of handling a plurality of video images used in the train position detection system according to the first embodiment of the present invention or the train position and environment change detection system according to the second embodiment. FIG. It is a basic diagram for demonstrating the kilometer determination technique used in the train position detection system by 1st Embodiment of this invention or the train position and environment change detection system by 2nd Embodiment. It is a basic diagram for demonstrating the kilometer determination technique used in the train position detection system by 1st Embodiment of this invention or the train position and environment change detection system by 2nd Embodiment. It is a basic diagram which shows the train position detection system by 5th Embodiment of this invention.

  Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described.

<First Embodiment>
A train position detection system (hereinafter referred to as “train position detection system”) using image processing according to the first embodiment will be described.

[Overall configuration of train position detection system]
FIG. 1 shows an outline of this train position detection system. As shown in FIG. 1, in this train position detection system, a digital video camera 20 having a voice input function and a vehicle shake measurement device 30 are connected via an image synchronization device 10. More specifically, the output terminal of the vehicle shake measuring device 30 and the input terminal of the image synchronization device 10 are connected by a cable C1, and the output terminal of the image synchronization device 10 and the audio input terminal of the digital video camera 20 are connected by a cable C2. It is connected. These image synchronizer 10, digital video camera 20, and vehicle motion measuring device 30 are installed in the vehicle, specifically, for example, in the leading vehicle.

  For example, the digital video camera 20 is installed in the driver's cab of the leading vehicle of the train so that a space ahead in the traveling direction can be photographed. More specifically, the digital video camera 20 is moved using, for example, a powerful sucker on the inner surface of the window glass in front of the driver's cab of the leading vehicle or on a table immediately in front of the inner surface of the window glass. It is installed so that there is no. A commercially available general-purpose digital video camera can be used as the digital video camera 20, but a high-resolution digital video camera such as a high-definition digital video camera is preferably used.

  The vehicle sway measurement device 30 includes, for example, at least one sensor selected from the group consisting of an acceleration sensor, an angular velocity sensor, and a tilt sensor. Among these, the acceleration sensor is a three-axis acceleration sensor that detects the acceleration in the front-rear direction, the left-right direction, and the up-down direction of the vehicle traveling on the track. A GPS receiver 38 is connected to the vehicle vibration measuring device 30 via a cable C3. The vehicle oscillation measuring device 30 is configured to acquire and record information on the vehicle position (latitude, longitude) and speed information (NMEA0183) from a GPS signal received by the GPS receiver 38. . Further, the vehicle sway measuring device 30 can acquire and record the position (about kilometer) and speed information on the track by the pulse voltage sent from the speed generator installed at the axle end of the vehicle. It is configured. Details of the vehicle oscillation measuring device 30 will be described later.

  The image synchronizer 10 includes an image (frame) of a track space photographed at a sampling interval selected by the digital video camera 20 while the vehicle is running, and time-series position information and speed acquired by the vehicle shake measuring device 30. Used to synchronize information and make a one-to-one association with each other. For this purpose, in the image synchronization device 10, the data index (or synchronization number) generated by the vehicle motion measurement device 30 is audio-encoded and recorded on the audio track of the digital video camera 20. Details of the image synchronization apparatus 10 will be described later. The vehicle sway measuring device 30 records the pulse voltage sent from the speed generator and associates it with the captured image in the same manner as described above.

  An example of how to install the digital video camera 20 will be described. That is, when the digital video camera 20 is installed, the vanishing point of the track is positioned at the center of the camera image, and the screen vertical direction is parallel to the gravity axis. At this time, in order to accurately perform these adjustments, an auxiliary line is displayed on the screen. In addition, since shooting is performed at the wide-angle end of the lens of the digital video camera 20, lens distortion and image center correction cannot be ignored when using this auxiliary line when the digital video camera 20 is installed. A tool program (camera setup program) in which an auxiliary line is drawn on the video camera image subjected to is developed, and a notebook personal computer (notebook PC) installed with this tool program is brought into the vehicle and used. At this time, an HDMI-USB converter that converts an HDMI (registered trademark) signal output from the digital video camera 20 into a format (H.264) that can be input by a PC is provided between the digital video camera 20 and the notebook PC. Connecting.

  The lens distortion correction coefficient and the image center deviation of the digital video camera 20 and the focal length used in later processing are separately calibrated. This is called camera internal parameter calibration. For this purpose, a camera internal parameter calibration program is installed in the notebook PC. The parameters of the camera internal parameter calibration are stored as a file, and are used in the work at the time of installation of the digital video camera 20 and the function of detecting the train position later.

  After shooting with the digital video camera 20 while running on the train, the motion / latitude / longitude / speed data and the data index are transferred from the vehicle motion measuring device 30 to the data processing PC 50 as a CSV file for the number of times of shooting, for example. Further, the digital video camera 20 passes a captured image and a sound-encoded data index issued by the vehicle motion measuring device 30 to the data processing PC 50. The data processing PC 50 is installed with a train position detection program for performing various processes including a frame matching process, which will be described in detail later. The data processing PC 50 is installed in an office inside the train or outside the train. The motion / latitude / longitude / speed data and the data index from the vehicle motion measurement device 30 may be passed to the data processing PC 50 via an SD card, for example. Similarly, a captured image from the digital video camera 20 and a voice-encoded data index issued by the vehicle motion measuring device 30 may be passed to the data processing PC 50 via an SD card, for example.

  The above-mentioned various data is read into a program (linked display program) that displays video camera images and various sensor data in an interlocked manner, decoded from the audio data of the video camera images into a data index, and the motion / longitude / latitude / velocity data and video There is a one-to-one correspondence with the frame of the camera image. In addition, the video camera image and the speed graph are synchronously displayed on the interlocking display program to check whether the data is valid.

  After the video camera image and the speed information are confirmed to be valid, the video camera image and the speed information of each frame are output from the interlocking display program and read into the train position detection program.

[Outline of data file input / output in train position detection system]
Data file input / output in this train position detection system will be described.

(1) As already described, the digital video camera 20 and the vehicle shake measurement device 30 are connected via the image synchronization device 10, and the video camera image captured by the digital video camera 20 and the vehicle shake measurement device 30 are obtained. Recorded speed information (speed data) and sensor data at the same time.
(2) In the SD card of the vehicle motion measuring device 30, several types of files are generated with the year / month / day / hour / minute / second as half the file name. Also, a video file having a file name of year / month / day / hour / minute / second is generated in the SD card of the digital video camera 20.
(3) Start (kick) the analysis program of the vehicle shake measurement device 30 from the synchronization viewer, and first read the _vbsd.bin file with the analysis program GUI. In the analysis program, the _gpsd.csv file, _xind.bin file, and _spdd.bin file are read from the link in the _vbsd.bin file.
(4) Next, using the analysis program GUI, output the .INI file, .PTP file, and .ANA file, and load them into the synchronization viewer.
(5) The synchronization viewer automatically copies the output .INI file, .PTP file, and .ANA file to the specified folder.
(6) Next, the synchronization viewer reads the .mts file, which is a video file, automatically decodes the data index recorded on the audio track, and creates a correspondence table file between the video camera image frame and the data index. Output.

[Details of train position detection system]
(Details of the image synchronizer 10)
Details of the image synchronization apparatus 10 will be described. FIG. 2 shows the image synchronizer 10. As shown in FIG. 2, the image synchronization apparatus 10 includes a digital / analog conversion (D / A conversion) circuit 11 for converting a data index (synchronization number) that is a digital signal into an audio level signal that is an analog signal. . In this digital / analog conversion, for example, “1” of the data index is converted into a frequency signal of frequency f _{1 in} the audio frequency band, and “0” is converted into a frequency signal of frequency f ₂ (≠ f ₁ ) in the audio frequency band. Typically, f ₂ > f ₁ is selected. As an example, FIG. 3 shows an example in which 6-bit “001100” is converted into a frequency signal as a data index. Specific examples of f ₁ and f ₂ are f ₁ = 1200 Hz and f ₂ = 2200 Hz, but are not limited thereto. This data index is assigned to time-series data (position information data, speed information data, and sensor data such as an acceleration sensor) acquired at each time interval selected from the vehicle motion measurement device 30. There are serial numbers for these data. The image synchronizer 10 has an input terminal 12, and the data index is sent to the input terminal 12 from the output terminal of the vehicle shake measuring device 30. The data index is supplied from the vehicle motion measuring device 30 to the input terminal 12 as a serial signal of serial communication standard. The serial communication standard is, for example, RS485, RS422A, or RS232C. The image synchronization apparatus 10 has an output terminal 13, and an audio level signal output from the digital / analog conversion circuit 11 is output from the output terminal 13. The output terminal 13 is connected to an audio input terminal of a digital video camera 20 having an audio input function.

  FIG. 4 shows a specific configuration example of the image synchronization apparatus 10. As shown in FIG. 4, the image synchronization apparatus 10 includes a TTL level conversion circuit 14, a digital / analog conversion circuit 11, and an audio voltage level conversion circuit 15. The TTL level conversion circuit 14 is connected to the input terminal 12 of the image synchronization apparatus 10. A serial signal according to the serial communication standard is supplied to the input terminal 12, and the level of this serial signal is converted to a TTL level by the TTL level conversion circuit 14. The data index thus converted to the TTL level is input to the digital / analog conversion circuit 11 and converted into an audio level signal. For example, the data index is supplied from the vehicle motion measuring device 30 to the input terminal 12 as an RS485 level signal, and this RS485 level signal is converted to a TTL level by the TTL level conversion circuit 14. The audio level signal output from the digital / analog conversion circuit 14 is input to the audio voltage level conversion circuit 15 for voltage level conversion. The output terminal of the audio voltage level conversion circuit 15 is connected to the output terminal 13 of the image synchronization apparatus 10. An audio input terminal of the digital video camera 20 can be connected to the output terminal 13. A specific example of the digital / analog conversion circuit 11 is a HART (Highway Addressable Remote Transducer) modem (for example, see Non-Patent Document 1. “HART” is a registered trademark). In the HART modem, “1” is converted into a frequency signal of 1200 Hz, and “0” is converted into a frequency signal of 2200 Hz. The image synchronization apparatus 10 has an input terminal 25, and an external microphone 26 can be connected to the input terminal 25. The external microphone 26 is used for monaural recording. The input terminal 25 is connected to the output terminal 13 so that the monaural sound (or stereo one channel) input to the input terminal 25 can be sent to the outside from the output terminal 13 together with the sound level signal as necessary. It has become.

  According to this image synchronizer 10, the data index given to the time series data (position information data, speed information data and sensor data) output from the vehicle motion measuring device 30 is converted into an audio level signal. Therefore, this audio level signal can be input to the audio input terminal of the digital video camera 20 and recorded. Therefore, it is possible to synchronize the position information data, the speed information data, and the sensor data with the image taken by the digital video camera 20 by using the audio level signal. Since the image synchronizer 10 can be easily configured, the manufacturing cost can be kept low and the size can be reduced.

(Details of the vehicle shake measuring device 30)
Details of the vehicle shake measuring device 30 will be described. FIG. 5 shows a configuration of the vehicle shake measuring device 30. The vehicle shake measuring device 30 includes a sensor processing unit 31, a display processing unit 32, a position information processing unit 33, a data storage unit 34, an external interface (I / F) control unit 35, and a power supply control unit 36. These sensor processing unit 31, display processing unit 32, position information processing unit 33, data storage unit 34, external interface control unit 35 and power supply control unit 36 are connected to each other via a high-speed serial bus 37. An output terminal of the external interface control unit 35 is connected to an input terminal (input terminal 12 shown in FIG. 4) of the image synchronization apparatus 10. The position information processing unit 33 is supplied with a GPS signal received by the GPS receiver 38 and a signal from the speed generator 39.

  The sensor processing unit 31 outputs the three-axis acceleration sensor, the angular velocity sensor (yaw, roll), and the inclination sensor (pitch, roll) for detecting the longitudinal, lateral, and vertical accelerations of the vehicle traveling on the track. A signal is supplied and predetermined processing is performed. The display processing unit 32 performs display control of a liquid crystal display (LCD), for example. The position information processing unit 33 measures the position (longitude and latitude) of the vehicle using the GPS signal received by the GPS receiver 38 and measures the position (about kilometer) on the track using the signal from the speed generator 39. The data storage unit 34 stores the sensor data processed by the sensor processing unit 31 and the data index attached to the sensor data, and an SD card is used as a recording medium, for example. The external interface control unit 35 converts the TTL level data index of the SCI (Serial Communication Interface) output from the sensor processing unit 31 into a serial communication standard, for example, RS485 level, and sends it to the input terminal 12 of the image synchronization apparatus 10. The power control unit 36 includes a battery 40, and controls the supply of power to the sensor processing unit 31, the display processing unit 32, the position information processing unit 33, the data storage unit 34, the external interface control unit 35, and the display.

  A specific configuration example (outer shape) of the vehicle shake measuring device 30 is shown in FIGS. 6A is a plan view, FIG. 6B is a front view, and FIG. 6C is a side view. As shown in FIGS. 6A, 6B, and 6C, the vehicle sway measurement device 30 includes a sensor processing unit 31, a display processing unit 32, a position information processing unit 33, and data storage shown in FIG. 5 inside a rectangular housing 30a. A unit 34, an external interface control unit 35, a power supply control unit 36, and a high-speed serial bus 37 are accommodated. Here, the sensor processing unit 31, the display processing unit 32, the position information processing unit 33, the external interface control unit 35, and the power supply control unit 36 are a sensor processing board, a display processing board, a position information processing board, an external interface control board, and a power supply, respectively. It is configured as a control board. An example of the size of the housing 30a is about 250 mm × 170 mm × 120 mm. Handles 30b and 30c are attached to both ends of the housing 30a, and the vehicle shake measuring device 30 can be carried by grasping these handles 30b and 30c with both hands. An LCD 30d is installed on the front surface of the housing 30a. The display on the LCD 30d is controlled by the display processing unit 32. A power switch 30e is attached to the front surface of the housing 30a. The power switch 30e is connected to the power controller 36. The power source of the vehicle sway measuring device 30 can be turned on / off by the power switch 30e.

  FIG. 7A shows an example of the waveform of the SCI (TTL) signal of the external interface control unit 35 of the vehicle motion measuring device 30, and FIGS. 7B and 7C show examples of the waveform of the RS485 level signal after the RS485 level conversion. 8A and 8B show examples of the waveform of the RS485 level signal input to the image synchronizer 10, and FIG. 8C shows an example of the waveform of the TTL level signal converted by the TTL level conversion circuit 14. FIG. 8D shows an example of the waveform of the audio level signal after it has been performed.

(Camera posture adjustment program)
In this train position detection system, it is also possible to create and use a camera attitude adjustment program that reads the result of the above-mentioned camera internal parameter calibration and displays a through image corrected in real time. By using this camera attitude adjustment program, the vanishing point of the track can be correctly set at the center of the image even if the train position detection engine does not have a function of correcting the roll angle of the digital video camera 20. That is, the camera image itself is misaligned as well as distorted and difficult to set correctly. However, the vanishing point can be easily set at the center of the image by using this camera posture adjustment program.

FIG. 9 shows a camera orientation adjustment program screen (actually a color screen). The functions of the screen of this camera posture adjustment program will be described as follows.
(1) Real-time display of images taken by the digital video camera 20
(2) Separately, the distortion correction coefficient and image center of the lens of the digital video camera 20 obtained by the above internal parameter calibration are read. Distortion correction and image center correction are performed in real time, and a corrected image is displayed.
(3) An intersection P of a cross line (for example, a red cross line) composed of a vertical line L1 and a horizontal line L2 displayed on the screen is the center of the image, and the digital video camera so that the vanishing point of the track overlaps the intersection P 20 postures are adjusted.
(4) The lower left diagonal line L3 and the lower right diagonal line L4 (for example, the yellow line) displayed at the lower center of the screen are “2. Left diagonal line” and “3. By selecting the “line” radio button and pressing the left arrow button (←) and the right arrow button (→) below it, the angle can be adjusted around the intersection P of the crosshairs. By making these left diagonal line L3 and right diagonal line L4 along the two rails, it can be determined whether or not the vanishing point is correct.
(5) Two vertical lines L5 and L6 (for example, yellow-green line) displayed on the screen indicate the vertical direction of the screen, and radios of “1: left vertical” and “4: right vertical” at the lower right of the screen. You can adjust the left and right position by selecting with the button and pressing the left and right arrow buttons below it. The roll angle of the digital video camera 20 is finely adjusted by making the left perpendicular line and right perpendicular line follow the perpendicular line of the building or the like.

By using the above camera posture adjustment program, the mounting posture of the digital video camera 20 is adjusted as follows.
The camera viewing axis is parallel to the ground plane. The camera viewing axis is parallel to the orbital direction. The camera imaging plane Y axis and the gravity axis are parallel.

  Although the posture adjustment of the digital video camera 20 can be performed with high accuracy by using the camera posture adjustment program, even when the digital video camera 20 is actually installed on the site, the posture adjustment accuracy is slightly increased. Even at the expense, it may be required to adjust the posture more easily. In such a case, the attitude of the digital video camera 20 is estimated from the captured video camera image in the train position detection program. At this time, for example, a guide line such as the camera posture adjustment program screen shown in FIG. 9 is displayed on the captured video camera image, and the user can designate the vanishing point of the rail and the building vertical line with the mouse. .

(Linked display program)
The interlock display program interlocks and displays GPS data and speed generator information recorded by the vehicle motion measuring device 30 and an image photographed by the digital video camera 20. FIG. 10 shows a screen of the interlocking display program 100. The functions of the linked display program are as follows.
(1) The data index is decoded from the audio data of the audio track of the video camera image, correlated with the speed information and sensor data of the vehicle motion measuring device 30, and the speed information is assigned to each video frame.
(2) The video frame rate of the digital video camera 20 and the speed information sampling rate of the vehicle motion measuring device 30 are generally different. For example, the video frame rate of the digital video camera 20 is 60 Hz, and the speed information sampling rate is 1 Hz. For this reason, linear interpolation is performed and velocity information is assigned to all video frames.
(3) The recorded video camera image and the acceleration sensor data, GPS speed data, and speed generator data are reproduced in conjunction with each other.
(4) Display speed information (GPS speed) corresponding to the currently displayed video frame.

[Frame matching technology]
The frame matching technique used in this train position detection system will be described.

  This frame matching algorithm includes (1) generation processing of a base video image feature database and (2) frame matching processing. Of these, the frame matching process (2) corresponds to the first video image (base video image) (the first video image) when there are a plurality of video images taken on the same route at different dates and times. ), And a process of searching for a corresponding frame of the base video image for each frame of the other video image (reference video image) (corresponding to the second moving image) is performed. An example is shown in FIG. As shown in FIG. 11, in order to compare the base video image with a plurality of reference video images, an image feature database is generated in advance. Although details of the image feature database will be described later, by extracting information used for frame matching from all frame images of the base video image in advance and creating a database, the same processing can be performed in frame matching processing with a plurality of reference video images. Prevent duplication. An example is shown in FIG. As shown in FIG. 12, all reference video images (reference video images 1 to 3 in this example) on the same route are subjected to frame matching with the base video image, and the result is output as a frame matching result. At this time, the frame matching result is a record of a pair of the reference video image frame number (the first frame of the video image is numbered 0 and the number counted up hereinafter) and the corresponding frame number of the base video image. And become a database.

  An image feature database generation processing flow is shown in FIG. The base video image and speed information input in FIG. 13 is a data set recorded by the vehicle motion measuring device 30, and includes an MPEG4 video image file, and includes speed information and kilometer information associated with a video frame. It consists of files and information that associates them with each frame. The image feature database generated in FIG. 13 is the distance information obtained by extracting image characteristic information for each frame image of the base video image, integrating the speed up to that point, and the kilometer information of the original data. Is one record. In the frame matching technique used here, a SURF (Speeded Up Robust Features) feature is used as a feature amount. These image features are used, for example, to search for corresponding points between these images as shown in FIG. 14B when the images shown in FIG. 14A obtained by photographing the same object at different dates and times are used as input images. . As shown in FIG. 14B, corresponding point estimation based on the SURF feature is performed (the positions of both ends of the oblique straight line are the positions of the SURF feature). The SURF feature is also used for searching for a specific object in the image (template matching). An example is shown in FIG.

  SURF features, like SIFT (Scale Invariant Feature Transform) features, are robust against changes in lighting, scaling, and rotation, and are effective for natural images that change every moment, such as railway track images. Conceivable.

  As an example, FIG. 16 shows the result of actually extracting SURF features from a video image acquired on the JR Kusatsu Line. FIG. 16 shows the SURF feature point positions where dots of three colors of red, green, and blue drawn on the video image are extracted. Red, green, and blue indicate the strength of the SURF feature (the red dot is the strongest, the green dot is the next strongest, and the blue dot is the next strongest), and those below a certain strength are excluded. The SURF feature is not detected in the distant scenery on the rail surface and in the center, but this is intentionally not detected in consideration of the adverse effect on the frame matching algorithm described later.

  As shown in FIG. 13, the above-mentioned SURF feature is detected for all frames of the base video image and stored in the image feature database. At this time, the distance at which the frame in which the SURF feature detection is performed is located from the position where the first frame is photographed is obtained from the velocity information by integrating the velocity, and stored in the database in association with the SURF feature.

In this algorithm, the SURF features are stored as an image database, and the purpose and reason for use in the frame matching process are as follows (1) to (3).
Show.

(1) It is checked whether an object on a specific frame of the reference video image exists in the specific frame of the base video image.
(2) In executing the above, even if the lighting environment fluctuates due to weather, time, etc., it is not affected.
(3) In executing the above, there is no influence even if there is an error in the camera mounting angle or vehicle attitude.

  It should be noted that the image feature database preferably has relatively large information for one feature so that the SURF feature achieves robustness as described above, and becomes relatively large in all video image frames (for example, Therefore, the data is stored as binary data.

  As described above, the frame matching process between the reference video image and the base video image is performed using the image feature database of the base video image. A frame matching processing flow is shown in FIG. This frame matching processing flow will be described in the order of (1) to (12) in FIG.

(1) As an input, a set of velocity information associated with a new reference video image is prepared.
(2) An image feature database of base video images prepared in advance is used.
(3) The user specifies a frame matching start frame from the frame image of the reference video image.
(4) A frame of the base video image corresponding to the start frame of the reference video image is searched. This search is performed in the processing flow shown in FIG. That is, as shown in FIG. 18, first, the SURF feature is extracted from the start frame of the reference video image. The SURF feature of the start frame of the reference video image is compared with the SURF feature of each frame stored in the image feature database of the base video image, and the similarity is calculated. The similarity is expressed by the following formula (1) as a ratio between the number of feature points corresponding to each of the frame images of the base video image and the reference video image described below and the number of SURF feature points extracted on the frame image of the reference video image. Define as follows.

  This similarity is substituted for Sim in the frame search flow of the base video image corresponding to the start frame of the reference video image shown in FIG.

  Further, the correspondence between feature points is obtained by selecting a pair having the shortest Euclidean distance of the feature description vector of the SURF feature. Here, the feature description vector used in the SIFT algorithm that is the basis of the SURF algorithm will be described. The feature description vector of the SIFT feature is a vector representing the amount of change in the image luminance around the feature point in each direction (see FIG. 19). The left figure of FIG. 19 is an input image, and it is assumed that there is a feature point at a point indicated by a central white circle. It is calculated in which direction the luminance at this point has the greatest gradient (the amount of change in luminance). In the case of this example, the luminance varies most in the upper left direction (arrow direction) in the left diagram of FIG. A 4 × 4 grid (green grid in the upper left figure) with this direction as the vertical axis is set, and the variation in luminance in each direction is calculated within each grid. By doing so, even when the image is rotated, the luminance variation in the grid is the same at the same point. The right diagram in FIG. 19 shows the amount of change in luminance with respect to eight directions by the length of the arrow. In this program, the amount of variation in four directions is used. Therefore, one feature point is described in the SURF feature with 4 × 4 grids × 4 directions = 64 values. That is, one feature point is characterized as a 64-dimensional vector.

  Regarding the feature points on two different images, the fact that the 64-dimensional vectors are similar can be presumed that the feature points and their surroundings are very similar images. In other words, since the Euclidean distance is short in the 64-dimensional space, the feature points on the two images can be brute-forced so that the most similar 64-dimensional vectors can be paired as corresponding points. . Considering the above-described method for obtaining the feature description vector, it is considered that the same point can be taken even if there is a rotation variation between images without being influenced by the absolute value of luminance.

  The SURF algorithm is designed to perform the same performance as SIFT at high speed, and the basic concept of feature point correspondence is the same.

  As described above, the corresponding points based on the SURF feature are detected, and the similarity is obtained by Expression (1). This process is performed on all the records in the image feature database, and the record having the maximum similarity is obtained. However, since processing time increases when performed on all the records of the image feature database, in actual processing, for example, the kilometer information stored in the image feature database and the kilometer information of the reference video image are matched, and the base The corresponding processing is performed about 100 m before and after the corresponding kilometer of the video image.

  When the record having the maximum similarity is obtained as described above, the frame number of the base video image corresponding to the record is obtained from the image feature database and output.

  The following shows how this processing is performed. FIG. 20 shows the SURF feature extraction result for the start frame image of the reference video image, and the SURF feature extraction process is performed on the start frame image of the reference video image at the upper right of the flowchart shown in FIG. The white spot position in FIG. 20 is the position extracted as the SURF feature.

  21A and 21B show how the SURF feature stored in the image feature database of the base video image is matched with the SURF feature of the start frame of the reference video image. The blue dots in FIGS. 21A and 21B are the same as the white dots in FIG. 20 and indicate the SURF features detected in the start frame of the reference video image. The red dot indicates the SURF feature on the base video image search frame read from the image feature database of the base video image. Some of the blue and red dots are connected by a yellow line between them, which indicates that the red and blue at both end points are corresponding points. FIG. 21A has only 20 corresponding points, and the degree of similarity is low. FIG. 21B shows the highest similarity and 146 corresponding points. FIG. 22 shows a frame of a base video image associated by this method. In FIG. 22, the right side is the start frame image of the reference video image, and the left side is the frame image of the base video image associated by the above algorithm.

(5) The state of the position (position where speed information is integrated) corresponding to the frame obtained in the above process is set.

  The state represents the probability of the shooting position of the current frame of the reference video image as a probability density. Since the obtained position is the most probable position, a normal distribution centered here is set. An example is shown in FIG. The horizontal axis of the graph in FIG. 23 is the position of the base video image (integration of the velocity assigned to the frame), and the vertical axis is the probability (probability) that the position where the current frame of the reference video image was taken is. . The left and right center of the graph is the position obtained in (4), and a normal distribution is assigned around this position (white vertical line). Hereinafter, it is considered that there is a position where the current frame of the reference video image is captured at a position where the probability is the highest while changing the probability density.

(6) The next frame image of the reference video image is read. Thereafter, it is read each time the processes from (7) to (10) are repeated. At this time, the speed information assigned to the frame is also read.

(7) The probability density shown in FIG. 23 in the current frame is estimated from the speed information. The following formula (2) is used for the estimation.
Here, k and i represent positions (in this system, for example, serial numbers of positions every 0.5 m). t represents time, and more specifically, the number of times of looping from (6) to (12) in the flowchart of FIG. p represents a probability density, and p _{i, t} represents a probability that a video camera exists at i at time t.

Equation (2) is obtained by multiplying the probability density distribution p _{i, t-1} where a video camera exists at a position i in the past t−1 and the probability existing at the position k when the current t is the current speed u _t. This is an accumulation of all positions. It shows how the current probability density distribution is changed by the current speed u _t based on the past probability distribution.

Here, consider where the position moves at the current speed u _t . Δt is the time between frames
Then, the amount of movement is l = u _t × Δt, but the speed (GPS speed information or speed generator information in the case of this system) data itself contains an error. This error is modeled as a normal distribution, and the central portion of Equation (2) is defined as a normal distribution as follows.
Here, μ = u _t × Δt and δ = Δtδ _s . δ _s is an error variance of speed information, and is set to 1 m / s in this program, for example.

  With the above calculation, the probability density distribution shown in FIG. 23 is set to the initial state, and the probability density distribution at time t−1 is updated to the probability density distribution of t thereafter.

  FIGS. 24A to 24C show how the probability density is changed by this calculation using actual data. 24A to 24C show that the time advances in the order of A, B, and C, and the probability density distribution is updated. It can be seen that the peak position of the probability density is gradually shifted to the right, that is, forward.

  With the above processing, the position where the current frame of the reference video image is captured is estimated from the speed information. However, the velocity information includes an error, and the probability density is introduced as described above. However, since the estimation is essentially the same as the integration of the velocity, the error of the position estimation increases with time.

(8) Every time the frame of the reference video image advances by 10, the state estimation correction described in (9) below is performed.

(9) Since the error is accumulated at the position estimated in (7) as it is, the same processing as that in (4) is performed to eliminate the accumulated error. The method will be described below.

  For expression (2), the similarity described in (4) is reflected as a weight on the probability density at each point. In this way, even if there is a false position where the degree of similarity increases due to incorrect matching, the probability will not be the highest when the probability density distribution due to speed is considered. The estimation based only on the speed information is corrected to the correct position by performing this process. This is shown in FIGS. The graph of FIG. 25A represents a state in which the peak of the probability density distribution is shifted from the true position as a result of repeating the process (7). The graph of FIG. 25B displays the similarity by matching the SURF feature of the image information database of the base video image with the current frame of the reference video image. In this case, the similarity is peaked at the true position, but it may happen that the similarity is high at other positions (for example, in a rural scene where there are no buildings around, only the utility pole appears regularly) ). Therefore, when matching is performed using only the similarity of images, a very large error may occur. The graph of FIG. 25C reflects the similarity by matching of FIG. 25B as a weight in the probability density distribution by the speed of FIG. 25A. As shown in FIG. 25C, it can be seen that the false peak that appeared in the matching similarity is lost because the probability density at the same position is almost 0, and the probability density is corrected to the correct position. Naturally, even if such a process is performed, there is still a possibility of deviation from the true position, but a large error is not generated, and a more appropriate position is estimated.

(10) The frame number of the base video image corresponding to the estimated position is acquired from the image feature database.

(11) (10) The corresponding frame of the base video image and the current frame of the reference video image are written as a pair in the frame correspondence information file.

(12) The processing from (6) to (10) is performed while increasing the frame number of the reference video image, and is performed up to the last frame (or designated frame) of the reference video image.

  In this train position detection system, the track is photographed by one digital video camera 20 facing forward in the traveling direction, which may cause the following problems. That is, if the sleepers of the trajectory have a repetitive pattern and there are few SURF features in other areas such as a rural scene, incorrect matching is performed. Moreover, since the distance between sleepers is short, the correction described with reference to FIGS. 25B and 25C may not work. Since many sleepers appear repeatedly, this phenomenon continues for a long time, resulting in a large error. In addition, there is a building far in the center of the image, and in the foreground a landscape with few SURF features such as a rural landscape, incorrect matching is performed. This is because a very distant landscape has very little movement in the image. In spite of the vehicle traveling several tens of meters, this cannot be detected on the image, and the error may be increased by the correction function by frame matching. In order to solve the problem peculiar to the video image of the railway, a mask image is introduced. FIG. 26 shows an example of the mask image. As shown in FIG. 26, in this mask image, the lower center trajectory portion and the central remote landscape portion are masked. This mask image is used for SURF feature extraction, and feature extraction is not performed in the black region. This eliminates the regular repetition of sleepers on the track and the effects of remote landscapes.

(Verification results on actual routes)
The verification results obtained by implementing the above algorithm and performing frame matching with the actual route video image will be described.

  In order to confirm the effect of frame matching used in this train position detection system, the following four types of frame matching were performed on the same data.

(1) After searching for the corresponding frame of the base video image with respect to the start position frame of the reference image, both video images are simply increased by one frame.
(2) After searching the corresponding frame of the base video image with respect to the start position frame of the reference image, frame matching is performed using only the velocity information.
(3) After searching for the corresponding frame of the base video image with respect to the start position frame of the reference image, frame matching is performed using only the SURF feature matching of the image.
(4) After searching for the corresponding frame of the base video image with respect to the start position frame of the reference image, frame matching is performed using the algorithm described above using matching of the velocity information and the SURF feature of the image.

  The accuracy of these four frame matchings was determined by comparing the image of the frame matching result by human eyes and how many orbital sleepers are displaced (minimum unit is 0.5). Further, since it is not possible to visually compare all video image frames, about 100 points are sampled every 200 frames for about 5 km. FIG. 27 shows a graph of the test results. The tested route is between Kusatsu and Tsuge on the JR Kusatsu Line. The shooting conditions are as follows.

Base video image Date: September 10, 2014, 11:00 Weather: Cloudy Image quality: Good Reference video image Date: September 12, 2014, 11:00 Weather: Sunny Image quality: Good Compared km 36.7 → 32. 7

  In FIG. 27, the horizontal axis represents the frame number of the reference video image, and the vertical axis represents the error from the base video image of the frame matching result described above by the number of sleepers. The graph shown by (1) shows the result of (1), the graph shown by (2) shows the result of (2), and the graph shown by (3) shows the result of (4). The result of the above (3) (SURF feature matching only) is not displayed because there are several tens of errors in several frames.

  FIG. 28 to FIG. 31 show examples of images showing the four frame matching results on the JR Kusatsu Line (the left image in the figure shows the base video image, and the right image shows the reference video image), and FIG. 32 shows the frame matching result. The graph of the frame number of is shown. 32, the horizontal axis represents the number of calculations, the vertical axis represents the frame number, the line graph indicated by (1) represents the frame number of the current video image, and the line graph represented by (2) represents the frame number of the associated previous video image. is there.

  The tested route is between the Kusatsu and Kusatsu Lines of the JR Kusatsu Line. The shooting conditions and frame matching results are as follows.

Base video image Date / time: September 10, 2014, 11:00 Weather: Cloudy Image quality: Good Reference video image Date / time, September 12, 2014, 11:00 Weather: Sunny Image quality: Good Compare section km approx. 35.908 → 0.1

  The frame matching result is good, the error is maximum and the number of sleepers is about 1.5, and it seems that environmental difference detection is sufficiently possible.

  33 to 36 show examples of frame matching results on the JR Nara line (the left image in the figure shows the base video image, and the right image shows the reference video image), and FIG. 37 shows the frame number of the frame matching result. The graph of is shown. The horizontal axis in FIG. 37 is the number of calculations, the vertical axis is the frame number, the line graph in (1) is the current video frame number, and the line graph in (2) is the previous video frame number associated with it.

  The route on which the test was conducted is between JR Nara Line Tanagura and Tofukuji Temple. The shooting conditions and frame matching results are as follows.

Base video image Date: December 19, 2013 14:00 Weather: Sunny Image quality: Good Reference video image Date: February 12, 2014 9:00 Weather: Rainy Image quality: Poor (window in front of the driver's cab of the top vehicle) Part of the suction cup used to fix the digital video camera 20 to the inner surface of the glass is shown, and the contrast is very poor due to refraction of the image due to raindrops, fog, or fogging of the glass window)
Comparative section kilometers 4.5 → 30.0

  The frame matching result is partially defective, and when the error is large, the number of sleepers is about six, and it is considered impossible to detect environmental differences. In addition, the image quality is too different in the part where the frame matching is successful.

(Consideration of verification results)
The video images used to acquire the data shown in FIG. 27 were taken by running the train on the same route at different dates and times, but the stopping stations are the same, and the running speed, acceleration / deceleration position, etc. Almost identical. In the graph of (1) in FIG. 27, the start positions of the base video image and the reference video image are made to coincide with each other, and only frames of both video images are sent one by one. The acceleration / deceleration position error is reflected as it is. Therefore, the error is expanded up to 7.5 sleepers. However, since the traveling time between stations is the same, the error does not increase cumulatively, and the error finally converges. The graph (2) performs frame matching by differentiating velocity information recorded in synchronization with the video image to calculate the distance. Looking at the graph of (2), the error suddenly expands from around 25000 frames, and finally the number of sleepers is 20 or more. Frame numbers 25000 to 28000 are stops at the station, and are areas where negative acceleration is applied. As a characteristic of the speed generator, there is a decrease in accuracy due to wheel slipping in the acceleration section, which seems to increase the error cumulatively. The graph (3) is frame-matched by the algorithm used this time, and has the highest accuracy among the three graphs. The error is the number of sleepers 0 to 1.5, which is about 90 pixels at the maximum and 9% in the vertical direction of the image. When performing difference detection, of course, most objects in the screen must overlap with the matched two-frame images of the base video and the reference video. The largest deviation is at the edge of the screen, and the deviation becomes smaller as it goes to the center. Therefore, the difference can be sufficiently detected if the displacement of the sleeper at the lower end of the image measured this time is 10% or less of the image. I think that the.

Possible causes of small errors are as follows.
(1) Error due to camera angle of view movement due to both pitching (2) Error due to SURF feature matching error

  In (1), when the vehicle pitches, the line of sight of the video camera mounted on the window glass shakes up and down, and the lower end of the screen moves, so the number of sleepers that can be seen at the same point varies. This is thought to be the cause. (2) is an essential problem of this algorithm. Since similar frame images continue in a camera image moving in the front-rear direction, the SURF feature similarity may be increased in the frames before and after the correct frame depending on the image conditions. It may be the highest. In the algorithm described with reference to FIGS. 25A to 25C, the algorithm works correctly when the frame having the maximum SURF feature similarity is a frame far from the correct matching frame, but an error occurs when the frame is close.

  Next, when FIG. 28 to FIG. 31 showing images obtained as a result of frame matching of the video images of the entire Kusatsu line are seen, it can be seen that the matching is good. Observing the entire Kusatsu line, it was a diverse landscape such as a yard, residential area, countryside, and mountainous area, but the frame matching function worked well in any case. In addition, the base video image and the reference video image have fluctuations in the illumination light on the order of fine weather and cloudy weather, but it seems that this difference can be overcome.

  In the frame number graph of FIG. 32, the horizontal axis represents the number of calculations. This indicates the calculation from (6) to (12) in FIG. This corresponds to the frame of the reference video image. However, when the speed corresponding to the frame of the reference video image is 0, it is skipped without being calculated. The graph of (1) in FIG. 32 displays the frame number of the reference video image to be processed as the value on the vertical axis. Therefore, it becomes a straight line with an inclination of 1. The vicinity of 15646 frame and 57366 frame represents a stop at the station. On the other hand, the graph (2) in FIG. 32 shows the frame number of the base video image associated by frame matching. Since the time from the start of video image recording to the departure at the first station differs, the intercepts at the left end of the graphs (1) and (2) in FIG. 32 are different. If the vehicle when the base video image and the reference video image are taken is traveling at exactly the same speed and acceleration / deceleration positions, the graphs of (1) and (2) in FIG. Shift to a completely identical graph. However, in the actual graph, the graph of (2) is a delicate curve. This is presumably because the vehicle speed, acceleration / deceleration position at the time of reference video image shooting and that of the vehicle at the time of base video image shooting are different. If the frame number of the reference video image is increased due to an error in frame matching, but the corresponding frame number of the base video image is decreased, it does not increase monotonously. Moreover, when the correspondence is largely wrong, it becomes a step shape or a spike shape in a section other than the stop at the station. In FIG. 32, since these shapes are not found, it can be said that the frame matching process is operating properly as a whole.

  Next, a description will be given of the result of verifying an example of extremely different weather using a video image of the JR Nara Line.

  33 to 36 show frame matching results when the base video image is clear and the reference video image is rainy. The conditions different from the Kusatsu Line data are not only the weather, but there are many raindrops on the window glass, and there are parts where the landscape cannot be seen clearly. In some sections, fogging of the window glass was also confirmed. Furthermore, a part of the suction cup of the video camera mounting jig is reflected in the lower part of the image, which is a disadvantageous image for performing the frame matching process. Also, the speed information of the Kusatsu Line uses a speed generator, whereas the Nara Line uses the speed information obtained by the GPS receiver 38 of the vehicle motion measuring device 30. First, when checking the frame matching graph of the frame matching result in FIG. 37, the graph of (2) is monotonically increasing and has no step-like or spike-like shape except for the station stop position on the way. Is considered to be operating. 33 to 36 also appear to be correctly frame-matched at first glance, but it can be confirmed that the position of the ground child in FIG. 36 is shifted by about 3 to 4 sleepers. FIG. 35 is also difficult to confirm because the frame image of the reference video image is unclear, but it can be seen that the base video image is ahead of the reference video image.

The reason why these relatively large errors have occurred is estimated as follows.
(A) The SURF feature cannot be correctly handled due to the deformation of the image due to raindrops.
(B) A part of the image is not captured by the lower suction cup, and the corresponding points cannot be taken correctly.
(C) GPS speed information has a large error in low speed areas such as departure and stop.

  As shown in FIG. 19A, since the SURF feature is matched using the luminance fluctuation direction of the surrounding image, if the light is refracted by raindrops, the SURF feature cannot be correctly matched. In (b), since a part of the image is missing, there is a possibility that the reference video image is determined to be in a state substantially before the actual position that matches the base video image (the size of the image is different). There is. Since (c) is inherently corrected by SURF feature matching, it is not a problem without (a) and (b), but if SURF feature matching is incorrect, the error is increased. It becomes a factor.

  From the above, the performance and characteristics of the frame matching function used this time are summarized as follows.

-Compared to speed and image matching, the frame matching function that implements the algorithm used in this train position detection system is the most accurate.
・ Fluctuation of lighting environment such as fine weather and cloudy weather can ensure the frame matching accuracy that can detect the environmental change difference.
-If there is a change in the lighting environment such as clear or rainy weather (and the change in image quality due to raindrops, etc.), the accuracy of detecting the change in the environment cannot be ensured.

  As described above, according to the first embodiment, the frame matching is performed between the SURF feature of the designated frame image of the reference video image and the SURF feature of each frame image of the base video image stored in the image feature database. Since the traveling position of the train is detected based on the similarity obtained by the comparison by processing, the traveling position of the train can be detected with high accuracy and at low cost.

<Second Embodiment>
A train position and environment change detection system (hereinafter referred to as a “train position and environment change detection system”) using image processing according to the second embodiment will be described.

[Configuration of train position and environmental change detection system]
This train position and environment change detection system is obtained by adding an environment change detection function to the train position detection system according to the first embodiment. The train position detection function of the train position and environment change detection system can be realized in the same manner as the train position detection system according to the first embodiment. Since this train position detection function can detect the traveling position of the train with high accuracy, it is possible to accurately extract time-series environmental changes.

  First, before explaining the environmental change extraction technology used in this train position and environmental change detection system, the construction limit obstacle detection technology developed by the present applicant, which is the basis of this environmental change extraction technology, will be described.

  A building limit trouble detection program is installed in the data processing PC 50. This construction limit obstacle detection program reads data that is recorded and associated with video image frames and speed information in a linked display program, and sets the target imaged in the video camera image to a predetermined position and size. It is determined whether or not the virtual building limit frame is hindered. If it is determined that the virtual building limit frame is hindered, for example, the color of the building limit frame is changed and displayed on the screen.

FIG. 38 shows an example of the detection screen of the building limit trouble detection program. The functions of this construction limit obstacle detection program will be described as follows.
(1) When the “Start” button is pressed, a video camera image with speed information is reproduced, and at the same time, a construction limit trouble detection process is started.
(2) A virtual building limit frame is displayed as a dotted line (for example, a blue dotted line) on the video camera image. When an object on the video camera image located at a position overlapping the building limit frame is inside the building limit frame, the dotted line is changed to a thick dotted line (for example, a red dotted line).
(3) The frame number, speed, construction limit frame lateral size, etc. are displayed.
(4) Enter the frame number and press the “jump” button to jump the image to the specified frame. FIG. 38 shows an example in which “1720” is input as the frame number.

FIG. 39 shows a setting screen of the building limit trouble detection program. The function of this setting screen is as follows.
(1) When the “Browse” button is clicked, a file dialog for input data is displayed, and a video file and a speed information file used for building limit failure detection can be designated.
(2) After reading the video file, press the “Start Image Preview” button to play back the image and check its contents.
(3) When the “Open camera internal parameter file” button is pressed, a file dialog is displayed, and a file in which camera internal parameters (camera lens focal length, distortion correction coefficient, image center coordinates) are described can be read.
(4) When the “Open camera external parameter file” button is pressed, a file dialog is displayed, and the camera external parameters (relative distance from the center of the track surface and the posture with respect to the track surface) can be read. The orbital coordinate system is as shown in FIG. 40 (Y axis is perpendicular to the orbital plane).
(5) When the “Open building limit detection frame setting dialog” button is pressed, a virtual building limit frame setting screen for specifying how to set a virtual building limit frame for the vehicle is displayed.

FIG. 41 shows a virtual building limit frame setting screen. The function of this setting screen is as follows.
(1) As schematically shown in the upper part of the virtual building limit frame setting screen in FIG. 41, the distance d from the digital video camera 20 to the position of the virtual building limit frame and the size r of the virtual building limit frame are assumed. And the dotted line (for example, blue dotted line) on the video camera image of FIG. 38 is drawn as if the virtual building limit frame exists there. The construction limit frame indicated by the dotted line on the video camera image varies in position and size depending on d and r and camera internal and external parameters. As the image processing accuracy, it is more advantageous to set the construction limit frame so that it spreads over the entire screen as much as possible. Therefore, d is adjusted as such. Also, in order to confirm that the algorithm is working correctly, the building limit frame is intentionally increased, and r is set to a large value in order to perform an operation to react to an object that does not actually interfere with the building limit. To do.
(2) Enter 1/2 of the building limit frame width in the radius of the building limit detection frame (in meters). For example, when 1.9 m is input, a limit frame line is generated at a virtual building limit frame at a distance of 1.9 m from the track center (see FIG. 42).
(3) The assumed distance from the digital video camera 20 to the virtual building limit frame is input as the distance to the building limit detection frame (in meters). Naturally, when this distance is increased, the virtual building limit frame represented by a dotted line (for example, a blue dotted line) on the video camera image is decreased, and when the distance is decreased, the building limit frame is increased.

(Building limit obstacle detection algorithm)
The construction limit obstacle detection algorithm will be described.

  A single-lens stereo view is performed using a plurality of frames of a video camera image of the digital video camera 20 attached in front of a train vehicle traveling on the track, and an object captured by a pixel at a building limit frame position is captured. Determine whether it is inside or outside the actual building limit.

  FIG. 43 shows the relationship between two bars imaged on the same pixel as the set virtual building limit frame when viewed from the digital video camera 20. The points indicated by ● are captured by the same pixel on the video camera image, but one is inside the building limit frame and one is outside the building limit frame. In this way, it is impossible to determine whether a building limit frame is inside or outside with a single two-dimensional image.

  In view of this, the principle of stereo measurement is introduced by taking the same scene from the position where the digital video camera 20 moves forward while traveling to solve this problem. FIG. 44 shows a state in which the digital video camera 20 has advanced. Assuming that the virtual building limit frame does not move, the building limit frame on the video camera image appears to approach, and thus increases toward the outside. Considering how the two bars inside and outside the building limit frame look like in this state, the bar inside the building limit frame appears outside the virtual building limit frame image on the video camera image. The sticks standing outside are shown on the video camera image inside the virtual building limit frame image (see FIGS. 45A and B).

Therefore, it is possible to determine whether the object shown in the building limit frame position on the two-dimensional image is actually inside or outside the building limit frame by the following procedure.
(1) Prepare images before and after the digital video camera 20 moves forward (2) Calculate where on the video camera image before the virtual building limit frame moves forward (3) Building limit frame mapping pixels from the vanishing point (4) Calculate the building limit frame map after movement (5) Track how the object at the pixel position on the building limit frame has moved between the images before and after moving forward . In this case, the tracking is limited to the straight line obtained in (3). (6) It is determined whether the position after movement is inside or outside the building limit frame mapping position obtained in (4) (inside If it is, it will not interfere with the building limit frame, it will interfere with the outside)

  In the process (1), in order to establish stereo measurement, images having different positions on the rail, that is, different viewpoints are prepared. In order to stabilize the stereo visual accuracy of each frame, it is desirable that the distance between the viewpoints of the two images before and after that used for stereo vision is constant. However, since the traveling speed varies, it cannot be specified by the number of frames. Therefore, it is necessary to calculate the distance between the frames from the integration of the speed information and search for a pair image having a predetermined distance.

  In the process (2), the virtual building limit frame is projected on the video camera image using the camera external and internal parameters, assuming that the virtual building limit frame exists at a position away from the digital video camera 20 by the specified distance. Since the construction limit frame is on a plane, it can also be obtained by homography conversion.

  The process (3) is performed for the tracking process performed in (5). When the digital video camera 20 moves forward, all the points shown in the video camera image move outward from the vanishing point on a straight line connecting the points (the effects of the curve and vibration are omitted here). To do). This straight line is called an epipolar line. FIG. 46 shows the vanishing point and the moving direction of the image.

  In (4), the same processing as in (2) is performed by shortening the distance between the virtual building limit frame and the digital video camera 20 by the amount of advancement. FIG. 47 shows an image in which the virtual building limit frame before and after the advance is displayed on the image before the advance. Here, the building limit frame indicated by the dotted line in which the dotted line of the small point (for example, the blue dotted line) and the dotted line of the larger point (for example, the red dotted line) are mixed is the one before the advance, the dotted line of the larger point (for example, The building limit frame indicated only by the red dotted line) is the one after the advance.

  In (5), it is searched where the object that was on the building limit frame in the image before moving has moved after moving forward. At this time, a small region of the image on the building limit frame before the advance is used as a template image, and the image after the advance is searched for where the same image as the template image is present. At this time, the search is performed two-dimensionally, but the one-dimensional search on the epipolar line explained in (3) may be performed (epipolar constraint). Also, the search range of the straight line is obtained from the building limit frame shown by the dotted line shown in FIG. It will be to the place beyond the building limit frame shown only by the dotted line (for example, red dotted line) of a big point (refer FIG. 48). If the distance between the building limit frames before and after the advance is 1, the tracking distance is slightly larger than 1. This is to detect an object protruding further inside than the building limit frame.

  As shown in FIG. 49, the tracking position is obtained by cutting out a small area on the building limit frame of the image of the point 1 as a template image and moving it on the epipolar line on the image of the point 2, Calculate the similarity profile. In this case, the peak position of the similarity profile is set as the tracking position, and the determination is made based on whether or not the position exceeds the building limit frame position at the point 2.

  Now, in extracting environmental changes, it seems to be important whether or not the approach of vegetation around the track is progressing. However, the vegetation has a very fine structure, so that the state of the image easily changes depending on the lighting environment. For this reason, in the method of detecting the difference between the two-dimensional images, a lot of noise occurs and it is not easy to extract the true difference. Therefore, in the second embodiment, the stability of the environment change extraction is improved by using the three-dimensional information of the object developed by the above-described construction limit trouble detection technique. Considering video images taken at two different times, specifically in the morning and evening, the lighting environment is completely different between the morning and evening, so it is difficult to distinguish the difference between the lighting environment and the environmental difference. . On the other hand, when the three-dimensional shape is measured in the morning and evening, it is possible to reduce the influence of changes in the lighting environment.

[Outline of environment change extraction processing]
In the above construction limit obstacle detection technology, as shown in FIG. 52, a virtual building limit frame (3.8 m width) is set on the rail in front of the video camera, and the imaged object is in this frame. An algorithm is used to detect whether or not the bite is invaded. In order to determine from the two-dimensional video image whether or not the virtual building limit frame has been cut, stereo vision using the fact that the video camera travels in the forward direction is used. On the other hand, in the environment change extraction, this virtual building limit frame is expanded in the left-right direction and the height direction as shown in FIG. That is, a multiplexed virtual building limit frame (the innermost frame is a standard frame and the three outer frames are enlarged frames) is used. As a result, a difference can be detected by checking whether or not the vegetation that did not interfere with the building limit frame in the first shooting will interfere with the building limit frame during subsequent shooting.

[Details of processing technology for environmental change extraction]
(1) Expansion of virtual building limit frame As shown in FIG. 53, the number of virtual building limit frames is expanded to set an arbitrary number of virtual building limit frames. As a result, it is possible to capture the progress of interference of obstacles in the direction of the track with arbitrary accuracy, and to accurately detect changes in the environment. The expansion was done by widening the virtual obstacle frame developed with the construction limit obstacle detection technology in the left-right and height directions. In order to perform expansion easily, it was made feasible by specifying the number and widening amount in meters. In the following examination and verification, an obstacle near the track was detected using one expanded virtual building limit frame (width 7.6 m).

(2) Detection of difference between base video image and reference video image When calculating the difference based on the detection results of both the base video image and the reference video image, the presence / absence and number of failures between the corresponding frames are calculated. If simple subtraction is performed, it may be difficult to obtain a desired result. FIG. 54 is a graph in which the number of virtual building limit obstacle frames is plotted at a location where a utility pole standing near the track is shown in the video image. The horizontal axis direction of the graph is synchronized using the result of frame matching. Video images taken by trains traveling at several tens of kilometers per hour are taken at several tens of frames per second (= several tens of centimeters per frame), and each frame shutter between the base video image and the reference video image. There is a difference of several tens of centimeters in the position where the mark is cut. Therefore, the correspondence between these frames includes an error of several tens of centimeters at the minimum. A simple subtraction between the base video image and the reference video image ignores this error, resulting in an unintended result as shown above. When the data of FIG. 54 is simply subtracted, the result shown in FIG. 55 is obtained. Originally, since the same utility pole is hindered in both the base video image and the reference video image, it is desirable that no difference in the third stage in FIG. 55 appears.

  In addition, in the graph shown in FIG. 55, there is a part where the plants near the track are partially detected as noise. In order to stabilize the processing, it is assumed that a certain amount of such noise is detected, and threshold processing is performed (in the graph above, noise is removed when calculating the third stage).

  Based on the above, this train position and environment change detection system calculates the difference according to the following method and performs robust detection.

(1) Subtract in consideration of the frame of interest and its vicinity.
(2) If the number of trouble points in the frame of interest is below a certain level, ignore it.

A specific algorithm is shown below.
(Difference calculation algorithm)
(A) The threshold value of the neighboring frame is t _1. (B) The threshold value of the number of trouble points is t _2. (C) The frame number of interest of the base video image is N.
(D) The frame number M of the reference video image corresponding to N is obtained using the frame matching result.
(E) A virtual building limit frame is provided in the Nth frame of the base video image, and the number of obstacle points CG _b (N) is obtained. A virtual building limit frame is provided in the Mth frame of the reference video image, and the number of obstacle points CG _r (M) is obtained. If both CG _b (N) and CG _r (M) are t ₂ or less, the difference Diff (N) of the Nth frame is set to 0. The process ends at this point.
(F) The number of trouble points is obtained in each frame of the interval [N−t ₁ , N + t ₁ ] of the base video image. If the number of trouble points in any frame is t ₂ or more, it is determined that there is a trouble object.
(G) The number of trouble points is obtained in each frame of the section [M−t ₁ , M + t ₁ ] of the reference video image. If the number of trouble points in any frame is t ₂ or more, it is determined that there is a trouble object.
If any of (H), (F), and (G) has an obstacle, the difference Diff (N) = 0 of the Nth frame is set. If there is no obstacle in both (F) and (G), the Nth frame difference Diff (N) = 0 is set. Otherwise, the difference Diff (N) = 0 of the Nth frame is obtained by the following equation.
However, Σ _p is a function for all trouble points, and hit _N (p) is a function that is 1 if the trouble point p in the Nth frame is troubled, and 0 otherwise. That is, the number of trouble points with different troubles makes the frame different.

  It is determined that the frame determined as Diff (N) ≠ 0 by the above procedure is different between the two video images, and the frame determined as Diff (N) = 0 is not different.

[Verification on actual routes]
The processing was applied to the video taken by the train traveling on the actual route, and the following two points were verified.

(1) Kusatsu Line Up 17.1km Point Base Video Image Shooting Date: September 10, 2014 Reference Video Image Shooting Date: September 12, 2014 (2) Kusatsu Line Up 22.9km Point Base Video Image Shooting Date: September 10, 2014 Reference video image shooting date: September 12, 2014

(A) Verification result at 17.1 km point on Kusatsu Line Ascending time The video image was visually confirmed to find a difference, and the found location was targeted. 56 shows the state of the processing target section of the base video image, and FIG. 57 shows the state of the processing target section of the reference video image. “# 000000” below each image in FIGS. 56 and 57 represents the frame number of the video image. Of these, the following three locations ((1) to (3)) have been focused on as differences detection targets.

(1) Location where both the base video image and the reference video image have obstacles A utility pole stands near the track, and any video image is considered to interfere with the virtual building limit frame. This corresponds to # 154474 in FIG. It is assumed that it is determined that there is no difference in the difference detection at the location.

(2) Location where there is no obstacle in both base video image and reference video image There is no object in the vicinity of the track, and it is considered that there is no obstacle to the virtual building limit frame in any video image. This corresponds to # 133138 and # 154410 in FIG. It is assumed that it is determined that there is no difference in the difference detection at the location.

(3) A place where there is an obstacle only in the base video image A person working in a gutter beside the track is shown only in the base video image, and this is a place that is considered to interfere with the virtual building limit frame only by the base video image. It is assumed that it is determined that there is a difference in the difference detection at the location. This corresponds to # 133189 in FIG. 56 and # 154548 in FIG.

Detection processing was performed on the video image including the above three locations. As the parameters of the algorithm, the threshold value of the neighboring frame is t ₁ = 3, and the threshold value of the number of trouble points is t ₂ = 4. The detection result of the obstacle by the virtual building limit frame is as shown in FIG. 58 and FIG. 58 shows a base video image, and FIG. 59 shows a reference video image. The red point in the image is a problem, and the blue point is a problem.

  The processing results are as shown in the graph of FIG. In this graph, the horizontal axis represents the time axis and the vertical axis represents the degree of detection. The first graph shows the number of obstacles in the virtual building limit frame in the base video image, the second graph shows the number of obstacles in the virtual building limit frame in the reference video image, and the third graph shows the difference between the two video images. It is the result of calculation.

(B) Result of verification at 29.1 km on the Kusatsu line Ascending the video image, looking for a difference, we looked at the found part. 61 and 62 show images in which the state of the target portion and the result of obstacle point detection by the virtual building limit frame (width 7.6 m) are superimposed and displayed. FIG. 61 shows the processing target section of the base video image, and FIG. 62 shows the processing target section of the reference video image. Of these, we focused on the weeds near the track as the object of difference detection. The base video image (# 78304 in FIG. 38) shows that weeds are growing, but the reference video image (# 58437 in FIG. 39) has been weeded and is not shown. Therefore, only the base video image is considered to hinder the virtual building limit frame.

Detection processing was performed on a video image including the above-described part. As the algorithm parameters, the threshold value of the neighboring frame is t ₁ = 3, and the threshold value of the number of trouble points is t ₂ = 4 (the same as the above-described verification result (A)). The processing results are shown in the graph shown in FIG. In this graph, the horizontal axis represents the time axis and the vertical axis represents the degree of detection. The first graph is the number of obstacles in the virtual building limit frame in the base video image, the second graph is the number of failures in the virtual building limit frame in the reference video image, and the third graph is the difference between the two video images. It is a result.

[Consideration of verification results]
The following considerations were made for each point of interest in the verification result of (A).

(1) Location where there is an obstacle in both the base video image and the reference video image In both the base video image and the reference video image, there is an obstacle in the power pole near the track, and the number of obstacles is increasing as shown in FIG. You can see from the graph. At the same location, there is an error in frame matching, and there is a slight shift in the peak position of the number of troubles originating from the utility pole. When the difference is calculated as it is, two peaks are generated as shown in FIG.

  The difference calculation algorithm used this time absorbs the error well by calculating the difference considering the frame of interest and its neighboring frames. Therefore, no difference is detected in the third graph of FIG.

(2) Locations where there are no obstacles in both the base video image and the reference video image The number of obstacles is 0 in both the base video image and the reference video image, and no difference is detected in the third graph in FIG. In addition, there are vehicles and workers on the side road located away from the track at the location, but these are not detected as obstacles.

(3) Location where only the base video image has a hindrance object It can be seen from the graph of FIG. 55 that the person working in the side groove near the track is hindering only the base video image, and the number of hindrances is increasing. When the difference is calculated, it becomes like the graph in the third row of FIG. 55, and it can be read that there is a difference in the part.

The following considerations were made for each point of interest in the verification result of (B).

  It can be seen from the graph in FIG. 63 that at the point of interest, weeds near the track are hindered in the base video image of FIG. 61 and the number of hindrances is increasing. On the other hand, in the reference video image of FIG. 62, weeds are weeded out, and there is almost no problem. When the difference is calculated, it becomes like the graph in the third row of FIG. 63, and it can be read that there is a difference in the part.

  In addition, the difference is hardly detected in the location where the utility pole near the track interferes with the virtual building limit frame. This is because the determination of whether there is a difference does not take into consideration the left or right side of the track, so that the trouble caused by the utility pole is confused with the trouble caused by weeds. The obstacle determination and the difference detection by the virtual building limit frame are divided on the right and left of the track, that is, the graph of FIG. 63 has six stages (first stage: base video image trouble detection left, second stage: base video image trouble detection right, 3 Stage: Reference video image trouble detection left, 4th stage: Reference video image trouble detection right, 5th stage: difference detection left, 6th stage: difference detection right) It becomes possible to cope with.

  As described above, it can be seen that the presence / absence of a difference can be correctly determined in accordance with the presence / absence of an obstacle near the track.

[Supplement to the examination of verification results]
(Application to utility poles for detecting obstacles with virtual building limit frames)
FIG. 64 shows the result of detecting the trouble at the location (1) in the reference video image frame by frame.

In the obstacle detection by the virtual building limit frame, detection is realized by searching for a similar rectangular area in the next frame using a pixel in the rectangular area near the target point of the target frame as a template. Here, the calculation of the similarity of the pixels in the rectangular region is performed by the following normalized cross-correlation (ZNCC).

  The position of the rectangular area where the value of ZNCC is maximum in the vicinity of the point of interest is obtained as a search result. However, when the maximum value of ZNCC is too small, it is possible to perform processing more stably if it is determined that the search is not applicable, so a lower limit threshold is provided. As a case where this lower limit threshold value can be applied, there is a case where the variance of pixel values is small because the rectangular area near the target point is dark.

  In the example of FIG. 59, the light is backlit due to the positional relationship between the sun and the camera, the surface of the utility pole is crushed in black, and the dispersion of pixel values is small. Therefore, it can be said that this is a disadvantageous situation for the above-mentioned similarity calculation by ZNCC. On the other hand, in the left and right contour portions of the utility pole, a scene with a large pixel value dispersion other than the utility pole is included in the target rectangle, and the disadvantageous situation for the calculation of ZNCC is reduced. Therefore, if the obstacle point is detected by the virtual building limit frame in several frames before and after the utility pole, two peaks corresponding to the left and right contours of the utility pole are obtained. It should be noted that this event depends on backlight conditions, vehicle speed and video frame rate.

(Examination of algorithms to detect changes in the degree of hindrance to objects near the track)
In (2) Detection of difference between base video image and reference video image in [Details of environment change extraction processing technology] above, an algorithm for detecting the presence or absence of obstacles near the track (= presence or absence of environmental change) Explained. Aiming at further sophistication of environmental change detection, we will consider the case of change as a case of change, assuming that the utility pole approaches the track.

  The matter described so far is detection processing using a virtual building limit frame having a width of 7.6 m (one side: 3.8 m). Assuming that the utility pole has approached 20 cm from the track side from here, processing is performed using a virtual building limit frame having a width of 7.2 m (one side: 3.6 m) as shown in FIG. It is assumed that the process of tilting the power pole is detected by three video camera images taken at the time point described below.

(A) Video image taken at the time before the utility pole tilts (B) Video image taken at the time when the utility pole tilts and interferes with the 7.6m wide virtual building limit frame (C) The utility pole is from (B) A video image taken at the time of further tilting and obstructing the 7.2m-wide virtual building limit frame

  When the obstacle detection process using the virtual building limit frame is executed for the three video images six times as described below, it is possible to determine whether or not the obstacle exists.

(1) For (A), processing using a 7.6m wide virtual building limit frame ... No hindrance (2) For (A), processing using a 7.2m wide virtual building limit frame ... No hindrance (3) For (B), processing using a 7.6m-wide virtual building limit frame ... There is a problem (4) For (B), processing using a 7.2m-wide virtual building limit frame ... No problem (5) For (C), processing using a 7.6m-wide virtual building limit frame ... There is a problem (6) For (C), processing using a 7.2m-wide virtual building limit frame ... There is a problem

  When the difference detection process is executed for the six trouble detection results for the following four times, it is possible to determine whether or not there is a difference.

(I) A difference detection process is performed on (a) and (c) ... there is a difference (II) A difference detection process is performed on (b) and (d) ... no difference (III) (a) and (e) The difference detection process is performed on the difference (IV) (b) (f). The difference detection process is performed on the difference (IV), (b), and (f).

  The following points can be understood from the above processing results.

・ In the difference detection using the obstacle detection result by the 7.6m width virtual building limit frame, the difference detection result is the same between (A) (B) and (A) (C) ((I) and (III))).
-In the difference detection using the obstacle detection result by the 7.2m width virtual building limit frame, the difference detection results are different between (A) (B) and (A) (C) ((II) and (IV ))

  That the difference detection result at the time of the two photographings differs depending on the width of the virtual building limit frame means that there is a change in the obstacle condition between the two shooting points. Therefore, it is suggested that the above algorithm can detect a change in the obstacle condition of an object near the track.

  As described above, according to the second embodiment, the traveling position of the train can be detected with high accuracy and at low cost, and thereby the time-series environment around the track on which the train travels can be detected. Changes can be detected easily and accurately.

<Third Embodiment>
In the third embodiment, software capable of handling a plurality of video images is incorporated into the train position detection system according to the first embodiment or the train position and environment change detection system according to the second embodiment. Will be described.

[Overview of software functions]
(1) Single playback function The two video images of the base video image and the reference video image are independently played back and paused.
(2) Simultaneous playback function Plays back two video images of a base video image and a reference video image at the frame rate at the time of shooting. However, since these two video images have different vehicle speeds at the time of shooting, the displayed scenes do not match when a certain time has elapsed from the start of playback. Since this discrepancy expands with time, it is difficult to compare the two video images and confirm the difference in the scene.
(3) Synchronous playback function using frame matching results Frame matching is used to adjust and display frames so that scenes in the video match at an arbitrary time. Considering the example of (2), scene mismatch occurs due to vehicle speed deviation after the start of playback, but synchronized playback is realized by adjusting the frame number to interpolate it. As a result, the scenes coincide with each other regardless of the passage of time. Therefore, it is possible to compare the two video images and examine the differences in the scenes.
(4) Frame matching execution function An interface necessary for performing frame matching between two video images of a base video image and a reference video image is provided, and processing can be performed while viewing the video. Video data and processing parameters are set when the software is started, and a series of processing is executed.

[Application Operations]
The application screen is shown in FIG. On this application screen, the operation is performed according to the following procedure for each function described in [Overview of Software Functions].

(1) Single playback (A) Press the [INPUT] button to load the base video image.
(B) Press the [|>] button to start playback of the base video image.
Similarly, the reference video image can be played back independently by the buttons (C) and (D).
(2) Simultaneous playback (A) Press the [INPUT] button to load the base video image.
(C) Press the [INPUT] button to read the reference video image.
(F) Press the [Simultaneous] button to open the file selection dialog. Specify the frame matching result file.
(D) Press the [|>] button to start playback.
(3) Synchronous playback (A) Press the [INPUT] button to load the base video image.
(C) Press the [INPUT] button to read the reference video image.
(F) Press the [Synchronize] button to open the file selection dialog. Specify the frame matching result file.
(D) Press the [|>] button to start playback.

  According to the third embodiment, in addition to the advantages similar to those of the first embodiment or the second embodiment, the advantage that the handling of the base video image and the reference video image becomes easy is obtained. Can do.

<Fourth embodiment>
In the fourth embodiment, in the train position detection system according to the first embodiment or the train position and environment change detection system according to the second embodiment, a technique for determining the kilometer distance with respect to the frame image of the reference video image Will be described.

  As shown in FIG. 67, a digital video camera 20, a GPS receiver 38, a longitudinal accelerometer 90, and an image database 100 installed on a train are connected to a kilometer determination computer 80 using an image. The longitudinal accelerometer 90 is for measuring the longitudinal acceleration of the train vehicle. When a speed generator can be used, the GPS receiver 38 and the longitudinal accelerometer 90 may be used instead. The image database 100 records the video image and speed information along the line on the first run of the same route to form a base video image. The contents of the base video image, for example, manually record the kilometer of each frame of the base video image. Create by allocating the process accurately. In the second run, the reference video image at the time of running is compared with the base video image stored in the image database 100, and the image content of the current reference video image is used by using the speed information as auxiliary data. Determine how many kilometers is in the base video image. By doing so, measurement data obtained by traveling on the same track can be accurately and stably associated with about a kilometer.

  In this kilometer determination technology, the accuracy of the speed acquired by the GPS receiver 38 or the accuracy of the speed measured by the longitudinal accelerometer 90 is corrected by the image information. Therefore, the GPS receiver 38 and the longitudinal accelerometer 90 In both cases, accuracy is not so required, and an inexpensive one is sufficient. Furthermore, since the digital video camera 20 is not required to have much frame rate and image resolution, an inexpensive one is sufficient.

  67 is connected to the management measuring instrument 110 as shown in FIG. That is, as shown in FIG. 68, the management measuring instrument 110 is connected to a kilometer determination computer 80 using an image. At the time of measurement, a measurement trigger pulse is sent from the kilometer determination computer 80 using an image to the management measuring instrument 110. The management measuring instrument 110 acquires various data at the timing of this measurement trigger pulse. The digital video camera 20 is imaged in conjunction with the measurement trigger pulse. That is, since the cumulative number of measurement trigger pulses corresponds to the frame number of the image captured by the digital video camera 20, this information is output to a file at the time of shooting as shown in the lower left of FIG. Keep it. If the above-mentioned kilometer determination process is performed from the video image recorded in this manner, the measurement trigger pulse number and the kilometer are associated with each other. Since the number of samples of the management measuring instrument 110 is the number of measurement trigger pulses, the kilometer is finally associated directly with the recorded data of the management measuring instrument 110.

  As described above, according to the fourth embodiment, the following various advantages can be obtained. That is, it is possible to determine a high-precision kilometer with a small and inexpensive device. By accurately associating the kilometer with the recorded data, if it is determined from the recorded data that field work is necessary, it is possible to accurately identify the field based on this kilometer, thereby improving the efficiency of field work. As a result, labor costs can be significantly reduced. In addition, it becomes possible to accurately compare the situation at the same point, and it is possible to automatically extract time-series environmental changes, thereby enabling advanced management of vast railway facilities. .

<Fifth embodiment>
[Train position detection system]
As shown in FIG. 69, in the train position detection system according to the fifth embodiment, a computer 200 in which an image synchronization program is installed is used instead of the image synchronization apparatus 10 used in the first embodiment. As the computer 200, for example, a PC can be used. Others are the same as in the first embodiment.

  According to the fifth embodiment, the same advantages as those of the first embodiment can be obtained.

  Although the embodiment of the present invention has been specifically described above, the present invention is not limited to the above-described embodiment, and various modifications based on the technical idea of the present invention are possible.

  For example, the numerical values, structures, configurations, shapes, circuits, and the like given in the above-described embodiments are merely examples, and different numerical values, structures, configurations, shapes, circuits, and the like may be used as necessary.

  DESCRIPTION OF SYMBOLS 10 ... Image synchronizer, 11 ... Digital / analog converting circuit, 12, 25 ... Input terminal, 13 ... Output terminal, 14 ... TTL level converting circuit, 15 ... Audio voltage level converting circuit, 20 ... Digital video camera, 26 ... External Microphone, 30 ... Vehicle sway measurement device, 30a ... Housing, 30b, 30c ... Handle, 30d ... LCD, 30e ... Power switch, 31 ... Sensor processing unit, 32 ... Display processing unit, 33 ... Position information processing unit, 34 ... Data storage unit 35 ... External interface control unit 36 ... Power source control unit 37 ... High speed serial bus 40 ... GPS receiver 39 ... Speed generator 50 ... Data processing PC 200 ... PC

Claims (13)

The SURF feature extracted from each frame image of the first moving image recorded by shooting the space ahead of the traveling direction with the camera while traveling on a train on which a camera capable of shooting the moving image is installed, and Having an image feature database storing location information associated with the SURF feature;
The designated frame image of the second video recorded by photographing the space ahead of the traveling direction with the camera while running the train on the track at a date and time different from the date and time when the first video was recorded. The train's travel position is detected based on the similarity obtained by comparing the SURF feature of each frame image of the first moving image stored in the image feature database with the frame matching process. Train position detection system using image processing characterized by   The train position detection system using image processing according to claim 1, wherein the position information is kilometer information associated with each frame image of the first moving image.   The train position detection system using image processing according to claim 1, wherein the position information is distance information obtained by speed integration from speed information associated with each frame image of the first moving image. .   4. The train position detection system using image processing according to claim 3, wherein the speed information is acquired by a speed generator or a GPS receiver installed in the train.   The similarity is between the frame image of the first moving image and the specified frame image of the second moving image with respect to the number of SURF feature points extracted from the specified frame image of the second moving image. The train position detection system using image processing according to any one of claims 1 to 4, wherein the ratio is the ratio of the number of SURF feature points with corresponding points.   A pair of SURF feature points having corresponding points between the frame image of the first moving image and the designated frame image of the second moving image is a pair having the shortest Euclidean distance of the feature description vector of the SURF feature. The train position detection system using image processing according to claim 5.   The position of the train is detected by reflecting the similarity as a weight in the estimation result of the train position based only on speed information associated with each frame image of the second moving image. A train position detection system using the image processing according to any one of claims 1 to 6.   When extracting a SURF feature from each frame image of the first moving image and the second moving image, a lower center trajectory portion and a central remote landscape portion of the frame image are masked with a mask image. A train position detection system using the image processing according to any one of claims 1 to 7.   A longitudinal accelerometer and a GPS receiver are installed on the train, and the first moving image and the longitudinal accelerometer are captured by photographing the space ahead of the traveling direction with the camera while the train is traveling on the track. The integration of the measured longitudinal acceleration or the speed information acquired by the GPS receiver is recorded, and an image database is created by allocating about a kilometer to each frame image of the recorded first moving image. Stored in the image database and the second movie recorded by shooting the space ahead of the traveling direction with the camera while running the train on the track at different dates after the date when the video was recorded. By comparing the first moving image with the first moving image, the designated frame image of the second moving image becomes which frame image of the first moving image. Train position detection system that uses the image processing according to claim 1, wherein the determining by determining whether the corresponding kilometrage corresponding to the frame image of the second video.   The train position detection system using image processing according to any one of claims 1 to 9, wherein the camera is a video camera or an industrial camera. The SURF feature extracted from each frame image of the first moving image recorded by shooting the space ahead of the traveling direction with the camera while traveling on a train on which a camera capable of shooting the moving image is installed, and Having an image feature database storing location information associated with the SURF feature;
The designated frame image of the second video recorded by photographing the space ahead of the traveling direction with the camera while running the train on the track at a date and time different from the date and time when the first video was recorded. The train's travel position is detected based on the similarity obtained by comparing the SURF feature of each frame image of the first moving image stored in the image feature database with the frame matching process. Train position and environment change detection system using image processing characterized by.   While the vehicle in which the camera is installed travels on the track, the camera captures a space ahead in the direction of travel and associates it with each frame image of the second video at different positions on the track. Time-series environmental changes around the trajectory are detected by comparing two images taken at a position where the distance between them calculated using the obtained velocity information becomes the selected distance. The train position and environment change detection system using image processing according to claim 11.   The train position and environment change detection system using image processing according to claim 11 or 12, wherein virtual obstacle frames are set in multiple stages in a direction perpendicular to the traveling direction of the train.