JP2021192181A - Abnormality notification system and abnormality notification method - Google Patents

Abstract

To notify a driver seat of a train of an abnormal state of a railroad crossing to support safe driving by a driver.SOLUTION: A server device 40 transparently superimposes a field image A and a teacher image B to generate a composite image C, and generates a trained model M based on the composite image C and correct answer data R to notify an information processing device 30 of the generated trained model M. The information processing device 30 generates an abnormality score for an input image E using this trained model M to notify a monitoring center device 60 of the abnormality score. The monitoring center device 60 notifies a display device of a train 70 of the abnormality score when the abnormality score is equal to or higher than a predetermined determination threshold value.SELECTED DRAWING: Figure 1

Description

The present invention relates to an abnormality notification system, a server device, a learned model generation program, and an abnormality notification method capable of determining an abnormality at a railroad crossing and promptly notifying the train approaching the railroad crossing of the abnormality.

Conventionally, a railroad crossing where a railroad track and a road intersect in a plane is a dangerous area where a moving object such as a pedestrian or a car and a train may collide with each other and cause a serious accident. For this reason, an alarm and a barrier are installed at the railroad crossing, and when a train approaches within a certain distance of the railroad crossing, two or more red flash lights provided on the railroad crossing blink alternately and from the alarm. It makes a clicking sound and then lowers the barrier bar of the automatic barrier to prevent moving objects from entering the railroad crossing.

Even if the invasion of the moving object into the railroad crossing by the blocking bar is suppressed in this way, there are cases where the train comes into contact with the moving object in the railroad crossing when passing through the railroad crossing, leading to a serious accident. Therefore, there is known a technique of capturing a railroad crossing with a fixed camera and displaying an image of the railroad crossing on a display device mounted on a driver's seat of a train (see, for example, Patent Document 1).

JP-A-2017-0877978

However, in the case of Patent Document 1, since a real-time image of a railroad crossing is displayed on a train display device, the driver must predict the presence or absence of an abnormality while watching this image. Since a wide variety of moving objects pass through railroad crossings until just before the train passes, it is difficult for the driver to predict anomalies while watching the images in advance.

As a result, even if the image is displayed on the display device of the train using this Patent Document 1, the driver of the train cannot properly deal with the situation inside the railroad crossing, and as a result, a serious accident may occur. ..

The present invention has been made to solve the above-mentioned problems, and is an abnormality notification system and a server device capable of notifying the driver's seat of a railroad crossing of an abnormal state of a railroad crossing and supporting safe driving by a driver. , A trained model generation program and an abnormality notification method.

In order to solve the above problems, the present invention can communicate with the information processing device arranged in or near the monitoring area of the crossing through which the railroad vehicle traveling on the track or a track similar thereto passes. An abnormality notification system having a server device and a monitoring center device for monitoring the railroad vehicle, detecting an abnormality in the monitoring area and notifying the railroad vehicle, wherein the information processing device is located around the railroad crossing. It has an image pickup means for capturing a site image and a site image notification means for notifying the server device of the site image captured by the image pickup means, and the server device stores teacher information including a predetermined teacher image. A trained model generation means that generates a trained model based on a storage means, a field image acquisition means that acquires the field image from the information processing device, a teacher image stored in the storage means, and the field image. And a trained model notification means for notifying the information processing apparatus of the trained model generated by the trained model generation means, and the information processing apparatus uses the trained model received from the server apparatus. The abnormality determination is performed, and the abnormality determination result is notified to the monitoring center device.

Further, in the above invention, in the above invention, the trained model generation means is a composite image generation means for generating a composite image by synthesizing a teacher's image stored in the storage means with the field image, and the synthesis. It is characterized by having a learning means for generating a trained model by repeating supervised learning by applying a composite image and an abnormality score generated by the image generation means to a predetermined multi-layer neural network.

Further, the present invention is characterized in that, in the above invention, the composite image generation means superimposes the teacher image stored in the storage unit on the field image in a translucent state to generate a composite image. ..

Further, in the above invention, in the above invention, when the information processing apparatus performs an abnormality at the crossing, the input image captured by the imaging means is input to the trained model and output from the trained model. It is characterized in that the abnormality determination result is calculated based on the risk probability to be determined.

Further, the present invention is a server device capable of communicating with an information processing device arranged in or near a monitoring area of a crossing through which a railroad vehicle traveling on a track or a track similar thereto passes, and is a predetermined teacher image. It has been learned based on a storage means for storing teacher information including the above, a field image acquisition means for acquiring a field image captured by the information processing apparatus, a teacher image stored in the storage means, and the field image. It is characterized by having a trained model generation means for generating a model and a trained model notification means for notifying the information processing apparatus of the trained model generated by the trained model generation means.

Further, the present invention is a trained model generation program executed by a server device capable of communicating with an information processing device arranged in or near a monitoring area of a crossing through which a railroad vehicle traveling on a track or a track similar thereto passes. The storage procedure for storing teacher information including a predetermined teacher image in the storage unit, the site image acquisition procedure for acquiring the site image captured by the information processing apparatus, and the teacher stored in the storage unit. It is characterized in that a computer is made to execute a trained model generation procedure for generating a trained model based on an image for use and the field image.

Further, the present invention comprises an information processing device arranged in or near a monitoring area of a crossing through which a railroad vehicle traveling on a track or a track similar thereto passes, a server device capable of communicating with the information processing device, and the above. It is an abnormality notification method in an abnormality notification system having a monitoring center device for monitoring a railroad vehicle, detecting an abnormality in the monitoring area, and notifying the railroad vehicle, wherein the information processing device is a site around the railroad crossing. An imaging step of capturing an image, a site image notification step of notifying the server device of a site image captured by the information processing apparatus in the imaging step, and a teacher information including a predetermined teacher image by the server device. Learning based on a storage step of storing in the storage unit, a site image acquisition step in which the server device acquires the site image from the information processing device, and a teacher image and the site image stored in the storage unit. A trained model generation step for generating a trained model, a trained model notification step for notifying the information processing device of the trained model generated by the trained model generation step, and the information processing device from the server device. It is characterized by including an abnormality determination result notification step of performing an abnormality determination using the received trained model and notifying the monitoring center device of the abnormality determination result.

According to the present invention, it is possible to notify the driver's seat of a railroad crossing of an abnormal state of a railroad crossing and support safe driving by the driver.

FIG. 1 is an explanatory diagram of an outline of an abnormality notification system according to an embodiment. FIG. 2 is a functional block diagram showing the configuration of the information processing apparatus shown in FIG. FIG. 3 is a diagram showing an example of the trained model shown in FIG. FIG. 4 is an explanatory diagram for explaining the calculation of the risk score. FIG. 5 is a functional block diagram showing the configuration of the server device shown in FIG. FIG. 6 is an explanatory diagram for explaining an outline of supervised learning in the server device shown in FIG. 1. FIG. 7 is an explanatory diagram for explaining the composition of the two images. FIG. 8 is a functional block diagram showing the configuration of the monitoring center device shown in FIG. 1. FIG. 9 is a flowchart showing a procedure for generating a trained model in the server device shown in FIG. FIG. 10 is a flowchart showing a processing procedure in the information processing apparatus shown in FIG. FIG. 11 is a diagram showing an example of the hardware configuration of the server device shown in FIG.

Hereinafter, preferred embodiments of the abnormality notification system, the server device, the trained model generation program, and the abnormality notification method according to the present invention will be described in detail with reference to the accompanying drawings.

<Overview of abnormality notification system>
First, the outline of the abnormality notification system according to the present embodiment will be described. FIG. 1 is an explanatory diagram for explaining an outline of the abnormality notification system according to the present embodiment. In this embodiment, a case where the application is applied to a railroad crossing through which a train passes will be described. In the following, the case of using a trained model of CNN (Convolutional Neural Network) consisting of a program will be mainly described.

Conventionally, railroad crossings where railroad tracks and roads intersect in a plane are dangerous areas where moving objects such as pedestrians, bicycles, carts, motorcycles or automobiles may collide with trains and cause serious accidents. For this reason, an alarm and a barrier are installed at the railroad crossing, and when a train approaches within a certain distance of the railroad crossing, two or more red flash lights provided on the railroad crossing blink alternately and from the alarm. It emits a warning sound and announces the arrival of a train to moving objects. After that, the barrier bar of the automatic barrier is lowered to prevent the moving object from entering the railroad crossing.

However, a train consisting of heavy vehicles weighing 30 to 40 tons per car cannot be stopped in a short time due to the emergency braking operation, and it takes several hundred meters from the time the train applies the emergency brake to the time the train stops. It takes several tens of seconds for the distance. Therefore, the train driver needs to grasp the abnormal situation in the railroad crossing as soon as possible.

Therefore, there is known a technique of displaying an image taken by a fixed camera installed at a railroad crossing on a display device mounted on the driver's seat of the train when the train reaches a predetermined position in front of the railroad crossing (Japanese Patent Laid-Open No. 2017). -087798 (see). According to this prior art, for example, when a car is stuck in a railroad crossing, the driver can detect the situation and perform an emergency braking operation.

However, in this prior art, the image that the train driver can see on the display device is the real-time image itself in the railroad crossing at the same time, so the driver shows signs of abnormality in the railroad crossing. It can be overlooked.

Therefore, in the present embodiment, the trained model M is provided in the information processing device 30 installed at the railroad crossing, the image captured by the camera 10 is input to the trained model M, and the abnormality probability output from the trained model M is obtained. The abnormality score based on the above is calculated and transmitted to the monitoring center device 60. The monitoring center device 60 transmits an abnormality score to the display device of the driver's seat of the train 70, and displays the abnormality score on the display device to reduce the situation where the driver of the train 70 overlooks the abnormality at the railroad crossing. Is possible. Along with this abnormality score, the image of the railroad crossing can be notified to the display device of the driver's seat of the train 70 and displayed on the display device.

This "abnormal score" is a score obtained by scoring a factor of danger, that is, a degree of possibility of causing an abnormality when the train passes the railroad crossing, and is, for example, any of 0 to 100 points. The closer it is to 100 points, the higher the possibility of abnormality. From the trained model M, the "abnormal probability (probability of causing an abnormality)" is output as a value including a decimal point of 0.00 to 1.00. The information processing apparatus 30 weights this abnormality probability by multiplying it by a coefficient such as a parameter of an environmental factor (weather, etc.) and a parameter of an accident rate at a railroad crossing, and sets a value normalized to a value of 0 to 100 as an abnormality score. be able to.

The server device 40 is a device that performs supervised learning on the CNN to generate a trained model M, and notifies the information processing device 30 of the generated trained model M. A large amount of teacher data is required for supervised learning about CNN. Therefore, in the present embodiment, a teacher image B (for example, an image including a vehicle stopped around a railroad crossing somewhere) prepared in advance is prepared. On the other hand, the information processing device 30 notifies the server device 40 of the site image A of the railroad crossing captured by the camera 10.

The server device 40 generates a learning image by superimposing the teacher image B on the site image A. Here, a new learning image can be acquired by changing the coordinate position of superimposing the teacher image B on the site image A. Further, a new learning image can be acquired by enlarging / reducing the teacher image B and superimposing it on the site image A. Therefore, it becomes easy to acquire a learning image, which is a problem when deep learning is performed.

At this time, the server device 40 is configured to transparently (semi-transparently) superimpose the teacher image B on the site image A. By transparently superimposing and generating a composite image, it becomes difficult to recognize the object in the composite image, and it becomes difficult to detect the object. By repeating such supervised learning, the trained model M becomes a robust model with high detection accuracy. Since the actual railroad crossing site image A includes a nighttime image, a rainy weather image, etc., it is desirable to output an appropriate abnormality score even in such a situation, so that it is a composite image that makes it difficult to detect an object. Deep learning is performed using C.

When detecting an abnormality at a railroad crossing, the information processing apparatus 30 inputs an input image captured by the camera 10 to the trained model M. As a result, the anomaly probability output from the trained model M is weighted by multiplying it by coefficients such as parameters of environmental factors (weather, etc.) and parameters of accident rate at railroad crossings, and the anomaly normalized to a value of 0 to 100. Calculate the score. Then, the calculated abnormality score is notified to the monitoring center 100.

When the abnormality score of the railroad crossing is equal to or higher than a predetermined value (for example, 70), the monitoring center device 60 is placed in the driver's seat of the train 70 located toward the railroad crossing and within a certain distance (for example, 300 m) from the railroad crossing. The abnormal score is notified to the on-board display device, and the abnormal score is displayed on the display device of the train. This allows the train driver to detect the possibility of a railroad crossing abnormality from the abnormality score.

Specifically, as shown in FIG. 1, when the trained model is generated, the information processing apparatus 30 transmits the site image A captured by the camera 10 to the server apparatus 40 (S1). The server device 40 transparently superimposes the teacher image B stored in the storage unit on the site image A to generate the composite image C. This composite image C becomes a learning image. Then, the composite image C and the correct answer data (abnormal probability) R are given to the CNN as teacher data, and supervised learning is performed by backpropagation or the like. In this way, a large number of composite images C are generated, and the CNN is subjected to supervised learning based on the appropriate composite image (learning image) and the correct answer data R (S2), and the trained model M is created. Notify the information processing apparatus 30 (S3). Since backpropagation is a well-known technique for deep learning, its description is omitted here.

Here, when the above-mentioned composite image C is an image showing a situation that cannot occur in reality, the composite image C is not appropriate and is not used as a learning image, and the composite image C is appropriate. For example, it is used as a learning image. For example, when the image of the automobile included in the composite image C shows a situation of floating in the air, the composite image C is not used as a learning image. Whether or not the composite image C shows an appropriate situation can be determined by the existing image processing technique. For example, the tires of an automobile may be detected by template matching, the area of the road may be specified, and it may be configured to determine whether or not the tires are in contact with the road. It should be noted that the composite image C can be input and the determination can be made by using a new trained model that outputs the probability of validity.

The correct answer data R, that is, the probability of being abnormal can be given by the person in charge of the supervised learning. Further, it is also possible to detect whether or not an abnormal state has occurred in the field image A by an image processing technique, and automatically specify the accurate data R based on the abnormal state of the field image A. For example, the situation where a car in a railroad crossing is burning is an obvious abnormal state, and this situation can be detected by known image processing. Therefore, it is possible to automatically calculate the probability of abnormality "0.9".

When performing an abnormality determination, the information processing apparatus 30 acquires an image from the camera 10 at regular time intervals (S4), inputs this image as an input image to the trained model M, and outputs the image from the trained model M. An abnormality score is calculated based on the abnormality probability (S5), and the abnormality score is notified to the monitoring center device 60 (S6).

When the abnormality score notified from the information processing apparatus 30 is equal to or higher than a predetermined value, the monitoring center device 60 notifies the display device of the train 70 of the abnormality score (S7), and displays the abnormality score on the display device. As a result, the driver of the train 70 can detect an abnormal state at a railroad crossing.

<Configuration of information processing device 30>
Next, the configuration of the information processing apparatus 30 shown in FIG. 1 will be described. FIG. 2 is a functional block diagram showing the configuration of the information processing apparatus 30 shown in FIG. As shown in FIG. 2, the information processing apparatus 30 is connected to the camera 10 and has a storage unit 31, a control unit 32, and a communication I / F unit 33.

The camera 10 is an image pickup device including, for example, a CCD (Charge Coupled Device), and captures an image around a railroad crossing. The communication I / F unit 33 is an interface unit for wirelessly communicating with the server device 40 or the monitoring center device 50. The storage unit 31 is a storage device such as a hard disk device or a non-volatile memory, and stores the learned model M.

Here, the layer structure of the trained model M will be described. The trained model M shown in FIG. 3 is a diagram showing an example of the layer structure of the trained model, and here, a convolution layer (Convolution) 51, a convolution layer (Convolution) 52, and an average pooling layer (Average Pooling) 53. It has a convolution layer (Convolution) 54, an average pooling layer (Average Pooling) 55, a fully connected layer (Fully Connect) 56, a fully connected layer (Fully Connect) 57, and an output layer (Softmax) 58.

The convolution layers 51, 52, 54 convolve a filter into a nearby node in the previous layer to generate a feature map in order to extract local features. The average pooling layers 53 and 55 further reduce the feature map output from the convolution layer, which is the previous layer, into a new feature map in order to summarize the local features. As described above, the hidden layer of CNN is formed by the convolution layer and the average pooling layer. The fully connected layers 56 and 57 combine the feature map from which the feature portion is extracted into one node, and output the value converted by a predetermined activation function. A well-known technique such as ReLU (Rectified Linear Unit) can be used for this activation function. The output layer 58 converts the output (characteristic variable) from the fully connected layer 57 into a probability using a softmax function, and outputs the probability of being correctly classified. It is also possible to add a dropout layer to avoid overfitting. Since the basic structure of CNN is a known technique, detailed description thereof will be omitted here.

The control unit 32 is a control unit that controls the entire information processing apparatus 30, and includes an image reception unit 32a, a site image notification unit 32b, an abnormality score calculation unit 32c, and a notification unit 32d. Actually, by loading these programs into the CPU (Central Processing Unit) and executing them, the processes corresponding to the image receiving unit 32a, the site image notification unit 32b, the abnormality score calculation unit 32c, and the notification unit 32d are executed. Will let you.

The image receiving unit 32a is a processing unit that receives an image transmitted from the camera 10. When a moving image (video) compressed by MPEG or the like is received from the camera 10, an image is cut out from the moving image at regular time intervals and temporarily stored in the storage unit 31 as a field image A. The site image notification unit 32b is a processing unit that notifies the server device 40 of the site image A.

The abnormality score calculation unit 32c is a processing unit that calculates an abnormality score using the trained model M, and the notification unit 32d is a processing unit that notifies the monitoring center device 60 of the abnormality score calculated by the abnormality score calculation unit 32c. Is. Specifically, as shown in FIG. 4, the input image E captured by the camera 10 is input to the trained model M, and the abnormality probability (0.00 to 1.00) output from the trained model M is acquired. do. This abnormality probability is weighted by parameters such as the weather and the past accident rate at railroad crossings, and this value is normalized to 0 to 100 to calculate the abnormality score.

<Configuration of server device 40>
Next, the configuration of the server device 40 shown in FIG. 1 will be described. FIG. 5 is a functional block diagram showing the configuration of the server device 40 shown in FIG. FIG. 6 is a diagram showing an outline of generation of the trained model M by the server device 40 shown in FIG. 1.

As shown in FIG. 5, the server device 40 includes an input unit 41, a display unit 42, a communication I / F unit 43, a storage unit 44, and a control unit 45. The input unit 41 is an input device such as a keyboard and a mouse, and the display unit 42 is a display device such as a liquid crystal panel and a display device. The communication I / F unit 43 is an interface unit for wirelessly communicating with the information processing device 30.

The storage unit 44 is a storage device such as a hard disk device or a non-volatile memory, and stores a teacher's image B, a composite image C, a CNND, and a trained model M. The teacher image B is an image prepared in advance for the CNND to perform supervised learning, and is, for example, an image including a vehicle stopped around a railroad crossing somewhere.

The composite image C is an image in which the teacher image B is transparently (semi-transparently) superimposed on the site image A and composited. CNN data D is a multi-layered convolutional model for creating a trained model M. The trained model M is a model obtained as a result of learning with a stop by deep learning using a composite image C and correct answer data R (abnormal probability).

The control unit 45 is a control unit that controls the entire server device 40, and includes a field image receiving unit 45a, a composite image generation unit 45b, a learning processing unit 45c, and a learned model transmission unit 45d. Actually, by loading and executing the trained model generation program including these in the CPU, it corresponds to the on-site image receiving unit 45a, the composite image generation unit 45b, the learning processing unit 45c, and the trained model transmitting unit 45d, respectively. The trained model generation process including the process will be executed.

The on-site image receiving unit 45a is a processing unit that receives the on-site image A of the railroad crossing from the information processing device 30. As shown in FIG. 6, the composite image generation unit 45b is a processing unit that transparently (semi-transparently) superimposes the teacher image B on the site image A and synthesizes the composite image C to generate the composite image C.

FIG. 7 is a diagram showing an example of the composite image C. In this example, the case where the image of the vehicle stopped at a certain railroad crossing is regarded as the teacher image B is shown. As shown in FIG. 7, when the composite image C is generated, the teacher image B is made translucent and composited with the field image A. It is assumed that the transmittance when compositing the images is set in advance. As a result, a large amount of learning data for performing deep learning can be generated.

By superimposing the teacher image B on the site image A transparently to generate the composite image C, it becomes difficult to recognize the object in the composite image C, and it becomes difficult to detect the object. By repeating such supervised learning, the trained model M becomes a robust model with high detection accuracy. Since the actual railroad crossing site image A includes a nighttime image, a rainy weather image, etc., it is desirable to output an appropriate abnormality score even in such a situation, so that it is a composite image that makes it difficult to detect an object. Deep learning is performed using C.

Here, as described above, when the composite image C is an image showing a situation that cannot occur in reality, the composite image C is not appropriate and is not used as a learning image, and the composite image C is appropriate. If so, use it as a learning image. For example, when the image of the automobile included in the composite image C shows a situation of floating in the air, the composite image C is not used as a learning image. Whether or not the composite image C shows an appropriate situation can be determined by the existing image processing technique. For example, the tires of an automobile may be detected by template matching, the area of the road may be specified, and it may be configured to determine whether or not the tires are in contact with the road. It should be noted that the composite image C can be input and the determination can be made by using a new trained model that outputs the probability of validity.

The learning processing unit 45c is a processing unit that performs deep learning supervised learning using the composite image C and the correct answer data R as learning data. Specifically, as shown in FIG. 6, the composite image C is converted into CNN data D. At the same time as inputting, back propagation is performed based on the correct answer data R, and the supervised learning process for determining the weight of each path is repeated to generate the trained model M.

The correct answer data R, that is, the probability of being abnormal can be given by the person in charge of the supervised learning. Further, it is also possible to detect whether or not an abnormal state has occurred in the field image A by an image processing technique, and automatically specify the accurate data R based on the abnormal state of the field image A. For example, the situation where a car in a railroad crossing is burning is an obvious abnormal state, and this situation can be detected by known image processing. Therefore, it is possible to automatically calculate the probability of abnormality "0.9".

The trained model transmission unit 45d is a processing unit that notifies the information processing device 30 installed at the railroad crossing of the trained model M. The information processing apparatus 30 will calculate the abnormality score using the trained model M.

<Configuration of monitoring center device 60>
Next, the configuration of the monitoring center device 60 shown in FIG. 1 will be described. FIG. 4 is a functional block diagram showing the configuration of the monitoring center device 60 shown in FIG. As shown in FIG. 4, the monitoring center device 60 includes an input unit 61, a display unit 62, a communication I / F unit 63, a storage unit 64, and a control unit 65.

The input unit 61 is an input device such as a keyboard and a mouse, and the display unit 62 is a display device such as a liquid crystal panel and a display device. The communication I / F unit 63 is an interface unit for performing wireless communication with a display device mounted on the information processing device 30 and the train 70.

The storage unit 64 is a storage device such as a hard disk device or a non-volatile memory, and stores an abnormality score 64a and a determination threshold value 64b. The abnormality score 64a is log data of the abnormality score received from the information processing apparatus 30, and the determination threshold value 64b is used when determining whether or not to notify the display device of the train 70 of the abnormality score. Threshold data.

The control unit 65 is a control unit that controls the entire monitoring center device 60, and includes an abnormality score receiving unit 65a, a notification determination unit 65b, and an abnormality score notification unit 65c. Actually, by loading and executing these programs in the CPU, the processes corresponding to the abnormal score receiving unit 65a, the notification determination unit 65b, and the abnormal score notification unit 65c are executed.

The abnormality score receiving unit 65a is a processing unit that receives the abnormality score transmitted by the information processing apparatus 30. The notification determination unit 65b is a processing unit that determines whether or not to notify the display device of the train 70 of the abnormality score received by the abnormality score receiving unit 65a. For example, when the determination threshold value 64b stored in the storage unit 64 is "70" and the abnormality score is 80, the notification determination unit 65b determines that the abnormality score is notified to the display device of the train 70, and is abnormal. When the score is 60, the notification determination unit 65b determines that the abnormality score is not notified to the display device of the train 70.

The abnormality score notification unit 65c is a processing unit that notifies the display device of the train 70 of the abnormality score when it is determined by the notification determination unit 65b to notify the display device of the train 70 of the abnormality score.

<Processing procedure of server device 40>
Next, the processing procedure of the server device 40 will be described. FIG. 9 is a flowchart showing a processing procedure of the server device 90 shown in FIG. When the server device 40 receives the site image A from the information processing device 30 (step S101), the server device 40 transparently superimposes the teacher image B on the site image A to generate the composite image C (step S102).

Then, the CNND is made to perform the supervised learning process using the composite image C and the correct answer data R (step S103). When less than a predetermined number of teachered learning is performed using the same teacher image B (step S104; No), the position when the teacher image B is superimposed, the enlargement / reduction ratio of the teacher image B, and the like are determined. The synthesis condition is changed (step S105), the process proceeds to step S102, and the same processing is performed.

On the other hand, when a predetermined number or more of supervised learning is performed using the same teacher image B (step S104; Yes), the process proceeds to step S101 to acquire a new field image A, and the same is true. Repeat the process.

Then, when the predetermined end condition is satisfied, the trained model M is stored in the storage unit 44 (step S107), and the trained model M is transmitted to the information processing apparatus 30 (step S108). Ends the processing of.

<Processing procedure of information processing device 30>
Next, the processing procedure when the abnormality determination is performed by the information processing apparatus 30 will be described. FIG. 10 is a flowchart showing a processing procedure when an abnormality determination is performed by the information processing apparatus 30 shown in FIG. As shown in the figure, when the information processing apparatus 30 receives the input image E (step S201), the information processing apparatus 30 inputs the input image E into the trained model M and has an abnormality probability (0.00 to 1.00). (Step S202).

After that, parameters such as weather and accident rate at railroad crossings are weighted to this abnormality probability, normalized to a value of 0 to 100, an abnormality score is calculated (step S203), and the calculated abnormality score is used in the monitoring center device 60. Transmit (step S204). The series of processes is repeated until a predetermined end condition is satisfied (step S205; No), and when the predetermined end condition is satisfied (step S205; Yes), the above-mentioned one lotus process is terminated.

<Relationship with hardware>
Next, the correspondence between the server device 40 according to the present embodiment and the main hardware configuration of the computer will be described. FIG. 11 is a diagram showing an example of a hardware configuration.

A general computer has a configuration in which a CPU 91, a ROM (Read Only Memory) 92, a RAM 93, a non-volatile memory 94, and the like are connected by a bus 95. A hard disk device may be provided instead of the non-volatile memory 94. For convenience of explanation, only the basic hardware configuration is shown.

Here, the ROM 92 or the non-volatile memory 94 stores a program or the like necessary for starting the operating system (Operating System; hereinafter, simply referred to as “OS”), and the CPU 91 is the ROM 92 or the non-volatile memory when the power is turned on. The OS program is read and executed from the non-volatile memory 94.

On the other hand, various application programs executed on the OS are stored in the non-volatile memory 94, and the CPU 91 executes the application program while using the RAM 93 as the main memory, so that the process corresponding to the application is executed. To.

The trained model generation program of the server device 40 of the present embodiment is also stored in the non-volatile memory 94 or the like like the other application programs, and the CPU 91 loads and executes the trained model generation program. become. In the case of the server device 40, the trained model generation program including the routine corresponding to the field image receiving unit 45a, the composite image generation unit 45b, the learning processing unit 45c, and the trained model transmission unit 45d shown in FIG. 5 is non-volatile. It is stored in the memory 94 or the like. When the trained model generation program is loaded and executed by the CPU 91, a trained model generation process corresponding to the field image receiving unit 45a, the composite image generation unit 45b, the learning processing unit 45c, and the trained model transmitting unit 45d is generated. .. The teacher image B, CNND, and the like are stored in the non-volatile memory 94 in advance.

As described above, in the present embodiment, the server device 40 transparently superimposes the field image A and the teacher image B to generate the composite image C, and based on the composite image C and the correct answer data R. The trained model M is generated and notified to the information processing apparatus 30. The information processing device 30 generates an abnormality score of the input image E using the trained model M and notifies the monitoring center device 60, and the monitoring center device 60 has an abnormality score equal to or higher than a predetermined determination threshold value. In this case, the display device of the train 70 is configured to notify the abnormality score, so that the driver of the train 70 can grasp the abnormality at the railroad crossing. At this time, it becomes possible to efficiently generate a learning image required for deep learning.

In the above embodiment, the case where the image including the automobile is used as the image B for the teacher has been described, but it is applied to the case where the image including various moving objects such as a person, a bicycle, and a motorcycle is used as the image B for the teacher. You can also do it. Further, although there is no moving object around the railroad crossing, an image in which an abnormality such as a fire has occurred around the railroad crossing can be used as the teacher image B.

Further, in the above embodiment, the case where the abnormal score is displayed on the display device of the train is shown, but the present invention is not limited to this, and the present invention may be configured to display the image of the railroad crossing together with the abnormal score. ..

Further, in the above embodiment, the case where the server device 40 performs supervised learning by deep learning is shown, but the present invention is not limited to this, and supervised learning is performed by cloud or edge computing of cloud. It can also be applied in some cases.

Further, in the above embodiment, the case where the present invention is applied to a train is shown, but the present invention is not limited to this, and the present invention may be applied to various railway vehicles traveling on a railroad track or a track similar thereto. can.

In addition, each configuration shown in each of the above embodiments is a schematic function, and does not necessarily have to be physically shown. That is, the form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / integrated in any unit according to various loads and usage conditions. Can be configured.

The abnormality notification system, the server device, the trained model generation program, and the abnormality notification method according to the present invention are suitable for notifying the driver's seat of a railroad crossing of an abnormal state of a railroad crossing and supporting safe driving by a driver.

A Site image B Teacher image C Composite image D CNN data E Input image M Trained model R Correct answer data 10 Camera 30 Information processing device 31 Storage unit 32 Control unit 32a Image reception unit 32b Site image notification unit 32c Abnormal score calculation unit 32d Notification unit 33 Communication I / F unit 40 Server device 41 Input unit 42 Display unit 43 Communication I / F unit 44 Storage unit 45 Control unit 45a Site image reception unit 45b Composite image generation unit 45c Learning processing unit 45d Learned model transmission unit 60 Monitoring center device 61 Input unit 62 Display unit 63 Communication I / F unit 64 Storage unit 64a Abnormal score 64b Judgment threshold 65 Control unit 65a Abnormal score reception unit 65b Notification judgment unit 65c Abnormal score notification unit 70 Train

The present invention determines the abnormality of the level crossing, to the abnormality notification system及beauty abnormality notification method capable of quickly notifying an abnormality with respect to a train approaching the the tread off.

The present invention was made to solve the above problems, an abnormal state of crossing notifies the driver's seat of the train, the abnormality notification system及beauty capable of supporting safe driving by the driver The purpose is to provide an abnormality notification method.

Hereinafter, with reference to the accompanying drawings, a preferred embodiment of the fault notification system及beauty abnormality notifying method according to the present invention will be described in detail.

Abnormality notification system及beauty abnormality notifying method according to the present invention, an abnormal state of crossing notifies the driver's seat of the train, is suitable for the case to support safe driving by the driver.

Claims (7)

An information processing device arranged in or near a monitoring area of a railroad track through which a railroad vehicle traveling on a railroad track or a similar track passes, a server device capable of communicating with the information processing device, and monitoring for monitoring the railroad vehicle. An abnormality notification system having a center device, detecting an abnormality in the monitoring area, and notifying the railway vehicle.
The information processing device is
An imaging means for capturing an image of the site around the railroad crossing,
It has a site image notification means for notifying the server device of the site image captured by the image pickup means.
The server device is
A storage means for storing teacher information including a predetermined teacher image,
A site image acquisition means for acquiring the site image from the information processing device,
A trained model generation means for generating a trained model based on the teacher image stored in the storage means and the field image, and a trained model generation means.
It has a trained model notification means for notifying the information processing apparatus of the trained model generated by the trained model generation means.
The information processing device is
An abnormality notification system characterized in that an abnormality determination is made using a learned model received from the server device and the abnormality determination result is notified to the monitoring center device. The trained model generation means is
A composite image generation means for generating a composite image by synthesizing a teacher's image stored in the storage means with the field image,
The first aspect of the present invention is to have a learning means for generating a trained model by repeating supervised learning by applying a composite image and an abnormality score generated by the composite image generation means to a predetermined multi-layer neural network. Described anomaly notification system. The composite image generation means is
The abnormality notification system according to claim 2, wherein the teacher image stored in the storage means is superposed on the site image in a translucent state to generate a composite image. The information processing device is
When an abnormality is performed at a railroad crossing, an input image captured by the imaging means is input to the trained model, and an abnormality determination result is calculated based on the risk probability output from the trained model. The abnormality notification system according to any one of claims 1 to 3. A server device capable of communicating with an information processing device arranged in or near a railroad crossing monitoring area through which a railroad vehicle traveling on a railroad track or a similar track passes.
A storage means for storing teacher information including a predetermined teacher image,
An on-site image acquisition means for acquiring an on-site image captured by the information processing apparatus,
A trained model generation means for generating a trained model based on the teacher image stored in the storage means and the field image, and a trained model generation means.
A server device including a trained model notification means for notifying the information processing apparatus of a trained model generated by the trained model generation means. A trained model generation program executed by a server device capable of communicating with an information processing device arranged in or near a railroad crossing monitoring area through which a railroad vehicle traveling on a railroad track or a similar track passes.
A storage procedure for storing teacher information including a predetermined teacher image in a storage unit, and
The on-site image acquisition procedure for acquiring the on-site image captured by the information processing apparatus, and the on-site image acquisition procedure.
A trained model generation program characterized by causing a computer to execute a trained model generation procedure for generating a trained model based on a teacher image stored in the storage means and the field image. An information processing device arranged in or near a monitoring area of a railroad track through which a railroad vehicle traveling on a railroad track or a similar track passes, a server device capable of communicating with the information processing device, and monitoring for monitoring the railroad vehicle. It is an abnormality notification method in an abnormality notification system having a center device, detecting an abnormality in the monitoring area, and notifying the railway vehicle.
An image pickup process in which the information processing device captures an on-site image of the area around the railroad crossing.
A site image notification step in which the information processing device notifies the server device of the site image captured in the image pickup process.
A storage step in which the server device stores teacher information including a predetermined teacher image in a storage unit, and
A site image acquisition process in which the server device acquires the site image from the information processing device, and
A trained model generation step of generating a trained model based on the teacher image stored in the storage unit and the field image, and
A trained model notification process for notifying the information processing apparatus of the trained model generated by the trained model generation step, and
The information processing apparatus includes an abnormality determination result notification step of performing an abnormality determination using a learned model received from the server apparatus and notifying the monitoring center apparatus of the abnormality determination result. ..

Images