KR102302979B1 - Data processing devices and data processing methods - Google Patents

Abstract

[Task] Properly respond to data inconsistency between business systems, enabling accurate data analysis.
[Solution means] In the data processing device 1040, the storage management unit 2050 storing the business data 2120 of each of the plurality of business systems, and the plurality of business systems can be recognized as one A first process for specifying an existing object from the business data, a second process for calculating feature data indicating characteristics between the business data based on a discrepancy in the business data related to the object between each business system, the feature amount It is set as the structure which has the main control part 2010 which performs the 3rd process which specifies the content of the data cleansing process concerning the said business data based on the data, and the 4th process which performs the data cleansing process in the said content. .

Description

Data processing devices and data processing methods

TECHNICAL FIELD The present invention relates to a data processing apparatus and a data processing method, and more particularly, to a technique for enabling data analysis with high precision by appropriately responding to data inconsistency between business systems.

Recently, an activity of analyzing data held by a business system and utilizing the results to reduce business costs or improve services is underway. On the other hand, in such data analysis, so-called data cleansing processing is required, such as a process for removing illegal and unnecessary data from a large amount of work data, and a process for converting target data into a format suitable for analysis.

As one of the prior art which implement|achieves the data cleansing process mentioned above, there exists a method of detecting a sensor abnormal value using machine learning (refer nonpatent literature 1, for example). In this method, a total of 12 types of values, such as "average value" and "standard deviation", are calculated as characteristic quantities for four representative outlier values in sensor data of automobiles, and machine learning techniques are used for these values. By applying , it is possible to detect an outlier pattern.

On the other hand, from a viewpoint of a data cleansing process, the subject for making the data of a some work system into an analysis object exists. Therefore, as one of the prior art for data processing in consideration of such problems, a first business device for executing a first business, a second business device for executing a second business, the first business device and the second business device A data holiday system having a business holiday device for managing a business situation by a business device, and a service providing device for requesting service provision to the first business device and the second business device through the business holiday device , the business continuity device includes a business status management table in which business statuses for each job successively notified from the first business device and the second business device are registered, types of service requests from the service providing device, and the first business device and a processing correspondence table defining whether a service according to the service request can be provided for each combination of the business conditions that the second business device can take, and which of the first business device and the second business device is the service a service management table defining whether a service according to a request is provided; a business process management table defining a service affected by the business situation for each business situation that can be taken by the business by the first business device and the second business device; means for receiving a service request from a service providing device; means for specifying, from the business process management table, a task affecting the provision of a service according to the service request; means for acquiring the current business status of the given business; means for judging whether or not the service according to the service request can be provided in the current business status based on the processing correspondence table; and the provision of the service is possible means for specifying whether the business device that provides the service is the first business device or the second business device based on the service management table, and the business device that provides the service means for transmitting a request for execution of the service; , means for receiving a response to a request for execution of the service from a business device that provides the service, and means for notifying the service providing device of the response (refer to Patent Document 1) etc. have been proposed.

In the prior art described above, by using a processing correspondence table that defines whether or not a service according to a service request can be provided for each combination of business conditions, it is possible to provide a service in consideration of the influence of the business situation between a plurality of business systems.

Japanese Patent Laid-Open No. 2013-58116

Keisuke Kurihara, Ryo Neyama, Chihiro Sannomiya, Kazunari Nawa, Sensor Outlier Detection by Machine Learning, FIT2015 (14th Information Science and Technology Forum), Volume 2 p179-182 (p. 180)

However, in any prior art, when data of a plurality of business systems are analyzed, data inconsistency between business systems cannot be detected and appropriately dealt with.

For example, the measurement data of the maintenance line management system that manages the maintenance status of the track of a railway company and the measurement data of the overhead wire management system that manages the maintenance status of the overhead wire are used as the analysis target data, and the track wear It is assumed that an analysis is conducted to investigate the correlation between and wire wear.

Even if the prior art is applied to this situation, if a discrepancy in kilometer distance is generated between the above-described measurement data for the same location, this cannot be detected and the correlation between different locations is analyzed as it is. That is, although it is originally necessary to analyze the correlation between the line wear and the wire wear in the same place, this cannot be performed, leading to a significant decrease in analysis accuracy.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object thereof is to provide a technique that enables data analysis with high precision by appropriately responding to data inconsistency between business systems.

The data processing apparatus of the present invention for solving the above problems includes a storage management unit storing business data of each of a plurality of business systems, and an object that can be uniquely recognized across the plurality of business systems as the business data. A first process specified from , a second process for calculating feature data indicating characteristics between the business data based on a shift in the business data related to the object between each business system, and the business based on the feature data It is characterized by having a main control unit that executes a third process for specifying the content of the data cleansing process related to the data, and a fourth process for performing the data cleansing process on the content.

Further, in the data processing method of the present invention, an information processing device having a storage management unit storing business data of each of a plurality of business systems sets an object that can be uniquely recognized across the plurality of business systems as the business data. A first process specified from , and a second process for calculating feature data indicating characteristics between the work data based on a shift in the work data related to the object between each work system, and based on the feature data A third process for specifying the content of the data cleansing process relating to the business data and a fourth process for performing a data cleansing process on the content are performed.

ADVANTAGE OF THE INVENTION According to this invention, data analysis with high precision becomes possible by responding appropriately to data inconsistency between business systems.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram showing an example of a network configuration including a data processing apparatus according to a first embodiment;
Fig. 2 is a diagram showing an example of the hardware configuration of the data processing apparatus of the first embodiment;
Fig. 3 is a diagram showing flow example 1 of the data processing method in the first embodiment;
Fig. 4 is a diagram showing a configuration example of a railway object definition table in the first embodiment;
Fig. 5 is a diagram showing an example of data processed by the feature amount calculation program of the first embodiment;
Fig. 6 is a diagram showing flow example 2 of the data processing method in the first embodiment;
It is a figure which shows the structural example of the cleansing process definition table of 1st Example.
Fig. 8 is a diagram showing a screen example of a cleansing processing result screen according to the first embodiment;
Fig. 9 is a diagram showing an example of log data of a cleansing processing result log in the first embodiment;
Fig. 10 is a diagram showing an example of a flow of a data processing method in the second embodiment;

--- First embodiment ---

EMBODIMENT OF THE INVENTION Hereinafter, one Embodiment of this invention is described in detail, referring drawings. 1 is a diagram showing an example of a network configuration including a data processing apparatus 1040 according to a first embodiment of the present invention.

In the present embodiment, the business data to be processed by the data processing device 1040 is obtained from a plurality of business systems. Therefore, in the network illustrated in FIG. 1 , the data processing device 1040 is configured to be communicatively connected to the business system terminal 1010 and the business system terminal 1020 via the network 1030 .

In this configuration, for example, the business system terminals 1010 and 1020 , the data processing device 1040 , and the data analysis processing terminal 1060 are computers such as personal computers and workstations, and the networks 1030 and 1050 are configured. This can be realized by configuring each with Ethernet.

These business system terminals 1010 and 1020 are terminals which accumulate|store the business data which generate|occur|produced in each business system. Of course, each of the business system terminals 1010 and 1020 may be the business system itself.

On the other hand, the data processing device 1040 acquires the business data from each of the business system terminals 1010 and 1020 mentioned above, and performs the data cleansing process required as a preprocessing of data analysis with respect to this. In addition, the data processing device 1040 can output the data that has undergone the data cleansing process to the data analysis processing terminal 1060 via the network 1050 .

The above-described data analysis processing terminal 1060 receives, as an input, data that has been cleansed of data obtained from the data processing device 1040, and performs detailed analysis (eg, correlation analysis, clustering, etc.) according to an analysis purpose specified in advance by a user or the like. ), and it becomes a terminal that outputs the analysis result data.

In addition, in this embodiment, although the form in which the business system terminals 1010 and 1020 are inputting business data into the data processing apparatus 1040 via the network 1030 was assumed, it is not limited to this. For example, a storage medium such as a portable hard disk, a USB memory, or a DVD storing business data is connected to the interface of the data processing device 1040 to execute a read operation, so that the business data is transferred offline to the data processing device ( 1040) may be adopted.

In addition, although the data processing apparatus 1040 and the data analysis processing terminal 1060 were comprised as different terminals in this embodiment, you may configure as the same terminal.

In addition, the hardware configuration of the above-described data processing apparatus 1040 is as follows. Fig. 2 is a diagram showing an example of the hardware configuration of the data processing apparatus 1040 according to the first embodiment. The data processing device 1040 in the present embodiment executes a storage management unit 2050 composed of an appropriate storage element such as an SSD (Solid State Drive) or a hard disk drive, and various programs held in the storage management unit 2050 . and a main control unit 2010 such as a CPU that performs various judgment, calculation and control processing while performing integrated control of the device itself, and is implemented with a keyboard or mouse that receives input such as an instruction to start or stop program execution from an administrator ( An input unit 2020 to be inputted, an output unit 2030 such as a display for displaying processing data, etc., connected to the networks 1030 and 1050, and other devices (eg, business system terminals 1010 and 1020) and data analysis A communication processing unit 2040 such as a network interface in charge of communication processing with the processing terminal 1060 is provided. In addition, each of these components is connected to each other in a communication unit 2060 mounted by a bus.

In addition, in the storage management unit 2050, various programs (feature amount calculation program 2070, data classification program 2080, data cleansing processing program 2090) for implementing functions necessary as the data processing device 1040 are provided. In addition, at least the railway object definition table 2100, the cleansing process definition table 2110, and the work data 2120 are stored. Specific examples of the railroad object definition table 2100 , the cleansing process definition table 2110 , and the business data 2120 described above will be described later.

Hereinafter, the actual procedure of the data processing method in this embodiment is demonstrated based on drawing. Various operations corresponding to the data processing method described below are realized by a program executed by the data processing apparatus 1040 . Then, this program is composed of codes for performing various operations described below.

Fig. 3 is a diagram showing flow example 1 of the data processing method in the first embodiment. Specifically, it is a flowchart showing a processing example of the feature amount calculation program 2070 in the data processing device 1040 . It is assumed that this feature-quantity calculation program 2070 is executed, for example, in accordance with an instruction from a data analyst, or is executed when new business data is acquired. In addition, it is assumed that the data processing apparatus 1040 acquires each business data in advance from the business system terminals 1010, 1020, etc., and holds it as the business data 2120 in the storage management part 2050.

In this case, first, the feature amount calculation program 2070 of the data processing device 1040 is an object that can be uniquely recognized across a plurality of business systems from the business data 2120 stored in the storage management unit 2050 . As such, a railway object is extracted (step 3010).

In this extraction, the feature amount calculation program 2070 reads, for example, each column name of the business data 2120 and the value of the column, and among them, the train number, etc. A corresponding one is specified as information about an object that can be uniquely recognized across a plurality of business systems, in this case a railroad object. Of course, when a plurality of railway objects are included in the business data 2120 , the feature amount calculation program 2070 extracts all railway objects from the business data 2120 .

Next, the feature amount calculation program 2070 reads the railway object definition table 2100 of the storage management unit 2050 (step 3020). Fig. 4 shows a configuration example of the railway object definition table 2100 in the first embodiment.

This railroad object definition table 2100 is a list of records comprising an object name 4010 , movement availability 4020 , and movement attributes 4030 .

Among them, the object name 4010 is a value indicating an object (ie, object) such as a train, station, passenger, track, overhead line, and signal in the railway system.

In addition, the movement permission 4020 is a value indicating whether the object indicated by the object name 4010 can be moved. becomes this

In addition, the movement attribute 4030 consists of each value of "position determination standard and characteristic calculation method", "speed/direction determination standard and characteristic calculation method", and "route rule and characteristic calculation method".

Among them, the position determination criterion is a value indicating a data item name or unit for calculating the position of the object, such as a kilometer distance, latitude and longitude. In addition, the speed/direction determination criterion is a value indicating a data item name and unit for calculating the moving speed and moving direction of the object, such as diagram data and time-series trend of latitude and longitude.

In addition, the route rule determines the movement route of the object, such as, for example, an arrangement in which the train's starting station, stopping station, and ending station are arranged, or business trip history data corresponding to the entry history data in the traffic IC card history data. It is a value indicating the rule for doing.

Note that the feature calculation method is a method or formula for calculating a feature amount from the work data 2120, a value indicating a threshold value for determining that there is a discrepancy in the work data between work systems, and the like.

In addition, as a value of the movement attribute 4030, for one object, the above-mentioned "position determination standard and feature calculation method", "speed/direction determination standard and feature calculation method", and "path rule and feature calculation method" Among the values of , at least one should just be set.

For example, when the object is "station", the station does not move, so "position determination standard and feature calculation method" must be set as the movement attribute 4030, but "speed/direction determination standard and feature calculation method" ' and "path rule and feature calculation method" do not need to be set.

Next, the feature amount calculation program 2070 starts the following processing one by one for the railroad objects extracted in the above-mentioned step 3010 (step 3030), and if the processing is performed on all the objects (step 3030: No), Terminate processing.

On the other hand, in the above-described step 3030, when it is determined that the next object to be processed exists (step 3030: Yes), the feature amount calculation program 2070 determines whether the object can be moved (step 3030). 3040).

The feature amount calculation program 2070 in the step 3040 refers to the value defined in the object movement permission 4020 column in the railway object definition table 2100, and determines whether the value is "probable", that is, it can be moved. will do

As a result of the above determination, if the object cannot be moved (step 3040: No), the feature amount calculation program 2070 determines that the position information of the object indicated by the work data 2120 related to the object is transferred between work systems. is the same, that is, whether the same object is recognized as being in the same place across business systems, and if the position information of the object does not indicate the same place between business systems, position information between business systems as feature data is calculated as a displacement of the location (step 3050).

Here, the above-described calculation method of the displacement of the location is determined with reference to the contents of "position determination criteria and characteristic calculation method" in the railway object definition table 2100 . For example, if the object is "station", the feature amount calculation program 2070 converts the data unit into "the distance in kilometres, latitude and longitude" regarding the station as indicated by the record in the railway object definition table 2100. . In addition, in the railway object definition table 2100, if the "position determination standard and characteristic calculation method" is not set for the object, the feature amount calculation program 2070 skips this step.

On the other hand, when it is determined in the above-described step 3040 that the object is a movable object (step 3040: Yes), the feature amount calculation program 2070 determines whether the same object exists in the same place at the same time across the work system. is determined (step 3060).

Here, the above-described method for calculating the displacement of the movable object is the same as the above-described step 3050, and it is determined with reference to the contents of "position determination criteria and characteristic calculation method" in the railway object definition table 2100. . For example, if the object is "train", the feature amount calculation program 2070, as shown in the record in the railroad object definition table 2100, "track circuit number for the train is a data unit with a center kilometer distance is converted", and when the difference in kilometer distance between the business systems is "7 km or more", it is determined that there is a location shift, and the kilometer distance difference is used as the feature data.

As a result of the above-mentioned determination, if the same object does not exist in the same place at the same time (step 3060: No), the feature amount calculation program 2070 calculates a displacement of the location as the feature data data (step 3070).

On the other hand, as a result of the above determination, if the same object is in the same place at the same time (step 3060: Yes), the feature calculation program 2070 determines whether the object is moving at the same speed and in the same direction at the same time. A determination is made (step 3080).

As a result of the above determination, if the object is not moving at the same speed and in the same direction at the same time (step 3080: No), the feature-quantity calculation program 2070 calculates the speed/direction shift as feature-quantity data. do (step 3090).

Here, the feature amount calculation program 2070 is a method for calculating the speed/direction deviation described above, referring to the contents of "Speed/direction determination standard and characteristic calculation method" in the railway object definition table 2100. make it as

For example, if the object is a "train", the feature amount calculation program 2070 calculates a vehicle speed difference as a speed difference, and if the calculated speed difference between business systems is 10 km/h or more, there is a discrepancy. It is determined that there is, and the speed difference is used as the feature data. In addition, the feature amount calculation program 2070 determines that there is a discrepancy between the business systems when the up/down line or route name is different between the business systems, and the identifier (eg, the route name is the same but only the up/down line is different when the direction is the same). “1” and, if the route name is different, “2”) is used as the feature data.

In addition, with respect to the said object, if "speed/direction determination standard and characteristic calculation method" is not set in the railway object definition table 2100, the characteristic amount calculation program 2070 skips this step.

Here, a specific example of calculating the difference in vehicle speed in this way will be described with reference to FIG. 5 . Fig. 5 is a diagram showing an example of data processed by the feature amount calculation program 2070 in the first embodiment.

Here, as the business data 2120, vehicle sensor data (consisting of train number, time, kilometer distance, and speed) accumulated by the vehicle information control system in the business system and the performance diagram accumulated by the operation management system, which is another business system, are used here. It is assumed that data (consisting of a train number, a track circuit number, and a passing time) is input to the feature amount calculation program 2070 of the data processing device 1040 .

In this case, the feature amount calculation program 2070 obtains the above-mentioned work data 2120, and extracts the train object of the train number "A1A001" as a railroad object uniquely specified throughout the work system. Of course, the feature amount calculation program 2070 includes "train number" as a column in the work data 2120, and when a value is set in the column, the work data is " It is assumed that the rules for judging that information on "train" are maintained in advance.

Since the above-described train object is a movable object, it is determined in each work data 2120 of the vehicle sensor data and the performance diagram data whether the train is in the same place at the same time, and furthermore, the train is in the same time. It is determined whether it is moving at the same speed.

Here, the feature amount calculation program 2070 extracts data including the time and speed of the train concerned from the work data 2120 of the vehicle information control system. In addition, the feature-quantity calculation program 2070 extracts the arrival time of each track circuit from the operation data 2120 of the operation management system, and determines the distance between the track circuits as the length of time between these arrival times (determined in advance). ) is divided by the division to calculate the speed for each time in the train concerned.

For example, with respect to the train whose train number is "A1A001", the arrival time to the track circuit which is the track circuit number "B" is "10:02:00", and the arrival time to the track circuit becomes the track circuit number "C". Assuming that the arrival time is "10:08:00" and the distance between the orbit circuits of the orbit circuit number "B" and the orbit circuit number "C" is "10 km", the feature-quantity calculation program 2070 is The average speed between circuits is calculated by dividing the distance “10km” by “6 minutes” which is the length of time between the times “10:08:00” and “10:02:00”, 10÷0.1 = “100km” /h", and let this be the speed of the train concerned at the time "10:08:00".

On the other hand, the vehicle sensor data indicates "110 km/h" as the speed of the train at this time "10:08:00". Therefore, comparing the speed of the train "A1A001" at the time "10:08:00" between the business systems, there is a difference of 10 km/h or more between "100 km/h" and "110 km/h" Therefore, it is determined that the speed regarding the train is different between the business systems, that is, it is not moving at the same speed, and the difference in the speed is calculated as the feature data.

Here, it returns to the description of the flow of FIG. On the other hand, if, as a result of the above-mentioned determination, the object is moving at the same speed and in the same direction at the same time (step 3080: Yes), the feature amount calculation program 2070 indicates that the object moves along the same path in the same time period. It is determined whether or not (step 3100).

As a result of the above determination, if the object is not moving on the same route in the same time period (step 3100: No), the feature-quantity calculation program 2070 calculates a deviation of the route as the feature-value data (step 3110).

Here, in calculating the deviation of the route, it is assumed that the feature amount calculation program 2070 makes a determination with reference to the contents of “Route rule and feature calculation method” in the railway object definition table 2100 . For example, if the object is a "train", if the sequence numbers and numbers of the starting station, the stopping station, and the ending station differ between business systems, it is determined that there is a discrepancy, and the station number with the different sequence is used as the feature data data.

In addition, in the railway object definition table 2100, if the "route rule and feature calculation method" is not registered with respect to the object, the feature amount calculation program 2070 skips this step.

On the other hand, as a result of the above determination, if the object is moving on the same route in the same time zone (step 3100: Yes), the feature amount calculation program 2070 returns the process to step 3030, and if there is an object to be processed next The processing of step 3040 or less is repeated, and when the processing has been performed for all objects (step 3030: No), the flow ends.

Then, the processing example of the data classification program 2080 and the data cleansing processing program 2090 in the data processing apparatus 1040 is demonstrated using the flowchart of FIG. The programs (data classification program 2080 and data cleansing processing program 2090) for executing the flow shown in Fig. 6 are executed, for example, according to an instruction from a data analyzer, or the characteristic quantity calculation program 2070 has been executed. to be executed later.

In addition, the data classification program 2080 and the data cleansing processing program 2090 receive a plurality of feature data calculated by the feature variable calculation program 2070 and object names corresponding to the feature data data as inputs.

First, the data classification program 2080 reads the cleansing process definition table 2110 of the storage management unit 2050 (step 6010).

In FIG. 7, the structural example of the cleansing process definition table 2110 in 1st Example is shown. This table is a list of records composed of an object name 7010 , a condition for a feature amount 7020 , and a processing content 7030 .

Among these, object name 7010 is set with object information such as train, station, passenger, track, overhead line, and signal. In addition, the characteristic quantity condition 7020 is, for example, "the average value of the characteristic quantity data is 20 or more and 50 or less, and the standard deviation is 10 or less", "the value of 70% or more of the characteristic quantity data is 10 or less", etc. Concordance conditions for quantity data are set. Further, the processing content 7030 is, for example, "an average value of the speed difference between the same train and the same time between the vehicle information control system and the operation management system is obtained, and from the data value of the data speed of the vehicle information control system. , subtract the average value”, “In the time between station arrival and station departure in the performance diagram of the operation management system, processing logic such as 'to do' is set.

Next, the data classification program 2080 starts the following processing one by one for a plurality of feature data input at the time of executing the present program and an object name corresponding to the feature data data (step 6020), and all If processing has been performed on the feature data and the object (step 6020: No), the processing moves to step 6050.

On the other hand, in the above-mentioned step 6020, if the following feature data and object to be processed remain (step 6020: Yes), the data classification program 2080 relates to the feature data and object, the cleansing process definition table It is determined whether there is a record matching the condition 7020 of the object name 7010 and the feature amount in (2110) (step 6030).

As a result of the above determination, if there is a matching record (step 6030: Yes), the data classification program 2080 passes the processing contents 7030 of the record to the data cleansing processing program 2090, and the data cleansing processing The data cleansing process by the program 2090 is performed with respect to the said work data (step 6040), and it returns to step 6020.

On the other hand, when there is no matching record in the above-described step 6030 (step 6030: No), the data classification program 2080 returns the process to step 6020.

On the other hand, as a result of the determination of step 6020, if processing is performed on all the feature data and objects (step 6020: Yes), the data cleansing processing program 2090 transmits the execution result of the data cleansing processing to the data analysis processing terminal ( 1060), or displayed as a screen 800 on its own output unit 2030, outputting a log file 900 of the execution result (step 6050), and ending the flow.

In FIG. 8, an example of the screen 800 which shows the execution result of a data cleansing process is shown, and the example of the log file 900 is shown respectively in FIG. In this screen 800 and log file 900, by outputting which data inconsistency was detected with respect to the data at which position in the file of which work data 2120, and how the data cleansing process was performed for it, the user can check the execution result.

--- Second embodiment ---

In the present embodiment, when the data classification program 2080 classifies data, a method for classifying by applying a technique of machine learning or deep learning will be described with reference to FIG. 10 . Here, it is assumed that the flow of the data classification program 2080 and the data cleansing processing program 2090 shown in FIG. 6 of the above-mentioned 1st Embodiment is modified and shown.

First, the data classification program 2080 reads the cleansing process definition table 2110 (step 10010). In the cleansing process definition table 2110 of the present embodiment, it is assumed that the identifier of the classification output from the machine learning program is registered in the column of the condition 7020 of the feature quantity. For example, when the identifier "X" is registered in the condition 7020 of the feature amount, the machine learning program says that "the average value of the feature amount data is 20 or more and 50 or less, and the standard deviation is 10 or less" When the content classified regarding the deviation between the feature data based on the quantity data corresponds and the identifier "Y" is registered, the machine learning program will display the past characteristics as "the value of 70% or more of the feature data is 10 or less" Based on the quantity data, the contents classified regarding the deviation between the characteristic quantity data correspond to each other.

Next, the data classification program 2080 starts the following processing one by one for a plurality of feature data input at the time of executing the present program and an object name corresponding to the feature data data (step 10020), and all If processing has been performed on the feature data and the object (step 10020: No), the flow advances to step 10070.

On the other hand, in the above-described step 10020, when the feature data and object to be processed remain (step 10020: Yes), the data classification program 2080 determines the deviation of the feature data in the n-dimensional feature vector. is converted to (step 10030). For example, the average value, the maximum value, the minimum value, and the standard deviation of the deviation of the feature data data are calculated and set as a four-dimensional feature variable vector.

Next, the data classification program 2080 inputs the above-described feature vector into a machine learning program or a deep learning program, and obtains a classification result as an output (step 10040). Here, you may designate a model name corresponding to an object name at the time of execution of a machine learning program or a deep learning program.

Next, the data classification program 2080 refers to the cleansing process definition table 2110 and determines whether there is a record matching the object name and the classification result (step 10050).

As a result of the above determination, if there is a matching record (step 10050: Yes), the data classification program 2080 instructs the data cleansing processing program 2090 to clean the processing contents of the record, and to the business data. The data cleansing process is executed (step 10060), and the flow returns to step 10020.

On the other hand, when there is no matching record in step 10050 described above (step 10050: No), the data classification program 2080 returns the process to step 10020.

On the other hand, if processing is performed on all the feature data and objects in the above-described step 10020 (step 10020: No), the data cleansing processing program 2090 transmits the execution result of the data cleansing processing to the data analysis processing terminal 1060 . is transmitted or displayed on the screen in its own output unit 2030, and a log file of the execution result is output (step 10070), and the flow is terminated.

As mentioned above, in each Example, the object in the field of a railway was taken as an example and demonstrated. However, it is not limited to this as a kind of object, It may apply to the processing of the business data with respect to objects in other mobility fields, such as a car, a truck, a bus, an aviation, and a ship. Since the present invention calculates the feature amount by paying attention to the movement availability and movement attribute of each object, it can be realized with the same processing flow in other mobility fields.

In addition, in the first embodiment and the second embodiment, processing examples in the case of handling business data in two business systems have been described. However, it is not limited to this, It is good also considering the business data of three or more business systems as a process object. For example, when dealing with business data of three business systems A, B, and C, you may perform feature quantity calculation and data cleansing processing for A and B, B and C, and C and A, respectively, and A and B You may perform feature-quantity calculation and data cleansing processing targeting three business data of and C. When the feature amount calculation or data cleansing processing is performed for the three business data A, B, and C, the feature calculation method for the three business data is registered in the railway object definition table, and the cleansing process definition By registering the conditions and processing contents of the feature amounts for three business data in the table, it can be realized with the same processing flow.

As mentioned above, although the best form etc. for implementing this invention were demonstrated concretely, this invention is not limited to this, Various changes are possible in the range which does not deviate from the summary.

According to this embodiment, it is possible to appropriately respond to data inconsistency between business systems, and to analyze data with good precision.

According to the description of the present specification, at least the following becomes clear. That is, in the data processing device of the present embodiment, in the second processing, when the object is an object that does not move, the position indicated by the business data relating to the object between the respective business systems. It is good also as calculating the said feature-quantity data based on the discrepancy of information.

According to this, for example, with respect to an object that does not move, such as a station in a railway system, the data cleansing process of the business data is appropriately executed, and data with high precision in response appropriately to data inconsistency between business systems. analysis becomes possible.

Moreover, in the data processing apparatus of this embodiment, in the said 2nd process, when the said object is a moving object, the business data regarding the said object between each business system shows, It is good also as calculating the said characteristic amount data based on the shift|offset|difference of the positional information in time.

According to this, for example, with respect to a moving object such as a train in the railway system, an appropriate data cleansing process based on the deviation of positional information in the business data is performed, and data inconsistency between business systems. By responding appropriately to this, data analysis with good precision becomes possible.

Moreover, in the data processing apparatus of this embodiment, in the said 2nd process, when the said object is a moving object, in each business system, the business data regarding the said object indicates, It is good also as calculating the said characteristic amount data based on the shift|offset|difference of the speed information in time.

According to this, for example, with respect to a moving object such as a train in the railway system, appropriate data cleansing processing based on the deviation of speed information in the business data is performed, and data inconsistency between business systems. By responding appropriately to this, data analysis with good precision becomes possible.

Moreover, in the data processing apparatus of this embodiment, in the said 2nd process, when the said object is a moving object, the business data regarding the said object between each business system shows, It is good also as calculating the said characteristic amount data based on the shift|offset|difference of the information of the movement direction in time.

According to this, for example, with respect to a moving object, such as a train in a railway system, an appropriate data cleansing process based on the shift|offset|difference of the information of the moving direction in the business data is performed, By appropriately responding to data inconsistencies, data analysis with high precision becomes possible.

Moreover, in the data processing apparatus of this embodiment, in the said 2nd process, when the said object is a moving object, in each business system, the business data regarding the said object indicates, It is good also as calculating the said characteristic amount data based on the shift|offset|difference of the movement path in time zone.

According to this, for example, with respect to a moving object, such as a train in a railway system, an appropriate data cleansing process based on the shift|offset|difference of the information of the movement route in the business data is performed, and the business system is separated. By appropriately responding to data inconsistencies, data analysis with high precision becomes possible.

Moreover, in the data processing apparatus of this embodiment, in the said 3rd process, the said main control part classifies the said shift|offset|difference by applying a machine learning algorithm with respect to the said feature-quantity data, and said data cleansing according to the result of the said classification. It is good also as what specifies the content of a process.

According to this, the discrepancy of the business data (related to the same object) between the business systems is efficiently classified by machine learning based on the tendency, etc., and further, the matching data cleansing process according to the classification result specific, executable.

1010, 1020: business system terminal
1030, 1050: Network
1040: data processing unit
1060: data analysis processing terminal
2010: main fisherman
2020: Input
2030: output
2040: communication processing unit
2050: Memory Management Department
2060: Ministry of Communications
2070: feature quantity calculation program
2080: Data Classification Program
2090: Data Cleansing Processing Program
2100: Railway object definition table
2110: Cleansing treatment definition table
2120: business data

Claims (8)

A storage management unit storing business data of each of the plurality of business systems;
The first process for specifying from the business data an object that can be uniquely recognized across the plurality of business systems, and the discrepancy between the business data regarding the object between the business systems When calculating the feature amount data representing a characteristic, when the object is a non-moving object, the feature amount data is calculated based on a shift in the positional information indicated by the business data on the object between the respective business systems. A main control unit that executes a second process, a third process for specifying the content of the data cleansing process related to the business data based on the feature data, and a fourth process for performing data cleansing process on the content A data processing device characterized in that it has. delete According to claim 1,
The main control unit,
In said 2nd process, when the said object is a moving object, based on the shift|offset|difference of the positional information in the same time indicated by the business data regarding the said object between each business system, the said characteristic Data processing device, characterized in that for calculating the amount data. According to claim 1,
The main control unit,
In the second process, when the object is a moving object, the feature data is calculated based on the shift in speed information at the same time indicated by the work data on the object between the respective work systems. A data processing device, characterized in that it is calculated. According to claim 1,
The main control unit,
In the said 2nd process, when the said object is a moving object, based on the shift|offset|difference of the information of the movement direction in the simultaneous time which the business data concerning the said object between each business system shows, the said feature amount Data processing device, characterized in that for calculating data. According to claim 1,
The main control unit,
In the second process, when the object is a moving object, the feature amount is based on a shift in movement path information in the same time zone indicated by the work data relating to the object between the respective work systems. Data processing device, characterized in that for calculating data. According to claim 1,
The main control unit,
The data processing apparatus according to claim 3, wherein, in the third process, the discrepancy is classified by applying a machine learning algorithm to the feature data, and the content of the data cleansing process is specified according to a result of the classification. An information processing device having a storage management unit storing business data of each of the plurality of business systems,
a first process for specifying, from the business data, an object that can be uniquely recognized across the plurality of business systems;
When calculating the feature amount data indicating the characteristics between the work data based on the shift in the work data related to the object between the respective work systems, when the object is a non-moving object, between the work systems, a second process of calculating the feature amount data based on a shift in positional information indicated by the business data relating to the object;
a third process for specifying the contents of the data cleansing process related to the business data based on the feature data;
A data processing method characterized by executing a fourth processing for performing data cleansing processing in the content.

Images