CN115827753A

CN115827753A - Public legal service data processing method and device, electronic equipment and storage medium

Info

Publication number: CN115827753A
Application number: CN202211477858.8A
Authority: CN
Inventors: 潘建忠; 李如旺
Original assignee: Guangzhou Hantele Communication Co ltd
Current assignee: Guangzhou Hantele Communication Co ltd
Priority date: 2022-11-23
Filing date: 2022-11-23
Publication date: 2023-03-21

Abstract

The embodiment of the application discloses a public legal service data processing method and device, electronic equipment and a storage medium. According to the technical scheme provided by the embodiment of the application, the method comprises the steps that original data of public legal service consultation of at least one acquisition source are acquired, wherein the acquisition source comprises a mobile big data source, a voice data source, a video data source, a webpage data source and a short message data source, and the original data comprises consultation standing book data, ticket data and consultant data; carrying out data cleaning and data conversion processing on the original data to obtain a first data set; desensitizing the first data to obtain a second data set; and integrating the second data set to obtain the fusion data of each consultant, and determining the query index of the fusion data of the corresponding consultant, so that the technical problem of low data acquisition efficiency can be solved, the data acquisition efficiency is improved, and the stability and the safety of data analysis are improved.

Description

Public legal service data processing method and device, electronic equipment and storage medium

Technical Field

The embodiment of the application relates to the technical field of data processing, in particular to a public legal service data processing method and device, electronic equipment and a storage medium.

Background

With the increasing awareness of the common law of the people, more and more people carry out public legal service consultation through different channels. With the larger and larger consulting data volume, public legal services big data analysis and big data decision-making become more and more important.

The data sources of the public legal service big data are dispersed at present, data source systems are different to form a data island, and data acquisition is lack of efficient synchronization, so that the public legal service big data analysis and big data decision cannot be realized by accurately acquiring the data in time.

Disclosure of Invention

The embodiment of the application provides a public legal service data processing method and device, electronic equipment and a storage medium, which can solve the technical problem of low data acquisition efficiency, improve the data acquisition efficiency and improve the stability and safety of data analysis.

In a first aspect, an embodiment of the present application provides a public legal service data processing method, including:

the method comprises the steps of collecting original data of public legal service consultation of at least one collection source, wherein the collection source comprises a mobile big data source, a voice data source, a video data source, a webpage data source and a short message data source, and the original data comprises consultation standing book data, ticket data and consultant data;

carrying out data cleaning and data conversion processing on the original data to obtain a first data set;

desensitizing the first data to obtain a second data set;

and integrating the second data set to obtain fusion data of each consultant, and determining a query index of the fusion data of the corresponding consultant.

Further, the consulting standing book data comprises consulting time, consulting channels, business types, consulting contents and lawyer suggestions;

the call ticket data comprises consultation starting time, consultation waiting time, consultation duration, receptionists and satisfaction degree;

the consultant data comprises name, mobile phone number, academic calendar, age, gender, province, city and specific address;

before the collecting of the raw data of the public legal service consultation of at least one collecting source, the method comprises the following steps:

in the original data in each acquisition source, the ledger data, the ticket data and the consultant data of the same consultant are correlated, and an original data record is correspondingly formed.

Further, the collecting the raw data of the public legal service consultation of at least one collecting source comprises:

collecting voice data of a consultant through voice consultation, performing semantic analysis on the voice data to obtain corresponding voice original data, and storing the voice original data in a voice data source database, wherein the voice original data comprises voice account data, voice call bill data and voice consultant data;

the method comprises the steps of collecting image-text data of a consultant through web page consultation, carrying out image analysis and character recognition on the image-text data, obtaining corresponding web page original data, and storing the web page original data in a web page data source database, wherein the web page original data comprises web page standing book data, web page ticket data and web page consultant data;

acquiring video data of a consultant through video consultation, performing picture recognition and voice recognition on the video data, acquiring corresponding video original data, and storing the corresponding video original data in a video data source database, wherein the video original data comprises video standing book data, video ticket data and video consultant data;

collecting image-text data consulted by a consultant through a short message, carrying out image analysis and character recognition on the image-text data, obtaining corresponding short message original data, and storing the short message original data in a short message data source database, wherein the short message original data comprises short message account data, short message ticket data and short message consultant data;

establishing a connection port with a mobile big data source to obtain big data original data of a mobile big database with corresponding authority, wherein the big data original data comprises big data ledger data, big data ticket data and big data consultant data;

the collecting of the raw data of the public legal service consultations of at least one collecting source comprises:

and acquiring corresponding voice original data, webpage original data, video original data, short message original data and mobile big data from the voice data source database, the webpage data source database, the video data source database, the short message data source database and the mobile big database in real time.

multithreading is carried out according to a preset acquisition control strategy, and original data of public legal service consultation of at least one acquisition source are acquired simultaneously, wherein each acquisition thread corresponds to each acquisition source;

and each thread acquires data corresponding to the original data according to the breakpoint continuous transmission strategy.

Further, after each thread performs data acquisition on corresponding original data according to the breakpoint continuous transmission policy, the method includes:

when a breakpoint appears in the acquisition process, resuming transmission after restarting;

when the restart times reach a threshold value, stopping collecting data;

or stopping collecting data when the error record of the collected original data exceeds a preset error record threshold or the error ratio exceeds a preset error ratio threshold.

Further, the performing data cleaning and data conversion processing on the original data to obtain a first data set includes:

performing data cleaning processing on the original data, and removing irregular data and non-conforming fact data to obtain first preprocessing data;

performing numerical value conversion on the first preprocessed data to obtain a first data set;

performing desensitization processing on the first data to obtain a second data set, including:

desensitizing the first data set, and performing online shielding, online deformation, character replacement or random replacement on sensitive field data in the first data set to obtain a desensitized second data set.

Further, the integrating the second data set to obtain the fusion data of each consultant, and determining the query index of the corresponding fusion data of the consultant includes:

fusing according to the consultant data in each acquisition source data in the second data set to obtain fused data corresponding to each consultant;

and establishing an index of the fusion data of each consultant so as to search and obtain the fusion data of the corresponding consultant through the index, wherein the index comprises a unique code identification.

Further, the integrating the second data set to obtain the fusion data of each consultant includes:

acquiring correspondingly matched legal provision information from an intelligent knowledge base according to the service type and the consultation content in the fusion data of each consultant;

and displaying the legal provision information in a fusion data page corresponding to the consultant for the user to view.

In a second aspect, an embodiment of the present application provides a public legal service data processing apparatus, including:

the system comprises a data acquisition unit, a data processing unit and a data processing unit, wherein the data acquisition unit is used for acquiring original data of public legal service consultation of at least one acquisition source, the acquisition source comprises a mobile big data source, a voice data source, a video data source, a webpage data source and a short message data source, and the original data comprises consultation standing book data, ticket data and consultant data;

the data preprocessing unit is used for carrying out data cleaning and data conversion processing on the original data to obtain a first data set;

the desensitization processing unit is used for performing desensitization processing on the first data to obtain a second data set;

and the data integration unit is used for integrating the second data set to obtain the fusion data of each consultant and determining the query index of the corresponding fusion data of the consultant.

the apparatus further comprises an association unit;

and the association unit is used for associating the ledger data, the ticket data and the consultant data of the same consultant in the original data in each acquisition source and correspondingly forming an original data record.

Further, the data acquisition unit is further configured to acquire voice data of a consultant through voice consultation, perform semantic analysis on the voice data, acquire corresponding voice original data, and store the voice original data in a voice data source database, where the voice original data includes voice standing book data, voice ticket data, and voice consultant data;

collecting video data of a consultant through video consultation, carrying out picture recognition and voice recognition on the video data, obtaining corresponding video original data, and storing the video original data in a video data source database, wherein the video original data comprises video standing book data, video ticket data and video consultant data;

Further, the data acquisition unit is further configured to perform multithreading according to a preset acquisition control strategy and concurrently acquire original data of public legal service consultation of at least one acquisition source, where each acquisition thread corresponds to each acquisition source;

Furthermore, the data acquisition unit is also used for resuming transmission after restarting when a breakpoint occurs in the acquisition process;

when the restarting times reach a threshold value, stopping collecting data;

Further, the data preprocessing unit is further configured to perform data cleaning processing on the original data, and remove irregular data and non-conforming fact data to obtain first preprocessed data;

Further, the data integration unit is further configured to perform fusion according to the data of the consultants in each acquisition source data in the second data set to obtain fusion data corresponding to each consultant;

Further, the device also comprises an intelligent matching unit;

the intelligent matching unit is used for acquiring correspondingly matched legal provision information from the intelligent knowledge base according to the service type and the consultation content in the fusion data of each consultant;

In a third aspect, an embodiment of the present application provides a public legal service data processing apparatus, including:

a memory and one or more processors;

the memory to store one or more programs;

when the one or more programs are executed by the one or more processors, cause the one or more processors to implement the public legal service data processing method of the first aspect.

In a fourth aspect, embodiments of the present application provide a storage medium storing computer-executable instructions for performing the public legal service data processing method of the first aspect when executed by a computer processor.

According to the method and the device, original data of public legal service consultation of at least one acquisition source are acquired, data cleaning and data conversion processing are carried out on the original data to obtain a first data set, desensitization processing is carried out on the first data set to obtain a second data set, integration processing is carried out on the second data set to obtain fusion data of each consultant, and an index of the fusion data corresponding to the consultant is determined. By adopting the technical means, the original data of the public legal service consultation of at least one acquisition source can be acquired, and the fusion data of each consultant can be obtained through processing, so that the problem of low data acquisition efficiency can be avoided, the synchronous acquisition of multiple acquisition sources is realized, and the data acquisition efficiency is improved. In addition, effective data and a unified data form are obtained by carrying out data cleaning and data conversion processing on the original data, the stability of subsequent data analysis is improved, and the data security is improved by obtaining a second data set which does not contain sensitive information of a consultant through desensitization processing.

Drawings

FIG. 1 is a flow chart of a method for processing public legal service data provided by an embodiment of the present application;

FIG. 2 is a schematic diagram of a multi-source data adaptation management provided by an embodiment of the present application;

FIG. 3 is a schematic diagram illustrating a connection between various devices provided by an embodiment of the present application;

FIG. 4 is a schematic diagram illustrating a display of a data transformation provided by an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating a display of data desensitization provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a public legal service data processing apparatus according to an embodiment of the present application;

fig. 7 is a schematic structural diagram of a public legal service data processing device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, specific embodiments of the present application will be described in detail with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some but not all of the relevant portions of the present application are shown in the drawings. Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the operations (or steps) as a sequential process, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations may be re-arranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.

With the improvement of public law awareness of people, more and more people consult public legal services through channels such as videos, voice or networks. With the larger and larger data volume of consultation, public legal services big data analysis and big data decision are more and more important. However, the existing public legal service big data has more problems, including that the data sources are dispersed, so that the acquisition degree is complex, the data acquisition systems are different, a data isolated island is easily formed, the data volume is large, so that the time delay is large, the data acquisition is lack of high-efficiency synchronization, and the data cannot be timely and accurately acquired to realize the public legal service big data analysis and big data decision.

The method, the device, the electronic equipment and the storage medium for processing the public legal service data aim to improve the efficiency of data acquisition by acquiring the original data of the public legal service consultation of at least one acquisition source and processing the original data to obtain the fusion data of each consultant when the public legal service consultation data is processed. In addition, effective data and a unified data form are obtained by carrying out data cleaning and data conversion processing on the original data, the stability of subsequent data analysis is improved, and the data security is improved by obtaining a second data set which does not contain sensitive information of a consultant through desensitization processing. Compared with a traditional public legal service data processing mode, the data sources are generally dispersed, data source systems are different, a data solitary is formed, and efficient synchronization is lacked in data acquisition, so that data cannot be timely and accurately acquired to realize public legal service big data analysis and big data decision. Based on this, the public legal service data processing method provided by the embodiment of the application is provided to solve the technical problem of low data acquisition efficiency in the prior art.

Fig. 1 is a flowchart of a public legal service data processing method provided in an embodiment of the present application, where the public legal service data processing method provided in this embodiment may be executed by a public legal service data processing device, the public legal service data processing device may be implemented by software and/or hardware, and the public legal service data processing device may be formed by two or more physical entities or may be formed by one physical entity. Generally, the public legal service data processing device may be a terminal device, such as a computer device or the like.

The following description will be given taking a computer device as an example of a subject that performs a public legal service data processing method. Referring to fig. 1, the public legal service data processing method specifically includes:

s101, collecting original data of public legal service consultation of at least one collection source, wherein the collection source comprises a mobile big data source, a voice data source, a video data source, a webpage data source and a short message data source, and the original data comprises consultation standing book data, ticket data and consultant data.

In order to realize concurrent data acquisition of multiple acquisition sources, the embodiment provides a multi-source data adapter management method. Fig. 2 is a schematic diagram of multi-source data adaptation management provided in an embodiment of the present application, and referring to fig. 2, based on a distributed system (F l i nk) technology, a read (Reader) plugin and a write (Wr iter) plugin are written at an engine end (engine end), and adapters of multiple data sources are integrated, where read data and written data form a synchronization task, and the synchronization task is translated into StreamGraph and executed at an F l i nk cluster end. The Temp l ate module loads a read (Reader) plug-in and a write (Wr iter) plug-in corresponding to a source database and a target database according to configuration information of a synchronous task, the read (Reader) plug-in realizes an I nputFormat interface, a DataStream object is obtained from the source database, the write (Wr iter) plug-in realizes an OutputFormat interface, and the target database is associated with the DataStream object. And the Temp l ate module connects a read plug-in (Reader) and a write plug-in (Wr iter) in series through a DataStream object to form an F l i nk task, and submits the F l i nk task to an F l i nk cluster end for execution.

Illustratively, the multi-source data adapter management supports various heterogeneous data sources, bidirectional acquisition of various data sources such as MySQL, orac l e, SQLServer, hi ve and Hbase can be realized, and the multi-source data adapter management is high in expansibility and flexibility. The adapted protocols include Mysq i n/og, orac l e, JDBC, FTP, HBase, enterprise WS and REST style protocols, etc.

Public legal service consultation can be understood as the activity of carrying out legal consultation by the public through a corresponding consultation channel. The consultation channel comprises a telephone, internet audio, video, internet web pages or short messages and the like. Fig. 3 is a schematic connection diagram of various devices according to an embodiment of the present application, and referring to fig. 3, at an acquisition source, an acquisition source includes a voice acquisition source, a video acquisition source, a web page acquisition source, a short message acquisition source, a mobile big data acquisition source, and the like. The voice acquisition source comprises a telephone audio acquisition source and an internet audio acquisition source, for example, the telephone audio acquisition source is a hotline telephone, and the audio information corresponding to the consultant and the telephone bill information corresponding to the telephone bill are acquired through the intelligent voice answering function of the hotline telephone; the internet audio acquisition source is a legal service network, an applet or an APP, and the internet audio information of the consultant is acquired through the legal service network, the applet or the APP. The video acquisition source comprises a telephone video acquisition source and an internet video acquisition source, VOLTE video information is acquired through the telephone video acquisition source, and internet video information of the consultant is acquired through a legal service network, a small program or an APP. The webpage acquisition source comprises a legal service network, a small program or an APP, and the image-text information of the consultant is acquired through the legal service network, the small program or the APP. The short message collection source comprises a telephone short message, and the short message information of the consultant is obtained through the telephone short message. The mobile big data acquisition source comprises a mobile big data label database or an internet database of an association organization, and the mobile big data of the corresponding counselor is acquired through the mobile big data label database of the acquired authority or the internet database of the association organization.

In one embodiment, the voice data of the consultant is collected and subjected to semantic analysis to obtain corresponding voice original data, wherein the voice original data comprises voice account data, voice call bill data and voice consultant data, and the voice original data is stored in a voice data source database. Illustratively, at a voice acquisition source end, through the access of telephone audio or internet audio, an intelligent voice robot or a corresponding seat worker answers the voice, receives voice information of a consultant, key information is automatically synchronized to an account system in a JSON (java script Object note i on, JS Object notation) format through an audio Interactive Voice Response (IVR) based on an interface mode, the key information comprises key skills such as a telephone number, a selected language, a selected service type and a selected manual service, and the consulting content is recorded and recorded to the account system by the intelligent voice robot or the seat worker, so that voice account data, voice call bill data and voice consultant data of the consultant are obtained.

It should be noted that, at the voice acquisition source, the voice account data, the voice ticket data and the voice counselor data of the same counselor are associated to form an original data record correspondingly. Or, at the voice acquisition source end, associating the voice standing book data, the voice call ticket data and the voice consultant data of each consultation, and correspondingly forming an original data record.

In one embodiment, picture recognition and voice recognition are performed on video data by collecting the video data of a consultant through video consultation, so that corresponding video original data are obtained, wherein the video original data comprise video standing book data, video call ticket data and video consultant data, and are stored in a video data source database. Illustratively, at a video acquisition source end, a video intelligent robot or a video seat worker answers through VoLTE video access, video data of a consultant is received, key information is automatically synchronized to an account system through video Interactive Voice Response (IVR) based on an interface mode in a JSON (JavaScript Object Notat, JS Object notation) format, the key information comprises a telephone number and multiple video service information, the consultation content is obtained by the video intelligent robot through video picture recognition and voice recognition, or the video seat worker obtains corresponding consultation content, records the consultation content to the account system, and obtains the video account data, the video call ticket data and the video consultant data.

It should be noted that, at the video acquisition source, the video standing book data, the video ticket data and the video advisor data of the same advisor are associated, and an original data record is correspondingly formed. Or, at the video acquisition source end, correlating the video standing book data, the video call ticket data and the video consultant data of each consultation, and correspondingly forming an original data record.

In one embodiment, at the web page acquisition source end, image-text data of a consultant consulted by a web page is acquired, image analysis and character recognition are performed on the image-text data, corresponding web page original data are acquired, the web page original data comprise web page standing book data, web page ticket data and web page consultant data, and the web page original data are stored in a web page data source database. Illustratively, at a webpage acquisition source end, an online chatting tool is accessed, the intelligent image-text robot or an online seat worker performs butt joint processing, the chatting tool is realized based on WebSockets, a user (a consultant) logs in a webpage by binding a telephone number to open the chatting tool, the chatting tool can open an interactive communication session between a browser of the user (the consultant) and a server, and image analysis and character recognition are performed on image-text information of the communication session to obtain webpage original data. The webpage original data are synchronized to the standing book system at regular time through a background interface, wherein the data comprise telephone numbers, consultation service types, consultation contents, consultation duration and the like, and therefore the webpage standing book data, the webpage ticket data and the webpage consultant data can be obtained.

It should be noted that, at the web page acquisition source end, the web page standing book data, the web page ticket data and the web page advisor data of the same advisor are associated, and an original data record is correspondingly formed. Or, at the web page acquisition source end, correlating the web page standing book data, the web page ticket data and the web page consultant data of each consultation, and correspondingly forming an original data record.

In one embodiment, at a short message acquisition source, image analysis and character recognition are performed on image-text data by acquiring image-text data consulted by a consultant through a short message, so as to obtain corresponding short message original data, wherein the short message original data comprises short message account data, short message ticket data and short message consultant data, and the short message original data is stored in a short message data source database. Illustratively, through the access of the short message chat tool, the short message chat tool is docked by the intelligent graphic robot or the online seat staff, the short message chat tool can open an interactive communication session between the mobile communication equipment (such as a mobile phone) of the user (a consultant) and the server, and image analysis and character recognition are carried out on graphic and text information of the communication session to obtain the original data of the short message. The short message original data is synchronized to the standing book system at regular time through a background interface, wherein the short message original data comprises a telephone number, a consultation service type, consultation content, consultation duration and the like, and therefore the short message standing book data, the short message ticket data and the short message consultant data can be obtained.

It should be noted that, at the short message acquisition source end, the short message account data, the short message ticket data and the short message advisor data of the same advisor are associated, and an original data record is correspondingly formed. Or, at the short message acquisition source end, the short message account data, the short message ticket data and the short message consultant data of each consultation are correlated, and an original data record is correspondingly formed.

In one embodiment, a connection port with a mobile big data source is established to obtain big data original data of a mobile big database with corresponding authority, wherein the big data original data comprises big data ledger data, big data ticket data and big data consultant data. Illustratively, at the mobile big data source end, such as a mobile big data tag database or an internet database of an association organization, public legal consultant profiling information is stored in the mobile big data tag database, including basic information of the consultant, historical consultant information, portrait of the consultant, and the like.

The voice acquisition source, the video acquisition source, the webpage acquisition source, the short message acquisition source and the mobile big data source are connected through the multi-source data adapter, and corresponding voice original data, webpage original data, video original data, short message original data and mobile big data are acquired from the voice data source database, the webpage data source database, the video data source database, the short message data source database and the mobile big database in real time, so that consultation standing book data, ticket data and consultant data of each acquisition source are acquired. The consulting ledger data includes consulting time, consulting channel, type of business, consulting content, and lawyer suggestions. The ticket data comprises consultation starting time, consultation waiting time, consultation duration, receptionists and satisfaction degree. The consultant data includes name, cell phone number, school calendar, age, gender, province, city and specific address. The multi-source data adapter is used for carrying out concurrent collection of multiple collection sources, so that high-efficiency synchronization of data collection is realized, the working efficiency of data collection is improved, the phenomenon of data isolated island is avoided, and the data of a multi-collection-source system can be compatible with each other.

In one embodiment, during acquisition, multithreading is performed through a preset acquisition control strategy to concurrently acquire the original data of the public legal service consultation of at least one acquisition source, wherein each acquisition thread corresponds to each acquisition source, and each thread performs data acquisition corresponding to the original data according to a breakpoint continuous transmission strategy. And when a breakpoint occurs in the acquisition process, restarting according to a preset acquisition control strategy, and continuing transmission after restarting. The data in the corresponding acquired data volume and the data in the acquisition source can be compared, so that the data after the acquired data volume is acquired, and continuous transmission is realized. The reason of the breakpoint can be the network reason or the breakpoint caused by other reasons, when the task fails due to the network or other reasons, the data is continuously synchronized by taking the error data record as the starting point, until the restart times reach the threshold value, the data collection is stopped, and the following data cannot be recorded into the target database. And starting real-time collection, when data in a data collection source is subjected to addition, deletion and modification operations, monitoring the changes by a synchronization task, synchronizing the changed data to a target database in real time, and finally establishing a public legal service data storage warehouse based on the obtained data. Through breakpoint continuous transmission, when the synchronous task fails, the re-running task is not needed, and only synchronous acquisition is continued from the breakpoint, so that re-running time and cluster resources are saved, and the data acquisition working efficiency is improved.

In an embodiment, a preset error record threshold and a preset error proportion threshold are set while the data is collected through the breakpoint continuous transmission, and when the error record of the collected original data exceeds the preset error record threshold or the error proportion exceeds the preset error proportion threshold, the data collection is stopped.

In one embodiment, an interface (acquisition source) which is configured and connected successfully by the multi-source data adapter is selected for acquisition, a target table is selected, field mapping is configured for the table account table, the phone list table and the consultant table, and table account data is obtained, wherein the table account data comprises a name, a mobile phone number, consultation time, a consultation channel, a service type, gender, province, city, consultation content, lawyer suggestion and the like; acquiring call ticket data, wherein the call ticket data comprises call starting time, waiting time, call time, answer lawyer, satisfaction degree and the like; consultant data is obtained, and the consultant data comprises name, mobile phone number, identity card, academic calendar, age, gender, province, city, address and the like. The method comprises the steps of configuring acquisition control options, configuring concurrent acquisition and multi-thread task number, setting an error record threshold and an error proportion threshold, starting breakpoint continuous transmission, configuring retry times and starting real-time acquisition. The breakpoint continuous transmission is mainly based on a checkpoint nt mechanism and carries out snapshot storage according to a Chandy-Lamport distributed snapshot algorithm at a certain time interval. And in the data acquisition process, judging according to conditions such as data type, numerical value must be filled, data length and the like, stopping acquisition if no wrong record occurs or the wrong proportion is greater than a configured numerical value, and not inputting the data into a target database. If the breakpoint continuous transmission is started, continuous transmission can be carried out according to the configured restart times, when a task fails due to network or other reasons, the data record which makes mistakes is taken as a starting point to continue synchronizing the data until the failure restart times are exceeded, and when the failure restart times are exceeded, the acquisition is stopped, and the data cannot be recorded into a target database. If the real-time acquisition is started, when the data in the data acquisition source is subjected to the operations of increasing, deleting and modifying (known by monitoring log data), the synchronization task monitors the changes, synchronizes the changed data to the target data source in real time, and finally establishes a public legal service data storage warehouse based on the acquired data. By using the Chandy-Lamport distributed snapshot algorithm, when the synchronization task fails, the re-running task is not needed, and only the synchronization is continued from the breakpoint, so that the re-running time and the cluster resources are saved. The real-time acquisition and continuous running are realized, the real-time acquisition of MySQL Bi n log, fi l ebeats, kafka and the like is supported, and the limitation of the concurrent number and the operation rate of the operation is supported. By setting the error recording threshold and the error ratio threshold in the data synchronization configuration information, the data synchronization task is stopped in time when an error occurs, and the waste of system resources is avoided.

In one embodiment, a multi-source data adapter supports both stream processing and batch processing through an underlying engine, on top of which a checkpoint mechanism and state mechanism, a watermark mechanism, and windows and triggers are placed. Wherein, the check point mechanism and the state mechanism are used for realizing fault-tolerant and stateful processing; a watermarking mechanism for implementing an event clock; a window and a trigger for limiting the calculation range and defining the time for presenting the result. In the aspect of realizing efficient batch processing, by introducing a backtracking method for scheduling and recovering by Microsoft Dryad, through a special memory data structure of hashing and sorting, a part of data can be overflowed from a memory to a hard disk when needed, and the time for generating a result is shortened as much as possible through an optimizer.

And S102, carrying out data cleaning and data conversion processing on the original data to obtain a first data set.

Data cleansing may be understood as the last procedure to find and correct recognizable errors in a data file, including checking data consistency, processing invalid and missing values, and the like. After the original data are collected from a plurality of collection sources, the original data are subjected to data cleaning processing, recognizable errors in data files are found and corrected, including processing invalid values, missing values and the like, irregular data and non-conforming fact data are removed, and first preprocessed data are obtained. And carrying out numerical value conversion on the first preprocessed data to obtain a first data set. And hierarchically storing the first data set subjected to the data cleaning and data conversion processing.

In one embodiment, data cleaning, value conversion and other processing are performed on data in the data acquisition process through two modes of SQL and F l i nkSQL, and recognizable errors in data files are found and corrected, wherein the recognizable errors include data consistency checking, invalid value processing, missing value processing and the like. And eliminating the irregular data and the inconformity fact data to eliminate the inconformity of the data. The dirty data generated in the data synchronization process is independently stored and recorded, simple reason analysis is supported, and the data quality is improved. Fig. 4 is a display schematic diagram of data conversion according to an embodiment of the present application, and referring to fig. 4, a corresponding source field to be converted is obtained from a conversion source, and a condition value of the source field is subjected to numerical value conversion, so as to obtain a conversion value corresponding to the conversion. Illustratively, the condition value "male" of the consultant is numerically converted into the conversion value "F", and the condition value "female" of the consultant is numerically converted into the conversion value "M", so that data storage and data analysis processing can be conveniently performed through a simpler conversion value during subsequent data processing, the data storage space is increased, and the work efficiency of data analysis processing is improved.

S103, desensitizing the first data to obtain a second data set.

Desensitization may be understood as masking sensitive information. Desensitization processing is carried out on the first data through a desensitization algorithm, and on-line shielding, deformation, character replacement, random replacement and the like are carried out on sensitive data in the first data set, so that desensitization results can be suitable for development, test and analysis scenes. And carrying out security desensitization treatment on a first data set which is stored in a layered manner and has sensitive data, wherein the desensitization data comprises a telephone number, an identity card number, a name, a home address and the like of a consultant, and carrying out online shielding, deformation, character replacement, random replacement and the like on the sensitive data in the first data set to obtain a second data set. Fig. 5 is a schematic diagram of data desensitization provided in an embodiment of the present application, in which data including sensitive information corresponding to an advisor is obtained from a desensitization source (a first data set), field desensitization is performed on the data of the advisor, and desensitization control is performed to obtain the desensitized data, for example, referring to fig. 5, after name information desensitization, a user view is shown as "yellow", and after identity card number information desensitization, a user view is shown as "43283 × 000000", so that masking of the sensitive information of the advisor is achieved, privacy of the advisor is protected, and security of data processing is improved.

S104, integrating the second data set to obtain fusion data of each consultant, and determining a query index of the fusion data of the corresponding consultant.

And integrating the data of the same consultant collected by each collection source according to the data of the consultant in each collection source data in the second data set to obtain fusion data corresponding to each consultant. For example, the same consultant is determined through the telephone number, the telephone number of the consultant in the original data acquired by the voice acquisition source, the video acquisition source, the webpage acquisition source, the short message acquisition source and the mobile big data source is regarded as the same consultant, and the data acquired by the same consultant from the voice acquisition source, the video acquisition source, the webpage acquisition source, the short message acquisition source and the mobile big data source are fused to obtain the fused data of different acquisition sources of the same consultant. And establishing an index of the fusion data of each consultant so as to search through the corresponding index to obtain the fusion data of the corresponding consultant. Wherein the index can be identified by a unique code, a one-dimensional code, a two-dimensional code or a telephone number.

And according to the service type and the consultation content in the fusion data of each consultant, correspondingly matching legal provision information is obtained from the intelligent knowledge base, and the legal provision information is displayed in the fusion data page of the corresponding consultant for the user to check. Legal provision information is matched with the intelligent knowledge base, so that users (lawyers or seat workers) can refer to the intelligently matched provision information, the intelligent degree of the system is improved, and the work efficiency of making consultation responses is improved.

In one embodiment, an intelligent knowledge base and an intelligent agent assistant are provided, wherein the intelligent knowledge base comprises various legal knowledge, clauses and the like and supports searching and fast searching to assist an agent attorney to process consultation problems. The intelligent agent assistant can automatically process the automatic reply to the consultant under the condition that the staff is not on line.

The public legal service data sources are legal network platforms, legal network apps, hot line systems and the like, data acquisition is carried out after the data source adapters are successfully configured and connected, the data acquisition supports automatic acquisition and multi-task acquisition, functions of increment synchronization, multi-channel control, dirty data management and error management in the data synchronization process are realized based on fragmentation and accumulator characteristics of a big data technology, intermittent continuous transmission, continuous running of streaming data, dirty data management and error control are realized based on a checkpoint nt mechanism, and real-time and efficient acquisition of public legal service consultant data, ticket data, consultant account data and the like is guaranteed. The data in the data synchronization process are filtered, converted, processed and the like in two modes of the database SQL and the F l i nkSQL, and the data quality is effectively improved.

The method comprises the steps of acquiring original data of public legal service consultations of at least one acquisition source, carrying out data cleaning and data conversion processing on the original data to obtain a first data set, carrying out desensitization processing on the first data set to obtain a second data set, carrying out integration processing on the second data set to obtain fusion data of each consultant, and determining an index of the fusion data corresponding to the consultant. By adopting the technical means, the synchronous acquisition of multiple acquisition sources can be realized by acquiring the original data of the public legal service consultation of at least one acquisition source and processing the original data to obtain the fusion data of each consultant, and the data acquisition efficiency is improved. In addition, effective data and a unified data form are obtained by carrying out data cleaning and data conversion processing on the original data, the stability of subsequent data analysis is improved, and the data security is improved by obtaining a second data set which does not contain sensitive information of a consultant through desensitization processing.

On the basis of the foregoing embodiment, fig. 6 is a schematic structural diagram of a public legal service data processing apparatus according to an embodiment of the present application. Referring to fig. 6, the public legal service data processing apparatus provided in this embodiment specifically includes: a data acquisition unit 21, a data preprocessing unit 22, a desensitization processing unit 23 and a data integration unit 24.

The system comprises a data acquisition unit 21, a data processing unit and a data processing unit, wherein the data acquisition unit 21 is used for acquiring original data of public legal service consultation of at least one acquisition source, the acquisition source comprises a mobile big data source, a voice data source, a video data source, a webpage data source and a short message data source, and the original data comprises consultation standing book data, ticket data and consultant data;

the data preprocessing unit 22 is configured to perform data cleaning and data conversion processing on the original data to obtain a first data set;

a desensitization processing unit 23, configured to perform desensitization processing on the first data to obtain a second data set;

and the data integration unit 24 is configured to integrate the second data set to obtain fusion data of each consultant, and determine a query index of the corresponding fusion data of the consultant.

Further, the consulting ledger data comprises consulting time, consulting channel, service type, consulting content and lawyer suggestion;

the consultant data comprises name, mobile phone number, academic calendar, age, sex, province, city and specific address;

the apparatus further comprises an association unit;

Further, the data acquisition unit 21 is further configured to acquire voice data of a consultant through voice consultation, perform semantic analysis on the voice data, acquire corresponding voice original data, and store the voice original data in a voice data source database, where the voice original data includes voice standing book data, voice ticket data, and voice consultant data;

Further, the data acquisition unit 21 is further configured to perform multithreading according to a preset acquisition control policy and concurrently acquire original data of public legal service consultation of at least one acquisition source, where each acquisition thread corresponds to each acquisition source;

Further, the data acquisition unit 21 is further configured to restart and then continue transmission when a breakpoint occurs in the acquisition process;

when the restart times reach a threshold value, stopping collecting data;

or when the error record of the acquired original data exceeds a preset error record threshold or the error ratio exceeds a preset error ratio threshold, stopping acquiring the data.

Further, the data preprocessing unit 22 is further configured to perform data cleaning processing on the original data, and remove irregular data and inconsistent data to obtain first preprocessed data;

Further, the data integration unit 24 is further configured to perform fusion according to the data of the consultants in each acquisition source data in the second data set to obtain fusion data corresponding to each consultant;

and establishing an index of the fusion data of each consultant so as to search and acquire the fusion data of the corresponding consultant through the index, wherein the index comprises a unique code identifier.

Further, the device also comprises an intelligent matching unit;

and displaying the legal provision information in a fusion data page of a corresponding consultant for a user to view.

In the method, data sources of channels such as a video end, a voice end, a picture and text end and the like in public legal services are adaptively connected through the multi-source data adapter. The data acquisition unit is used for efficiently and accurately acquiring, storing and storing information, consultation contents and the like of a public legal service consultant in real time, establishing a multi-bin original layer, cleaning the data of the original layer through the data preprocessing unit, filtering invalid data and dirty data, converting part of character types into numerical values, and improving the accuracy and the integrity of the data. The desensitization processing unit is used for desensitizing the telephone number, the identity card number, the name, the family detailed address and the like of the consultant, so that the data security is ensured.

The public legal service data processing device provided by the embodiment of the application can be used for executing the public legal service data processing method provided by the embodiment, and has corresponding functions and beneficial effects.

An embodiment of the present application provides a public legal service data processing apparatus, and with reference to fig. 7, the public legal service data processing apparatus includes: a processor 31, a memory 32, a communication module 33, an input device 34, and an output device 35. The number of processors in the public legal services data processing apparatus may be one or more, and the number of memories in the public legal services data processing apparatus may be one or more. The processor, memory, communication module, input device, and output device of the public legal services data processing apparatus may be connected by a bus or other means.

The memory 32 serves as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the public legal service data processing method according to any embodiment of the present application (e.g., a data acquisition unit, a data preprocessing unit, a desensitization processing unit, and a data integration unit in a public legal service data processing apparatus). The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system and an application program required by at least one function; the storage data area may store data created according to use of the device, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The communication module 33 is used for data transmission.

The processor 31 executes various functional applications of the device and data processing by executing software programs, instructions, and modules stored in the memory, that is, implements the above-described public legal service data processing method.

The input device 34 may be used to receive entered numeric or character information and to generate key signal inputs relating to user settings and function controls of the apparatus. The output device 35 may include a display device such as a display screen.

The public legal service data processing equipment can be used for executing the public legal service data processing method provided by the embodiment, and has corresponding functions and beneficial effects.

Embodiments of the present application also provide a storage medium storing computer-executable instructions that, when executed by a computer processor, are configured to perform a public legal service data processing method, the public legal service data processing method including: the method comprises the steps of collecting original data of public legal service consultation of at least one collection source, wherein the collection source comprises a mobile big data source, a voice data source, a video data source, a webpage data source and a short message data source, and the original data comprises consultation standing book data, ticket data and consultant data; carrying out data cleaning and data conversion processing on the original data to obtain a first data set; desensitizing the first data to obtain a second data set; and integrating the second data set to obtain fusion data of each consultant, and determining a query index of the fusion data of the corresponding consultant.

Storage medium-any of various types of memory devices or storage devices. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the internet). The second computer system may provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media residing in different locations, e.g., in different computer systems connected by a network. The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.

Of course, the storage medium storing the computer-executable instructions provided in the embodiments of the present application is not limited to the public legal service data processing method described above, and may also perform related operations in the public legal service data processing method provided in any embodiments of the present application.

The public legal service data processing apparatus, the storage medium and the public legal service data processing device provided in the foregoing embodiments may perform the public legal service data processing method provided in any embodiment of the present application, and reference may be made to the public legal service data processing method provided in any embodiment of the present application without detailed technical details described in the foregoing embodiments.

The foregoing is considered as illustrative of the preferred embodiments of the invention and the technical principles employed. The present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A public legal service data processing method is characterized by comprising the following steps:

desensitizing the first data to obtain a second data set;

2. The method of claim 1, wherein the consulting ledger data includes consulting time, consulting channel, type of business, consulting content, and attorney advice;

3. The method of claim 1, wherein said collecting raw data of public legal services consultations of at least one collection source is preceded by:

4. The method of claim 1, wherein said collecting raw data of public legal services consultations of at least one collection source comprises:

5. The method of claim 4, wherein after each thread performs data collection corresponding to raw data according to the breakpoint resume policy, the method comprises:

when a breakpoint occurs in the acquisition process, resuming transmission after restarting;

when the restart times reach a threshold value, stopping collecting data;

6. The method of claim 1, wherein the performing data cleansing and data transformation processing on the raw data to obtain a first data set comprises:

7. The method of claim 1, wherein the integrating the second data set to obtain the fused data of each consultant and determining the query index of the fused data of the corresponding consultant comprises:

8. The method of claim 2, wherein said integrating the second data set to obtain the fused data for each consultant comprises:

9. A public legal service data processing apparatus, comprising:

a desensitization processing unit, configured to perform desensitization processing on the first data to obtain a second data set;

10. A public legal service data processing apparatus, comprising:

a memory and one or more processors;

the memory for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-8.

11. A storage medium storing computer-executable instructions for performing the method of any one of claims 1-8 when executed by a processor.