JP7312296B1 - Judgment device and judgment method - Google Patents

Abstract

A determination device and determination method capable of appropriately determining a message containing an inappropriate expression is provided. SOLUTION: The determination device includes a management unit that manages a message between a requester and a provider, and a control unit that determines whether or not the message is inappropriate. A predictive model constructed with message variables relating to one or more words contained in the message and attribute variables relating to attributes of the message is used to determine whether the message is inappropriate. [Selection drawing] Fig. 2

Description

The present invention relates to a determination device and a determination method.

In recent years, with the development of information communication technology, various services have been provided. Techniques for detecting inappropriate expressions included in communications have also been proposed in providing various services. For example, in a service that provides advertisements, a technology has been proposed for detecting inappropriate expressions contained in advertisements and restricting display of advertisements containing inappropriate expressions (for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2021-26690

By the way, as one of various services, there is known a service (hereinafter referred to as a matching service) that assists the requester in receiving various information from the provider by matching the requester and the provider. . For example, if the requester is a job seeker (e.g., student) who wishes to work for a specified company, and the provider is a current employee or retired employee of the specified company, the requester wishes to find a job at the specified company or change jobs. A job seeker (e.g., a member of society) who wants to find a job, and the provider is an agent who mediates employment or job change to a predetermined company.

In the matching service, it is assumed that two-way communication will be carried out between the requester and the provider, and that messages will be sent and received several times.

Under such circumstances, it is desirable to determine messages that contain inappropriate expressions, but if inappropriate expressions are set broadly, there is a possibility that the number of messages containing inappropriate expressions will be enormous. be. Therefore, restricting the transmission of messages containing inappropriate expressions hinders smooth communication, and visual inspection of messages containing inappropriate expressions increases the workload involved in the visual inspection.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made to solve the above-described problems, and it is an object of the present invention to provide a judgment device and judgment method capable of appropriately judging a message containing an inappropriate expression. .

One aspect of the disclosure includes a management unit that manages a message between a requester and a provider, and a control unit that determines whether the message is inappropriate, the control unit comprising: is a judgment device that judges whether or not the message is inappropriate by using a prediction model constructed from message variables relating to one or more words included in the message and attribute variables relating to attributes of the message.

One aspect of the disclosure comprises a step A of managing a message between a requestor and a provider, and a step B of determining whether the message is inappropriate, wherein the step B includes determining whether the message is inappropriate using a predictive model constructed by message variables relating to one or more words contained in and attribute variables relating to attributes of the message. be.

ADVANTAGE OF THE INVENTION According to this invention, the determination apparatus and determination method which enable it to determine appropriately the message containing an inappropriate expression can be provided.

FIG. 1 is a diagram showing a determination system 100 according to an embodiment. FIG. 2 is a diagram showing the determination device 30 according to the embodiment. FIG. 3 is a diagram showing a determination method according to the embodiment. FIG. 4 is a diagram showing a determination method according to the embodiment. FIG. 5 is a diagram for explaining learning according to Modification 1. FIG. FIG. 6 is a diagram for explaining the determination according to Modification 1. FIG.

Embodiments will be described below with reference to the drawings. In addition, in the following description of the drawings, the same or similar reference numerals are given to the same or similar parts.

However, it should be noted that the drawings are schematic, and the ratio of each dimension may differ from the actual one. Therefore, specific dimensions should be determined with reference to the following description. In addition, it is needless to say that the drawings may include portions having different dimensional relationships or ratios.

[Summary of Disclosure]
A determination device according to an outline of the disclosure includes a management unit that manages a message between a requester and a provider, and a control unit that determines whether the message is inappropriate, wherein the control unit , using a predictive model constructed by message variables relating to one or more words contained in the message and attribute variables relating to attributes of the message, to determine whether the message is inappropriate.

A determination method according to an overview of the disclosure, one aspect of the disclosure includes a step A of managing a message between a requester and a provider, and a step B of determining whether the message is inappropriate. wherein the step B determines whether the message is inappropriate using a predictive model constructed by message variables relating to one or more words contained in the message and attribute variables relating to attributes of the message. and a step.

In an overview of the disclosure, a determination device determines whether a message is inappropriate using a predictive model constructed by message variables relating to one or more words contained in the message and attribute variables relating to attributes of the message. According to such a configuration, when assuming a matching service between a requester and a provider, etc., it is possible to determine inappropriateness by considering attributes of the message instead of simply making a determination based on words included in the message. messages containing such expressions can be properly determined.

Note that the requester may be a job seeker who wishes to belong to a predetermined organization, and the provider may be an employee or retired employee of the predetermined organization. A requester may be a job seeker who wishes to belong to a given organization, and a provider may be an agent who mediates membership to the given organization.

[Embodiment]
(judgment system)
The determination system according to the embodiment will be described below. FIG. 1 is a diagram showing a determination system 100 according to an embodiment.

As shown in FIG. 1, the determination system 100 has a first terminal 10, a second terminal 20, and a determination device 30. FIG. The first terminal 10, the second terminal 20, and the determination device 30 are connected by a network 200. FIG. Although not particularly limited, the network 200 may be configured by the Internet network. Network 200 may include a local area network, a mobile communication network, or a VPN (Virtual Private Network).

The determination system 100 (determination device 30) is a system (device ).

For example, the requester may be a job seeker (eg, student, working person) who wishes to belong (eg, find a job or change jobs) to a predetermined organization (eg, company, etc.). A provider may be a resident or retiree of a given organization, or an agent who mediates belonging (eg, employment or job change) to a given organization. In such cases, the matching service may be considered a recruitment support service.

For example, the requester may be a person who wishes to acquire the predetermined information, or a person who provides the predetermined information. In such cases, the matching service may be considered an information support service.

A case in which the matching service is a recruitment support service will be exemplified below. Although not particularly limited, the recruitment support service may provide the requester (e.g., student) with a provider (e.g., enrolled or retired) who graduated from the same school (high school, college, graduate school, etc.) as the requester. May include services to introduce. Recruitment assistance services may include services that introduce a requestor (eg, an employee) to a provider (eg, an agent) who will help the requester (eg, an employee) find a job or change jobs.

The first terminal 10 is a terminal used by a requester. For example, the first terminal 10 may be a personal computer, a smart phone, or a tablet terminal. First terminal 10 transmits a message to second terminal 20 . Although not particularly limited, the message may be in the form of text data such as e-mail or chat. However, the message may be in the form of data obtained by converting voice into text. Hereinafter, the first terminal 10 may also be referred to as a requester.

The second terminal 20 is a terminal used by the provider. The second terminal 20 may be a personal computer, a smart phone, or a tablet terminal. The second terminal 20 transmits a message to the first terminal 10. FIG. Although not particularly limited, the message may be in the form of text data such as e-mail or chat. However, the message may be in the form of data obtained by converting voice into text. In the following, the second terminal 20 may also be referred to as a provider.

The determination device 30 determines whether communication (messages in the embodiment) between the first terminal 10 and the second terminal 20 is inappropriate. The determination device 30 may obtain the message in real time by relaying the message, or may obtain the message in real time or afterward from a server that relays the message. Details of the determination device 30 will be described later.

(Determination device)
A determination device according to an embodiment will be described below. FIG. 2 is a diagram showing the determination device 30 according to the embodiment. As shown in FIG. 2 , the determination device 30 has a communication section 31 , a management section 32 and a control section 33 .

The communication unit 31 is configured by a communication module. The communication module can be a wireless communication module conforming to standards such as IEEE802.11a/b/g/n/ac/ax, LTE, 5G, 6G, etc., and a wired communication module conforming to standards such as IEEE802.3 may be

In the embodiment, the communication unit 31 receives messages between the first terminal 10 and the second terminal 20. FIG. The communication unit 31 may receive the message in real time by relaying the message. The communication unit 31 may receive messages in real time or afterward from a server or the like that relays messages.

The management unit 32 is configured by a storage medium such as an SSD (Solid State Drive) or HDD (Hard Disk Drive), and stores various information.

In the embodiment, the manager 32 constitutes a manager that manages messages between the first terminal 10 (requester) and the second terminal 20 (provider). Messages may be managed in association with a combination of requester and provider. Messages may be managed in association with message transmission times.

The management unit 32 may manage attributes of the requester. If the requester is a student, the attributes of the requester include information that identifies the requester (name, ID, etc.), occupation of the requester (e.g., student, college student, graduate student, part-time job, etc.), requester's Affiliation (eg, home school, home university, department, etc.) may also be included. If the requester is a member of society, the attributes of the requester include information that identifies the requester (name, ID, etc.), industry or occupation of the requester (for example, current or past industry or occupation), requester affiliation (eg, current or past company).

The management unit 32 may manage attributes of the provider. If the provider is enrolled or retired from a given organization, the attributes of the provider include information that identifies the provider (name, ID, etc.), origin of the provider (for example, school attended, university attended, graduated faculty, etc.). The attribute of the provider may include the provider's affiliation (for example, affiliated company, etc.). Attributes of the provider may include age of the provider, occupation in which the provider is engaged, and the like. If the provider is an agent, information that identifies the provider (name, ID, etc.), affiliation of the provider (for example, the agent company to which the agent belongs, etc.), industry that the provider specializes in mediation (for example, Manufacturing industry, IT industry, etc.) or job type (eg, software engineer, legal personnel, etc.). Agents may also be referred to as third parties.

Control unit 33 may include at least one processor. At least one processor, CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), one or more Integrated Circuit, one or more Discrete Circuit, and may be configured by a combination thereof .

In the embodiment, the control unit 33 constitutes a control unit that determines whether or not the message is inappropriate. In other words, the control unit 33 determines messages containing inappropriate expressions. Inappropriate expressions may be combinations of one or more inappropriate words as described below.

Specifically, the control unit 33 determines whether or not the message is inappropriate using a prediction model constructed from message variables relating to one or more words included in the message and attribute variables relating to attributes of the message. . Specifically, the control unit 33 uses a predictive model to calculate the probability that a message is inappropriate, and determines that a message whose calculated probability exceeds a threshold is inappropriate. Details of the prediction model will be described later.

First, the message variables are whether inappropriate words appear in messages managed (accumulated) by the management unit 32, the number of times inappropriate words appear in messages, and the number of times inappropriate words appear in messages. may be defined by one or more data selected from the order in which

Here, inappropriate words may be words that are determined to be inappropriate in light of the purpose of the matching service. Inappropriate words include, but are not limited to, illegal words, antisocial words, words intended to slander or slander, religious words, words aimed at inappropriate encounters, matching It may contain words etc. for the purpose of communication in a method other than the service. Inappropriate encounters may include attempting to contact the requestor or provider without using the matching service, attempting to contact the requestor or provider in violation of the matching service's terms, and the like. For example, an inappropriate encounter may include an agent seeking affiliation with a given organization attempting to contact a requester without using a matching service.

Inappropriate words may be defined by regular expressions (patterns of strings). For example, inappropriate words may be defined by the word alone or in combination with other words.

Inappropriate words may be manually registered by a matching service provider (hereinafter referred to as an operator or the like). Inappropriate words may be managed by the management unit 32 .

Second, the message variable may be a variable that is set for n (where n is an integer equal to or greater than 2) messages between the requestor and the provider. Specifically, a case is assumed in which message communication is performed n times between the same requester and the same provider. Assuming such a case, a message variable may be set for each of the n messages.

Here, a different weighting value may be applied to the message variable for each of the n messages. For example, the weighting value of the message variable of the nth message may be higher than the weighting value of the message variable of the n-1th message. A weighting value may be a value for adjusting the importance of a message variable.

The attribute variable may be defined by one or more data selected from requester attributes, provider attributes, message transmission time, message transmission times, and message transmission frequency.

The attributes of the requester may include, as described above, information identifying the requester, occupation of the requester, affiliation of the requester, and the like. The attributes of the requester can be identified by information managed by the management unit 32. FIG.

As described above, the attributes of the provider may include information identifying the provider, the origin of the provider, the affiliation of the provider, the age of the provider, the occupation in which the provider is engaged, and the like. . The attribute of the provider can be identified by information managed by the management unit 32. FIG.

The transmission time of the message may be the time when the message is transmitted from the requester to the provider, or the time when the message is transmitted from the provider to the requester. The transmission time of the message can be identified by information managed by the management unit 32. FIG.

The number of message transmissions may be the number of times the requester sends the message to the provider, or the number of times the provider sends the message to the requester. The number of message transmissions is the number of transmissions of messages between the requestor and the provider (i.e., the sum of the number of messages sent from the requester to the provider and the number of messages sent from the provider to the requester). may The number of message transmissions can be specified by information managed by the management unit 32 .

The number of message transmissions may be the frequency of messages sent from the requester to the provider in a certain period of time, or the number of times the message was sent from the provider to the requester in a certain period of time. The number of message transmissions may be the number of transmissions of messages between the requester and the provider (i.e., the frequency of sending messages from the requester to the provider and from the provider to the requester over a period of time). . For example, the certain period may be set arbitrarily, such as one day or one week. The message transmission frequency can be identified by information managed by the management unit 32 .

(prediction model)
A prediction model according to the embodiment will be described below. The probability that a message is inappropriate may be expressed as p=F(m,x), where m is the message variable and x is the attribute variable. p may be a probability of taking a value between 0 and 1. In such cases, F is an example of a prediction model (prediction function).

For example, consider the case of determining whether the n-th message is inappropriate. In such cases, the message variable m and the attribute variable x may be expressed as follows.

First, the message variable m may contain the variables shown below.

Message variables for the nth message: m _n _1, m _n _2, m _n _3,…
Message variables for the n-1th message: m _n-1 _1, m _n-1 _2, m _n-1 _3,…
Message variables for the n-2th message: m _n-2 _1, m _n-2 _2, m _n-2 _3,…
…

“m _n _1, m _n _2, m _n _3,…”, “m _n-1 _1, m _n-1 _2, m _n-1 _3,…”, “m _n-2 _1, m _n-2 _2 , m _n-2 _3, …” indicates whether inappropriate words appear in the message, the number of times inappropriate words appear in the message, and the order in which the inappropriate words appear in the message. and so on. m _n _1, m _n _2, m _n _3,…, m _n-1 _1, m _n-1 _2, m _n-1 _3,…, m _n-2 _1, m _n-2 _2, m _n-2 A different weighting value may be applied to each of _3, . As noted above, the message variable may have a different weight applied to each of the n messages.

Second, the attribute variable x may contain the variables shown below.

Requestor attribute variables: xs_1, xs_2, …
Provider attribute variables: xo_1, xo_2, …

'xs_1, xs_2, . As described above, "xo_1, xo_2, ..." is information that identifies the provider, and attributes of the provider include the provider's origin, affiliation, age of the provider, occupation in which the provider is engaged, etc. may contain. A different weighting value may be applied to each of xs_1, xs_2, . A different weighting value may be applied to each of xo_1, xo_2, .

Under this premise, the determination device 30 determines that the n-th message is inappropriate when p=F(m, x) exceeds the threshold p_0 (=0 to 1). On the other hand, the determination device 30 determines that the n-th message is inappropriate when p=F(m, x) does not exceed the threshold p_0 (=0 to 1). When determining that the n-th message is inappropriate, the determination device 30 may notify the operator or the like that the message is inappropriate.

Here, the threshold p_0 may be set to allow a message to be erroneously determined to be inappropriate. That is, the threshold p_0 may be set so that an erroneous determination that the message is not inappropriate is unlikely to occur, and erroneous determination that the message is inappropriate is allowed. The threshold p_0 may be set based on a history of whether messages (past messages) managed (accumulated) by the management unit 32 are inappropriate. The threshold p_0 may be set by an operator or the like.

(Determination method)
A determination method according to the embodiment will be described below. 3 and 4 are diagrams for explaining the determination method according to the embodiment.

First, construction of a prediction model will be described with reference to FIG.

As shown in FIG. 3, messages are transmitted and received between the first terminal 10 and the second terminal 20 in steps S10 and S11. Steps S10 and S11 may be repeated.

In step S12, the determination device 30 manages (accumulates) messages between the first terminal 10 and the second terminal 20. FIG. The decision device 30 may receive the message in real time by relaying the message. The determination device 30 may receive a message in real time or afterward from a server that relays the message.

In step S13, the determination device 30 constructs a prediction model and stores the constructed prediction model. The prediction model may be constructed based on the messages (past messages) managed (accumulated) in step S12.

Here, the operator or the like may refer to the messages (past messages) managed (accumulated) in step S12 and manually specify the following contents. An operator or the like may manually identify whether each of the messages is inappropriate. An operator or the like may manually identify inappropriate words. Operators and others may specify the types of message variables that define the prediction model. The operator or the like may specify the types of attribute variables that define the prediction model. The operator or the like may specify a weighting value to apply to the message variables. The operator or the like may specify weighting values to apply to the attribute variables. An operator or the like may specify a threshold for determining whether each message is inappropriate.

In this way, it should be noted that the operator or the like can build a prediction model on the premise that sufficient messages between the requester and the provider can be managed by the management unit 32.

Second, the determination of whether a message is inappropriate will be described with reference to FIG.

As shown in FIG. 4, messages are transmitted and received between the first terminal 10 and the second terminal 20 in steps S20 to S23. Transmission and reception of messages between the first terminal 10 and the second terminal 20 may be repeated.

In step S30, the determination device 30 uses the prediction model constructed in step S13 to determine whether the message is inappropriate. Specifically, the determination device 30 calculates the probability that the message is inappropriate, and determines that the message for which the calculated probability exceeds a threshold is inappropriate.

Note that the message to be judged as inappropriate may be the message transmitted from the first terminal 10 to the second terminal 20 in step S22, and the message transmitted from the second terminal 20 to the first terminal in step S23. It may be a message sent to 10.

(Action and effect)
In an embodiment, the determination device 30 determines whether or not a message is inappropriate using a predictive model constructed from message variables relating to one or more words contained in the message and attribute variables relating to attributes of the message. According to such a configuration, when assuming a matching service between a requester and a provider, etc., it is possible to determine inappropriateness by considering attributes of the message instead of simply making a determination based on words included in the message. messages containing such expressions can be properly determined.

[Modification 1]
Modification 1 of the embodiment will be described below. In the following, mainly the differences with respect to the embodiments will be described.

In the embodiment, the case where the prediction model is a prediction function F constructed by an operator or the like is exemplified. On the other hand, in Modification 1, a prediction model is constructed by learning about messages (past messages) managed by the management unit 32 .

Here, it is assumed that the operator or the like specifies whether or not the message managed by the management unit 32 is inappropriate. That is, it assumes that messages have already been labeled as inappropriate or not.

In Modification 1, a prediction model is constructed by learning learning data, using message variables, attribute variables, and whether or not a message is inappropriate as learning data. Since the messages have already been labeled whether they are inappropriate or not, the learning data can be considered to be training data.

First, as shown in FIG. 5, the learning unit 35A learns the correlation between message variables, attribute variables, and whether or not the message is inappropriate. The learning unit 35A may be provided in the control unit 33 described above. Message variables and attribute variables are specified by information managed by the management unit 32 . Whether or not the message is inappropriate is specified by an operator or the like. Learning unit 35A constructs a prediction model for calculating the probability that a message is inappropriate as a learning result.

As a learning method, deep learning represented by AI (Artificial Intelligence) may be used. For example, as a learning method, logistic regression may be used, or Gradient Boosting Decision Tree may be used.

Second, as shown in FIG. 6, the determination unit 35B inputs the message variables and attribute variables of the target message to be determined as inappropriate to the prediction model, and calculates the probability that the target message is inappropriate. can be calculated. The determination unit 35B may determine that a message whose calculated probability exceeds a threshold is inappropriate. The determination unit 35B may be provided in the control unit 33 described above.

Modification 1 illustrates the case where the prediction model is a model for calculating the probability that a message is inappropriate. However, Modification 1 is not limited to this. A predictive model may be a model for identifying whether a message is inappropriate.

[Modification 2]
Modification 2 of the embodiment will be described below. In the following, mainly the differences with respect to the embodiments will be described.

Modification 2 mainly describes a case where the determination device 30 determines whether or not a message regarding an agent who mediates belonging to a predetermined organization is inappropriate.

For example, the attribute variables may include a history of scout mails to the requester (number of times, frequency, time, etc.) and a history of reply mails to scout mails from the requester (number of times, frequency, time, etc.).

For example, attribute variables may include membership type, such as whether the requester is a paying member who pays for the matching service or whether the requester is a free member who does not pay for the matching service.

Furthermore, in Modification 2, other variables may be used in addition to the message variables and attribute variables as variables for constructing the prediction model. Other variables may include variables related to matching services (hereinafter service variables).

For example, service variables include the history of the requester's application for belonging to a given organization (application history), the history of interviews or interviews between the person in charge of the given organization and the requester (interview/interview history), and the history of the requester's attention. organization registration history (favorite registration history), requester's work history (resume) update history (resume update history), requester's browsing history of job postings of a given organization (browsing history), job postings of a given organization may include a history (search history) of searches by the requester.

For example, the service variables may include whether or not a requester has been selected in a given organization, the progress history of the requester's selection in the given organization, the change history of job postings in the given organization, the hiring history of the requester in the given organization, and the like.

In such a case, the determination device 30 may have a database that manages information for identifying service variables, and may communicate with the database that manages information for identifying service variables. good. Matching services may include services such as management of recruiters' job postings, management of job applicants' work histories, management of job applicant selection, and the like. The requester mentioned above may be read as a job seeker. The predetermined organization or provider described above may be read as adopter.

[Other embodiments]
Although the present invention has been described by the above-described embodiments, the statements and drawings forming part of this disclosure should not be construed as limiting the present invention. Various alternative embodiments, implementations and operational techniques will become apparent to those skilled in the art from this disclosure.

In the above disclosure, inappropriate words are manually registered by an operator or the like. However, the above disclosure is not so limited. On the premise that whether or not the message managed by the management unit 32 is inappropriate is specified by an operator or the like, inappropriate words are automatically registered by analyzing the message specified as inappropriate. may be For example, the analysis may be a technique such as TF-IDF (Term Frequency-Inverse Document Frequency).

Although not specifically mentioned in the above disclosure, messages between requesters and providers may be one-to-one messages, one-to-many messages, and many-to-many messages. There may be.

In the above disclosure, the case where the management unit 32 is provided in the determination device 30 has been exemplified. However, the above disclosure is not so limited. The management unit 32 may be included in the same external server that can communicate with the determination device 30 as a separate entity from the determination device 30, or may be included in a separate external server that can communicate with the determination device 30. FIG.

Although not specifically mentioned in the above disclosure, a program that causes a computer to execute each process performed by the determination device 30 may be provided. Also, the program may be recorded on a computer-readable medium. A computer readable medium allows the installation of the program on the computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be, for example, a recording medium such as CD-ROM or DVD-ROM.

Alternatively, a chip configured by a memory storing a program for executing each process performed by the determination device 30 and a processor executing the program stored in the memory may be provided.

10... first terminal, 20... second terminal, 30... determination device, 31... communication unit, 32... management unit, 33... control unit, 40... third party, 100... determination system, 200... network

Claims (3)

a management unit that manages messages between the job seeker and the provider in a recruitment support service that introduces the provider to the job seeker;
A control unit that determines whether the message is inappropriate,
The control unit uses a predictive model to determine whether the message is inappropriate,
The predictive model uses, as learning data, message variables related to one or more words included in the message, attribute variables related to attributes of the message, service variables related to the recruitment support service, and whether or not the message is inappropriate. constructed by learning the training data,
The management unit manages information for identifying the service variables,
The message variable is defined by one or more data selected from whether inappropriate words appear in the message and the number of times inappropriate words appear in the message,
The attribute variable is defined by one or more data selected from attributes of the job seeker, attributes of the provider, transmission time of the message, number of transmissions of the message, and frequency of transmission of the message,
The service variables include a history of the job seeker's application for belonging to a predetermined organization, a history of an interview or an interview between the person in charge of the predetermined organization and the job seeker, a registration history of the organization of interest to the job seeker, Update history of the job seeker's work history, history of the job seeker's viewing of job postings of the predetermined organization, history of search by the job seeker of job postings of the predetermined organization, presence/absence of selection of the job seeker at the predetermined organization , the progress history of the selection of the job seeker in the predetermined organization, the change history of the job offer slip of the predetermined organization, and the employment history of the job seeker in the predetermined organization , defined by one or more data selected Device. When communication of n messages between the same job seeker and the same provider is executed, the control unit controls the above based on the message variable set for each of the n messages 2. The determination device according to claim 1, which determines whether n messages are inappropriate. Step A in which a determination device manages messages between the job seeker and the provider in a recruitment support service that introduces the provider to the job seeker;
A step B in which the determination device determines whether the message is inappropriate,
The step B includes determining whether the message is inappropriate using a predictive model,
The predictive model uses, as learning data, message variables related to one or more words included in the message, attribute variables related to attributes of the message, service variables related to the recruitment support service, and whether or not the message is inappropriate. constructed by learning the training data,
The step A includes managing information for identifying the service variable,
The message variable is defined by one or more data selected from whether inappropriate words appear in the message and the number of times inappropriate words appear in the message,
The attribute variable is defined by one or more data selected from attributes of the job seeker, attributes of the provider, transmission time of the message, number of transmissions of the message, and frequency of transmission of the message,
The service variables include a history of the job seeker's application for belonging to a predetermined organization, a history of an interview or an interview between the person in charge of the predetermined organization and the job seeker, a registration history of the organization of interest to the job seeker, Update history of the job seeker's work history, history of the job seeker's viewing of job postings of the predetermined organization, history of search by the job seeker of job postings of the predetermined organization, presence/absence of selection of the job seeker at the predetermined organization , the progress history of the selection of the job seeker in the predetermined organization, the change history of the job offer slip of the predetermined organization, and the employment history of the job seeker in the predetermined organization , defined by one or more data selected Method.