JP7766995B1 - Content generation system, method, and program - Google Patents

Abstract

[Problem] In light of the problems with the prior art, the objective is to simultaneously realize content generation that is high quality on average and has little variation, and furthermore, for the system itself to autonomously and continuously optimize itself during operation. [Solution] The present invention solves the above problems with the configurations described in claims 1 to 3. In particular, the essential components of the system are: 1. self-evaluation vector S for each stage, 2. W update by an online weight learning unit, 3. σ convergence judgment and ternary gate judgment, and 4. control logic that loops these. [Selected Figure] Figure 1

Description

This invention relates to content generation technology that automatically generates text or multimodal content by coordinating multiple heterogeneous LLMs and dynamically learning and quantitatively evaluating its quality. In particular, it applies to systems and methods including AI workflow management, self-evaluation scoring, online weight learning, and ternary gate judgment.

Although generation using a single LLM is widely used, the quality is unstable due to fluctuations in model characteristics and prompts. Traditionally, (i) objective indicators were lacking, and (ii) repeated testing and manual tuning were required, resulting in high operational costs.
Frameworks for linking multiple LLMs, such as AutoGen, and self-assessment techniques, such as LLM as a Judge, have been reported. However, dynamic weight learning, which integrates self-assessment at multiple stages, and
- Loop control based on the standard deviation convergence condition that represents the quality variation,
- No system has been disclosed that has an automated operation mechanism that combines three-value disclosure/regeneration/discard gates, and a configuration that directly incorporates external KPIs (duration, bounce rate, search ranking, etc.) into the loss function.
**External KPIs are non-stationary, noisy data acquired in real time, and require scaling and delay correction with the internal self-assessment score, making it technically difficult to simply incorporate them into a loss function.**

U.S. Patent No. 1,101,7780 U.S. Patent No. 12,111,859

Contentstack website, 2024 Kontent.ai website, 2024 CrafterCMS official documentation, 2024

In light of the issues with the conventional technology mentioned above, the goal is to simultaneously achieve content generation that is of generally high quality and has little variation, and for the system itself to continuously and autonomously optimize itself during operation.

The present invention solves the above problems by providing the configurations set forth in claims 1 to 3. In particular,
1. Self-evaluation vector S for each stage,
2. Update W by online weight learning unit.
3. Convergence test of σ and ternary gate test,
4. A key feature is that the control logic that loops these processes is an essential component.

・Improved the average consensus score by +17 points (e.g., from 75 to 92), reduced the standard deviation by -12 points (from 15 to 3), and stabilized quality.
- Automatic regeneration reduces the intermediate rejection rate from 35% to 8%, significantly reducing operational costs.
- Convergence judgment prevents over-learning, and performance does not deteriorate even over long-term operation.
According to the present invention, the weights are updated autonomously based on the actual performance index,
(i) Search rankings improved by an average of 1.4 positions,
(ii) A 30% reduction in the number of LLM calls;
The technical effect was confirmed.

The Collaborative Pipeline Control Unit (1) receives the input topic and sequentially executes the pipeline processing by the LLMs: draft generation -> structuring and grammar correction -> expression and tone adjustment -> final audit. The results of each stage are passed to the Stage Scoring Unit (3), which records the self-evaluation score S_i calculated by the LLMs themselves. In parallel, the Operational Metric Acquisition Unit (8) acquires KPIs such as visit time, bounce rate, and search result ranking from an external access analysis service, normalizes them, and inputs them into the Dynamic Weight Learning Unit (7a). 7a online updates the weight vector W using a loss function that includes S_i and the KPIs, and the Convergence Judgment Unit (6b) monitors the convergence of the standard deviation σ. After convergence, the weighted aggregation unit (4) calculates the consensus score S_final using the updated W, and the gate decision unit (6a) automatically decides whether to publish, regenerate, or discard the data by referring to the preset (or dynamically updated) thresholds T_pub/T_rej. Stage 1 -> ... -> Stage N -> Score acquisition -> Weight learning -> σ convergence? -> Gate judgment -> Branching for publication/regeneration/discard. When regeneration is performed, an arrow is drawn that loops back to Stage 1. Scoring/weight update sequence diagram Messages for S_i calculation, W update processing, σ calculation, and gate judgment are shown on the timeline. Gradient descent based on equation (1) -> standard deviation calculation -> convergence check ε comparison -> branching to fix W/update. Quality Convergence Graph: Horizontal axis = number of generations, vertical axis = average and standard deviation of consensus score. Convergence is visualized with two lines.

(1) Hardware configuration
Each LLM container is deployed on a Kubernetes(R) cluster. The control unit (1) performs asynchronous orchestration using the NATS message queue, and metadata such as S, W, and σ is stored in PostgreSQL(R).
Recommended resources: 4x A100 GPUs, 256 GB RAM, average latency 1.2 s/stage.

(2) As shown in software flow diagram 2, Stage 1 LLM receives input topic T and generates draft C_1, followed by C_2...C_n. At the end of each stage, a self-evaluation S_i is given. At the end of stage N, weight learning unit (7a) updates W using equation (1) and checks the convergence of σ.

(3) Online learning parameters: learning rate eta = 0.002 (bias correction coefficients beta_1 = 0.9, beta_2 = 0.999)
・Convergence threshold epsilon = 0.5
・Batch size M = 50 generations

The actual measured values of 7-1 to 7-4 are shown in Table 1 and Figure 5, and the improvement in the average value and the convergence of the standard deviation were confirmed.

In a preferred embodiment of the present invention, the initial value of the weight vector W was set to {0.35, 0.20, 0.20, 0.25}. The convergence threshold epsilon was set to 0.5, the publication threshold T_pub was set to 90, and the regeneration threshold T_rej was set to 60. Furthermore, the publication approval condition was set to a standard deviation σ of 3.0 or less for every 100 generations. These values are merely examples, and other values can also be set.

The obtained search result ranking r and average visit time t (seconds) are normalized to the range of 0 to 1.
The loss function L' is defined as L' = 1/2(S_target - Σw_i S_i)^2 + beta_1・(1-t) + beta_2・r.
Beta_1 and beta_2 are just an example of values that are appropriately set within the range of 0.0 to 1.0, and the present invention is not limited to these coefficient values.

(5) Modified Example
Includes variable stage configurations for N>=2, dynamic changes to T_pub/T_rej, multilingual support, and multimodal LLM.

Table 1 below shows examples of self-assessment scores hypothetically obtained based on a simulation to demonstrate the validity of the present invention.

| Table 1: Validity evaluation of self-assessment scores (provisional data) |
|---|---|---|
|Evaluation items|Average AI self-assessment score|Average human expert evaluation score|
|Content Validity (Content Accuracy)|87.8|88.5|
|Grammar/Structure Consistency|90.0|89.5|
|Natural Expression/Tone Adjustment|94.0|93.2|
|SEO Optimization|82.0|83.1|
|Overall rating (collective score)|88.5|88.6|

(Note: This data is a hypothetical simulation result and may differ from actual figures.)

"The experimental results shown in Table 1 are hypothetical numerical examples based on simulations to demonstrate the validity of the present invention. Measurement results during actual system operation may differ slightly from these example values, but they are listed as representative values to demonstrate the effects and validity of the present invention."

This system can be applied to a wide range of fields, including CMS, news distribution, and automatic generation of e-commerce product descriptions, and can be used in both SaaS and on-premise formats.

Code Name
1 Cooperative Pipeline Control Unit
2a-2d LLM group
3. Stage-specific scoring section
4 Weighted aggregation part
5. AI communication log DB
6a Gate Judgment Section
6b Convergence judgment section
7a Dynamic weight learning unit
8. Means for obtaining actual operational indicators (KPI acquisition section)

Claims (15)

The system consists of a collaborative pipeline control unit that executes different types of large-scale language models (hereinafter referred to as "LLM") sequentially in a predetermined order in multiple stages and passes the generation results of each stage to the next stage; a stage-specific scoring unit that acquires the evaluation score S_i (a real value between 0 and 100 points) calculated by at least one LLM at the end of each stage and records it in a database; a dynamic weight learning unit that calculates the standard deviation σ of the evaluation score vector S for each generation over the most recent M times (M>=10) and updates the weight vector W by online learning according to the formula w_j <- w_j - eta dL/dw_j (where L = 1/2(S_target - Σw_i S_i)^2, eta is a predetermined learning rate); a convergence judgment unit that fixes W and proceeds to the next batch processing if the difference between the previous σ_prev and the current σ_now is equal to or less than a predetermined threshold epsilon; and a consensus score S_final＝Σw_i A content generation system characterized by comprising a weighted aggregation unit that calculates S_i, and a gate determination unit that determines whether to approve publication, re-generation, or discard by referring to a pre-set or dynamically updated publication threshold T_pub and a re-generation threshold T_rej.
A content generation method using the content generation system described in claim 1, wherein input topics are input to a collaborative pipeline control unit for sequential generation, a self-assessment score is obtained for each stage, a weight learning unit updates W using the formula while monitoring the convergence of σ, a gate determination unit automatically determines whether to publish, regenerate, or discard based on T_pub/T_rej, and each of the above processes is looped until the convergence determination condition is met. A computer-readable non-transitory recording medium having recorded thereon a program for causing a computer to execute the method of claim 2. The content creation system of claim 1, wherein the stage configuration consists of four stages: draft generation, structuring and grammar correction, expression and tone adjustment, and final audit. The content generation system described in claim 4, wherein the weight vector W has a predetermined initial value. The content generation system described in claim 1, wherein the convergence threshold epsilon is equal to or less than a predetermined threshold. The content generation system according to claim 1, wherein the publication threshold T_pub and the regeneration threshold T_rej are each set to a predetermined value. The content creation system described in claim 7, characterized in that a requirement that the standard deviation σ for each predetermined number of items be below a predetermined condition is added to the conditions for approval of publication. The content generation system according to claim 1, characterized in that time-stamped natural language comments are stored in a log database along with the self-assessment score. The content generation system described in claim 1, wherein each component is implemented as a microservice on a Kubernetes(R) cluster and is called via an external API. The content generation system of claim 1, wherein the weight learning unit updates the weight vector W using reinforcement learning in addition to gradient descent. The content generation system of claim 1, wherein the LLM is a multimodal configuration including an image generation model. The content generation system of claim 1, characterized by a retry control that, when a regeneration signal is output, regeneration is performed up to three times, and if T_pub is still not reached, the content is discarded. The content creation system of claim 1, further comprising a monitoring mechanism that transmits the collegial score, σ, and W to an external dashboard in real time. The content generation system of claim 1 further comprises an actual performance indicator acquisition means, wherein the actual performance indicator acquisition means acquires at least one of page visit time, exit rate, or search result ranking via an external access analysis service, and the dynamic weight learning unit updates the weight vector using a loss function including the actual performance indicator.