Home > Blog > プログラミング > TruthTorchLMによるLLMのハルシネーション検出

TruthTorchLMによるLLMのハルシネーション検出

大規模言語モデル（LLM）の目覚ましい進化は生活やビジネスに革新をもたらす一方で、事実に基づかない情報を生成するハルシネーションが問題となっています。特に医療や金融といった高リスクな分野では、LLM出力の信頼性と正確性が不可欠です。この重要な課題に対処するため、LLM出力の真実性を評価・予測するための包括的なオープンソースPythonライブラリであるTruthTorchLMが開発されました。

TruthTorchLMは、30以上の真実性評価手法（Truth Methods）を提供し、Google検索による事実確認から不確実性推定、複数LLM連携技術まで多岐にわたります。既存のGuardrailsやLM-Polygraphといったツールとは異なり、文書に基づく検証手法だけでなく、より広範なアプローチを統合している点が特徴です。HuggingfaceとLiteLLMの両フレームワークとシームレスに互換性があり、生成、評価、キャリブレーション、長文生成の真実性予測といった主要機能を統一されたインターフェースで提供します。これにより、技術者は容易に新しい手法を実装・評価でき、LLMの信頼性向上に貢献することができます。

TruthTorchLMとは？

1. TruthTorchLMの概要と開発背景

TruthTorchLM（TTLM）は、大規模言語モデル（LLMs）の出力の真実性を予測するための包括的なオープンソースライブラリです。近年、LLMの目覚ましい発展とともに、その出力が事実に基づかない情報、いわゆる「ハルシネーション」を生成する問題が浮上しています。特に、信頼性と正確性が不可欠な高リスクな状況（医療、法律、金融など）において、このハルシネーションの検出は喫緊の課題となっています。

TruthTorchLMは、既存のツールキット（例: GuardrailsやLM-Polygraph）が抱える限界に対処し、より広範で拡張可能な真実性予測手法を提供することを目指して開発されました。これにより、LLMの信頼性を向上させ、より安全なAIシステムの構築に貢献します。

2. 主な特徴と機能

TruthTorchLMは、その目的を達成するために以下の主要な特徴と機能を備えています。

  • 真実性予測の加速と多様な手法の提供: 研究を加速させ、真実性予測手法へのアクセスを容易にするために導入されました。30以上の「Truth Methods」と呼ばれる真実性予測手法を特徴としており、これらは計算コスト、アクセスレベル（ブラックボックス、グレーボックス、ホワイトボックス）、グラウンディングドキュメントの要件、および教師の種類（自己教師ありまたは教師あり）において多様なトレードオフを持っています。これにより、様々なユースケースや要件に応じた最適な手法を選択することが可能です。
  • 既存ツールとの差別化: 既存のツールキットと比較して、より包括的なアプローチを提供します。
    • Guardrailsがドキュメントベースの検証に焦点を当てているのに対し、TruthTorchLMは不確実性推定手法など、外部ドキュメントに依存しない幅広い方法をサポートしています。
    • LM-Polygraphが不確実性ベースの手法に限定されているのに対し、TruthTorchLMは教師ありアプローチ、ドキュメントチェック技術、ツールベースの戦略など、より広範なコレクションを提供します。
  • シームレスな互換性: HuggingFaceおよびLiteLLMとシームレスに互換性があり、ローカルホスト型モデルとAPIベースモデルの両方をサポートします。これにより、既存のLLMワークフローに容易に統合し、真実性評価・予測機能を組み込むことができます。

TruthTorchLMに実装されている真実性予測手法

TruthTorchLMは、LLMの出力の真実性を評価・予測するために、30種類以上の多様な「真実メソッド（Truth Methods）」を実装しています。これらのメソッドは、計算コスト、LLMへのアクセスレベル（ブラックボックス、グレーボックス、ホワイトボックス）、外部ドキュメントの要件、および教師の種類（自己教師ありまたは教師あり）といった様々な側面で異なるトレードオフを持ちます。これにより、ユーザーは自身のニーズや利用可能なリソースに応じて最適な手法を選択できます。

以下に、TruthTorchLMに実装されている代表的な真実性予測手法の一部を、その概要、特徴、メリット、デメリットとともにご紹介します。

手法概要特徴メリットデメリット
LARSLLMの内部状態を観察し、出力ごとの真実性を数値化する。特徴量としてトークン表現等を使用。LLMの出力の内部表現を解析。LLMの出力のみで追加外部リソース不要。内部状態観察のテクニカルノウハウ必要、公開実装が少ない。
ConfidenceLLMが各出力に対して割り当てる確信度スコアや確率値を指標とする。LLMがどの程度自信を持って答えているか数値化。計算コストが低く、実装が容易。自信あり＝正しいとは限らず、ハルシネーション時も高信頼度になることがある。
Entropy生成時の確率分布のばらつきをエントロピーとして計測し、不確実性の指標とする。不確実性を確率論的に計測、エントロピー大＝迷い。モデルの不安定性も分かる、数理的に扱いやすい。低エントロピーでもハルシネーションする例がある。
SelfDetection出力の正しさを自己言及させ、自己評価を通じて事実性を推定。自問自答方式、自己評価による検出。外部知識に依存せず汎用性がある。LLM自身のバイアスに影響されやすく、根拠が曖昧。
AttentionScore出力におけるアテンション重みを解析し、重要情報への集中度から信頼性を評価。内部の注意配分から根拠乏しい出力を識別。LLMの動作解釈に役立つ。解釈が難しい場合が多く、ブラックボックス性を完全には克服できない。
CrossExamination複数の視点やモデルで同じ質問を行い、回答の矛盾や整合性から真偽を判定。矛盾や不一致を調べる比較的外部的な手法。ハルシネーションの再発見（再現性）を測定しやすい。計算コストが高く、複数回答・モデルが必要。
EccentricityConfidence出力の確信度と異常性（分布の特異性）を組み合わせて真偽を評価。確信と異常の複合尺度。信頼・異常両面をカバーするため判定の幅が広い。パラメータ調整や閾値決定が難しい場合がある。
EccentricityUncertainty不確実性と異常性の組合せにより、信頼できない出力を検出。不確実性に重みハルシネーション傾向が目立ちやすい。精度と安定性の両立が難しい場合がある。
GoogleSearchCheck出力内容をGoogle等の検索結果と比較し、事実性の裏付けを取る。実社会情報と突き合わせる外部ベース手法。情報の即時アップデート性や精度が高い。APIや外部サービスへの依存、コスト増大、最新情報でも誤情報には対応不可。
Insideモデルの中間表現（hidden states）の層ごとの動きを追跡して解析。hidden stateの変化や構造分析。LLMの内部動作の可視化、学習応用も可能。実装や解釈が難易度高、汎用性に課題。
KernelLanguageEntropyカーネル法により言語生成過程の分布エントロピーを高次元空間で分析。高次元・複雑なパターン捉える。複雑な非線形分布も扱える。高度な数学的知識や演算資源が求められる。
MARS出力の信頼性や一貫性を行列ベースでスコア化する手法。出力間の信頼性や一貫性を行列的に評価。スケーラブルな設計。行列表現の設計や数値安定性に注意が必要。
MatrixDegreeConfidenceモデル出力を構造化し、関係性ネットワーク中のdegreeを信頼尺度に使用。関連・依存関係のネットワークで評価。状態変化を微細に観察。専門知識とチューニングが必要。
MatrixDegreeUncertainty上記と同様の構造を使って不確実性を定量化。不確実性を行列degreeで分析。信頼感の偏りも発見可。定量化の手法選択に左右される。
MiniCheck単純な質問で素早く正誤を推定する軽量な事実性チェック。小規模タスクや一部出力に対して評価。軽量かつ素早いフィードバックが可。部分的な傾向しか分からず、網羅性に劣る。
MultiLLMCollab複数LLMに同一質問を投げて意見を集約、整合性を検証する手法。多数決や意見不一致を集約、協力検証。1モデルより安定性・精度が上がる。モデル間の質によりバラツキ、運用コスト増。
NumSemanticSetUncertainty同義文セットの多様性とその中のばらつきから不確実性を推定。多様性と不確実性の相関で判定。多義的課題への適応力。セットの選定や閾値設定が課題。
PTrue出力が真である確率を、統計的手法で事後的に推定。モデル予測の正確性を直接確率化。解釈性が高い、数値目標として扱いやすい。真偽の定義や教師データの整備が必要。
Saplmaルールや構文パターンに基づき出力の事実性を評価する。ルール・パターン型推論中心。未知語や新規表現にも対応可。高度なパターン設計が必要、柔軟性に課題。
SemanticEntropy意味ベクトル分布のエントロピーを用いて情報の曖昧さを測定。高次元意味空間のばらつき・特徴を考慮。抽象的な多義性もカバー。意味空間の構成や尺度設計が課題。
sentSAR文単位で構文的に信頼度や不確実性を推定する方式（詳細未公開）。文単位でハルシネーション傾向追跡。部分的誤り、冗長検出に有効。意味論・文法の両面評価が必要。
SumEigenUncertainty出力表現行列の固有値合計により、表現の安定性や特異性を評価。特異値・固有ベクトル由来の尺度。行列的な高次関係も定量化。実装・解釈の難度が高い。
tokenSAR出力文の各トークンごとの信頼度や不確実性を個別に評価。単語単位で微細な誤りを計測。局所的異常や誤出力を追跡可。出力が長いとコスト、意味の集約も課題。
VerbalizedConfidenceモデル自身に出力の信頼度を文章で説明させ、主観評価として扱う。自己申告的な信頼評価。可読性が高く、説明変数として有用。主観的で正確性が保証されない。
DirectionalEntailmentGraph出力間の意味的な含意関係をグラフ化し、整合性や矛盾を評価。発言間の含意関係のグラフ解析。論理的な整合・矛盾を可視化しやすい。グラフ構築・パス探索の難易度が高い。

TruthTorchLMの使い方

TruthTorchLMは、大規模言語モデル（LLM）の出力における真実性（truthfulness）を評価・予測するための包括的なオープンソースPythonライブラリです。30種類以上の「真実メソッド（truth methods）」を提供し、Google検索チェックから不確実性推定、複数のLLMによるコラボレーション技術まで、幅広い手法を網羅しています。また、HuggingfaceやLiteLLMと完全に互換性があり、最小限のコード変更で真実性評価・予測をワークフローに統合できます。

以下に、TruthTorchLMの主要な使用例をPythonコードと共に解説します。

1. インストール

まず、TruthTorchLMを使用するために環境を設定し、ライブラリをインストールします。Python 3.10以上の環境が必要です。

# PyTorchを事前にインストールしておく（利用するバージョン等は任意に選択可能）
$ pip install torch

# pipを使ってTruthTorchLMをインストールする
$ pip install TruthTorchLM

# または、GitHubからクローンしてインストールする
$ git clone https://github.com/Ybakman/TruthTorchLM.git
$ cd TruthTorchLM
$ pip install -r requirements.txt

2. 認証情報の設定

OpenAIモデルやSerper APIなどの外部サービスを使用する場合、APIキーを環境変数として設定する必要があります。

import os

# OpenAIモデルを使用する場合
os.environ["OPENAI_API_KEY"] = 'your_open_ai_key'

# 長文生成評価などでSerper APIを使用する場合（https://serper.dev/）
os.environ['SERPER_API_KEY'] = 'your_serper_api_key'

3. モデルの設定

TruthTorchLMは、HuggingfaceモデルとAPIベースのモデルの両方をサポートしています。

from transformers import AutoModelForCausalLM, AutoTokenizer
import TruthTorchLM as ttlm
import torch

# Huggingfaceモデルの設定例
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16).to('cuda:0')
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", use_fast=False)

# APIモデルの設定例
api_model = "gpt-4o"

4. 真実性値を含むテキストの生成 (短文QA)

TruthTorchLMは、モデルの出力が真実であるかどうかを示す「真実性値」を付与してメッセージを生成します。これは主に短文のQ&Aに役立ちます。

# 使用する Truth Method を定義する
# TruthTorchLMには30種類以上の Truth Method が利用可能
lars = ttlm.truth_methods.LARS()
confidence = ttlm.truth_methods.Confidence()
self_detection = ttlm.truth_methods.SelfDetection(number_of_questions=5)
truth_methods = [lars, confidence, self_detection]

# チャット履歴を定義する
chat = [{"role": "system", "content": "You are a helpful assistant. Give short and precise answers."},
        {"role": "user", "content": "What is the capital city of France?"}]

# Huggingfaceモデルで真実性値を含むテキストを生成する場合
# output_hf_model = ttlm.generate_with_truth_value(
#    model=model,
#    tokenizer=tokenizer,
#    messages=chat,
#    truth_methods=truth_methods,
#    max_new_tokens=100,
#    temperature=0.7
# )
# print("Huggingface Model Output:", output_hf_model)

# APIモデルで真実性値を含むテキストを生成する場合
output_api_model = ttlm.generate_with_truth_value(
    model=api_model,
    messages=chat,
    truth_methods=truth_methods
)
print("API Model Output:", output_api_model)
出力例 …
API Model Output: {'generated_text': 'The capital city of France is Paris.', 'normalized_truth_values': [0.7300199033425558, 0.49995302424937516, 0.5], 'unnormalized_truth_values': [0.9947235584259033, -0.00018790300305226992, np.float64(-0.0)], 'method_specific_outputs': [{'truth_value': 0.9947235584259033, 'generated_text': 'The capital city of France is Paris.', 'normalized_truth_value': 0.7300199033425558}, {'truth_value': -0.00018790300305226992, 'generated_text': 'The capital city of France is Paris.', 'normalized_truth_value': 0.49995302424937516}, {'truth_value': np.float64(-0.0), 'entropy': np.float64(0.0), 'consistency': 1.0, 'generated_questions': ['What is the capital city of France?', 'Which city serves as the capital of France?', 'What city is designated as the capital of France?', "What is the name of France's capital city?", 'Can you tell me the capital of France?'], 'generated_texts': ['The capital city of France is Paris.', 'Paris is the capital of France.', 'Paris is the capital of France.', 'The capital city of France is Paris.', 'The capital of France is Paris.'], 'clusters': [['The capital city of France is Paris.', 'Paris is the capital of France.', 'Paris is the capital of France.', 'The capital city of France is Paris.', 'The capital of France is Paris.']], 'normalized_truth_value': 0.5}]}

5. Truth Methodのキャリブレーション

異なるTruth Methodの真実性値は、範囲が異なるため直接比較できない場合があります。calibrate_truth_method関数を使用すると、真実性値を共通の範囲に正規化し、解釈しやすく、比較可能な出力にすることができます。

# 正確性評価モデルを定義する
model_judge = ttlm.evaluators.ModelJudge('gpt-4o-mini')

# 各 Truth Method に正規化の方法（Normalizer）を設定する
for truth_method in truth_methods:
    truth_method.set_normalizer(ttlm.normalizers.IsotonicRegression()) # Isotonic回帰など

# TruthTorchLMの組み込みデータセット（例: 'trivia_qa'）を使用してメソッドをキャリブレーションする
calibration_results = ttlm.calibrate_truth_method(
    dataset='trivia_qa',
    model=api_model,
    truth_methods=truth_methods,
    correctness_evaluator=model_judge,
    size_of_data=5,  # 実用上は1000くらいのサイズがほしいが実行に相当な時間を要する
)
print("Calibration Results:", calibration_results)
出力例 …
Calibration Results: {'previous_context': [{'role': 'system', 'content': 'You are a helpful assistant. Answer the following question in a single brief but complete sentence.'}], 'user_prompt': 'Question: {question} Answer:', 'generation': ['The Royal Albert Hall is located on Kensington Gore.', "Isaac Newton wrote 'Philosophiæ Naturalis Principia Mathematica', published in 1687.", 'The correct way to address a Duke is "Your Grace."', 'The salary paid to a Church of England clergyman is called a "stipend."', 'Valerie Trierweiler was the partner of French politician François Hollande.'], 'generation_correctness': [1, 1, 1, 1, 1], 'question_text': ['The Royal Albert Hall is in which London road?', 'Published in 1687, who wrote ‘Philosophae Naturalis Principa Mathematica’?', 'What is the correct way of addressing a Duke?', 'What is the salary paid to a Church of England clergyman called?', 'Valerie Trierweiler was the partner for some years of which politician?'], 'ground_truths': [['Kensington gore', 'Kensington Gore'], ['Isaac Newton', 'Sir Isaak Newton', 'Isaacus Newtonus', 'Hannah Ayscough', "Isaac Newton's tooth", 'Sir isaac newton', 'Sir Isaac Newton', 'Isaac newton', "Isaac Newton's middle years", 'I. Newton', 'Newtonian science', 'Sir Newton', 'Sir Issac Newton', 'Issac Newton', 'Newton isaac', 'Newton Isaac', 'Issac newton', 'Cultural depictions of Isaac Newton', "Newton's", 'Sir Issaac Newton', 'Isac Newton'], ['Grace (style)', 'His Grace', 'Her Grace', 'His/Her Grace', 'Your Grace', 'Her grace', '"""YOUR GRACE"""'], ['STIPEND', 'Stipend'], ['Francois Hollande', 'Francois Holland', 'François Hollande', 'François Holland', 'Francois hollande', 'Hollande', 'François Gérard Georges Nicolas Hollande', 'Francis Holland']], 'truth_methods': ['LARS', 'Confidence', 'SelfDetection'], 'truth_method_0': {'name': 'LARS with {\'normalizer\': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6b3a2c710>, \'ue_type\': \'confidence\', \'number_of_generations\': 0, \'batch_generation\': True, \'lars_model\': RobertaForSequenceClassification(\n  (roberta): RobertaModel(\n    (embeddings): RobertaEmbeddings(\n      (word_embeddings): Embedding(50273, 768, padding_idx=1)\n      (position_embeddings): Embedding(514, 768, padding_idx=1)\n      (token_type_embeddings): Embedding(1, 768)\n      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): RobertaEncoder(\n      (layer): ModuleList(\n        (0-11): 12 x RobertaLayer(\n          (attention): RobertaAttention(\n            (self): RobertaSdpaSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): RobertaSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): RobertaIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output): RobertaOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n    )\n  )\n  (classifier): RobertaClassificationHead(\n    (dense): Linear(in_features=768, out_features=768, bias=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n    (out_proj): Linear(in_features=768, out_features=1, bias=True)\n  )\n), \'lars_tokenizer\': RobertaTokenizerFast(name_or_path=\'duygunuryldz/LARS\', vocab_size=50265, model_max_length=512, is_fast=True, padding_side=\'right\', truncation_side=\'right\', special_tokens={\'bos_token\': \'<s>\', \'eos_token\': \'</s>\', \'unk_token\': \'<unk>\', \'sep_token\': \'</s>\', \'pad_token\': \'<pad>\', \'cls_token\': \'<s>\', \'mask_token\': \'<mask>\', \'additional_special_tokens\': [\'[prob_token_0]\', \'[prob_token_1]\', \'[prob_token_2]\', \'[prob_token_3]\', \'[prob_token_4]\', \'[prob_token_5]\', \'[prob_token_6]\', \'[prob_token_7]\']}, clean_up_tokenization_spaces=True, added_tokens_decoder={\n\t0: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t50264: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),\n\t50265: AddedToken("[prob_token_0]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50266: AddedToken("[prob_token_1]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50267: AddedToken("[prob_token_2]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50268: AddedToken("[prob_token_3]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50269: AddedToken("[prob_token_4]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50270: AddedToken("[prob_token_5]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50271: AddedToken("[prob_token_6]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50272: AddedToken("[prob_token_7]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n}\n), \'device\': \'cuda\', \'number_of_bins\': 8, \'edges\': [6.197085156145477e-08, 0.48822552817208426, 0.8300329021045139, 0.9682016628129141, 0.9969601631164551, 0.9998409748077393, 0.9999935626983643, 1.0], \'model_for_entailment\': None, \'tokenizer_for_entailment\': None}', 'truth_values': [0.9744278788566589, 0.9845095276832581, 0.9912342429161072, 0.987522542476654, 0.9948524832725525], 'normalized_truth_values': [1.0, 1.0, 1.0, 1.0, 1.0]}, 'truth_method_1': {'name': "Confidence with {'normalizer': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6b23d1ac0>, 'scoring_function': <TruthTorchLM.scoring_methods.length_normalized_scoring.LengthNormalizedScoring object at 0x7ff6ee7fd6a0>}", 'truth_values': [-0.04567849170762202, -0.06678165141858985, -0.1583464831716507, -0.04866549613219067, -0.02941110392789662], 'normalized_truth_values': [1.0, 1.0, 1.0, 1.0, 1.0]}, 'truth_method_2': {'name': 'SelfDetection with {\'normalizer\': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6a781ed20>, \'tokenizer_for_entailment\': DebertaTokenizer(name_or_path=\'microsoft/deberta-large-mnli\', vocab_size=50265, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side=\'right\', truncation_side=\'right\', special_tokens={\'bos_token\': \'[CLS]\', \'eos_token\': \'[SEP]\', \'unk_token\': \'[UNK]\', \'sep_token\': \'[SEP]\', \'pad_token\': \'[PAD]\', \'cls_token\': \'[CLS]\', \'mask_token\': \'[MASK]\'}, clean_up_tokenization_spaces=False, added_tokens_decoder={\n\t0: AddedToken("[PAD]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t1: AddedToken("[CLS]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t2: AddedToken("[SEP]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t3: AddedToken("[UNK]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50264: AddedToken("[MASK]", rstrip=False, lstrip=True, single_word=False, normalized=True, special=True),\n}\n), \'model_for_entailment\': DebertaForSequenceClassification(\n  (deberta): DebertaModel(\n    (embeddings): DebertaEmbeddings(\n      (word_embeddings): Embedding(50265, 1024, padding_idx=0)\n      (LayerNorm): DebertaLayerNorm()\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): DebertaEncoder(\n      (layer): ModuleList(\n        (0-23): 24 x DebertaLayer(\n          (attention): DebertaAttention(\n            (self): DisentangledSelfAttention(\n              (in_proj): Linear(in_features=1024, out_features=3072, bias=False)\n              (pos_dropout): Dropout(p=0.1, inplace=False)\n              (pos_proj): Linear(in_features=1024, out_features=1024, bias=False)\n              (pos_q_proj): Linear(in_features=1024, out_features=1024, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): DebertaSelfOutput(\n              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n              (LayerNorm): DebertaLayerNorm()\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): DebertaIntermediate(\n            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output): DebertaOutput(\n            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n            (LayerNorm): DebertaLayerNorm()\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n      (rel_embeddings): Embedding(1024, 1024)\n    )\n  )\n  (pooler): ContextPooler(\n    (dense): Linear(in_features=1024, out_features=1024, bias=True)\n    (dropout): Dropout(p=0, inplace=False)\n  )\n  (classifier): Linear(in_features=1024, out_features=3, bias=True)\n  (dropout): Dropout(p=0.1, inplace=False)\n), \'number_of_questions\': 5, \'prompt_for_generating_question\': \'Given a question, paraphrase it to have different words and expressions but have the same meaning as the original question. Please note that you should not answer the question, but rather provide a re-phrased. These paraphrased questions should be different from each other. Previous paraphrased questions: {previous_questions}. Only output a single paraphrased question, nothing else. Question: {question}\', \'system_prompt\': \'You are a helpful assistant. Give short and precise answers.\', \'prompt_for_entailment\': "Determine whether the answer {seq1} is Contradicted or Same with the answer {seq2} for the question {context}. You need to check whether the two answers exactly describe the same thing such as the same entity, digit, or arithmetical results. If the two answers are the same, give \'Same\', otherwise give \'Contradicted\' as the result.", \'system_prompt_for_entailment\': \'You are a helpful assistant. Give short and precise answers.\', \'batch_generation\': True, \'question_max_new_tokens\': 128, \'question_temperature\': 1.0, \'generation_kwargs\': {}, \'user_prompt\': \'{question}?\', \'output_type\': \'entropy\', \'method_for_similarity\': \'semantic\'}', 'truth_values': [np.float64(-0.0), np.float64(-0.0), np.float64(-0.0), np.float64(-0.0), np.float64(-0.5004024235381879)], 'normalized_truth_values': [1.0, 1.0, 1.0, 1.0, 1.0]}}

6. Truth Methodの評価 (短文QA)

evaluate_truth_method関数を使用して、AUROC、AUPRC、Accuracy、F1、Precision、Recall、PRRなどのさまざまな評価指標でTruth Methodを評価できます。

# Truth Methodを評価する
results = ttlm.evaluate_truth_method(
    dataset='trivia_qa',
    model=api_model,
    truth_methods=truth_methods,
    eval_metrics=['auroc', 'prr', 'accuracy'], # 評価指標のリスト
    size_of_data=5,  # 実用上は1000くらいのサイズがほしいが実行に相当な時間を要する
    correctness_evaluator=model_judge,
)
print("Evaluation Results:", results)
出力例 …
Evaluation Results: {'eval_list': [{'auroc': nan, 'accuracy': np.float64(1.0), 'prr': np.float64(0.5)}, {'auroc': nan, 'accuracy': np.float64(1.0), 'prr': np.float64(0.5)}, {'auroc': nan, 'accuracy': np.float64(1.0), 'prr': np.float64(0.5)}], 'output_dict': {'previous_context': [{'role': 'system', 'content': 'You are a helpful assistant. Answer the following question in a single brief but complete sentence.'}], 'user_prompt': 'Question: {question} Answer:', 'generation': ['Hanna Glawari is the title character in the operetta "The Merry Widow" by Franz Lehár.', 'Citric acid is extracted from the juice of lemons, oranges, limes, and grapefruit.', 'Meryl Streep won the Best Supporting Actress Oscar for the 1979 film "Kramer vs. Kramer."', 'AFC Wimbledon play their home games at Kingsmeadow, also known as the Cherry Red Records Stadium.', 'Martin Clunes played the character Mr. Chips in the 2002 TV adaptation of "Goodbye, Mr. Chips".'], 'generation_correctness': [1, 1, 1, 1, 1], 'question_text': ['Hanna Glawari is the title character in which operetta by Franz Lehár?', 'What type of acid is extracted from the juice of lemons, oranges, limes, and grapefruit?', 'Meryl Streep has won 2 Oscars for Best Actress but for which 1979 film did she win Best Supporting Actress?', 'Which Football League club play their home games at Kingsmeadow, also known as the Cherry Red Records Stadium?', 'Who played the character Mr Chips in the 2002 TV adaptation of Goodbye Mr Chips'], 'ground_truths': [['Die lustige Witwe', 'The Merry Widow', 'THE MERRY WIDOW', 'Die Lustige Witwe', 'Merry widow', 'Merry Widow', 'Merry Widow Waltz'], ['E330', 'Sour salt', 'Hydrogen citrate', 'HOOCCH2(OH)C(COOH)CH2COOH', 'Citric acid', 'E 330', 'Citric Acid', '3-hydroxypentanedioic acid-3-carboxylic acid', '2-hydroxy-1,2,3-propanetricarboxylic acid', 'Citrus acid', 'ATC code A09AB04', 'ATCvet code QA09AB04', 'Acid of lemon', 'Citric (acid)'], ['Kramer v kramer', 'Kramer versus Kramer', 'Ted Kramer', 'KRAMER vs KRAMER', 'Joanna Kramer', 'Kramer v. kramer', 'Kramer vs. Kramer', 'Kramer vs Kramer', 'Kramer vs kramer', 'Kramer Vs. Kramer', 'Kramer v. Kramer', 'Kramer vs. kramer'], ['A. F. C. Wimbledon', 'AFC Wimbledon', 'List of AFC Wimbledon managers', 'A.F.C. Wimbledon', 'Afc wimbledon', 'The Dons Trust', 'AFCW', 'A F C Wimbledon', 'AFC WIMBLEDON', 'The dons trust'], ['Philippa Braithwaite', 'Lucy Aston', 'Martin Clunes']], 'truth_methods': ['LARS', 'Confidence', 'SelfDetection'], 'truth_method_0': {'name': 'LARS with {\'normalizer\': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6b3a2c710>, \'ue_type\': \'confidence\', \'number_of_generations\': 0, \'batch_generation\': True, \'lars_model\': RobertaForSequenceClassification(\n  (roberta): RobertaModel(\n    (embeddings): RobertaEmbeddings(\n      (word_embeddings): Embedding(50273, 768, padding_idx=1)\n      (position_embeddings): Embedding(514, 768, padding_idx=1)\n      (token_type_embeddings): Embedding(1, 768)\n      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): RobertaEncoder(\n      (layer): ModuleList(\n        (0-11): 12 x RobertaLayer(\n          (attention): RobertaAttention(\n            (self): RobertaSdpaSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): RobertaSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): RobertaIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output): RobertaOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n    )\n  )\n  (classifier): RobertaClassificationHead(\n    (dense): Linear(in_features=768, out_features=768, bias=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n    (out_proj): Linear(in_features=768, out_features=1, bias=True)\n  )\n), \'lars_tokenizer\': RobertaTokenizerFast(name_or_path=\'duygunuryldz/LARS\', vocab_size=50265, model_max_length=512, is_fast=True, padding_side=\'right\', truncation_side=\'right\', special_tokens={\'bos_token\': \'<s>\', \'eos_token\': \'</s>\', \'unk_token\': \'<unk>\', \'sep_token\': \'</s>\', \'pad_token\': \'<pad>\', \'cls_token\': \'<s>\', \'mask_token\': \'<mask>\', \'additional_special_tokens\': [\'[prob_token_0]\', \'[prob_token_1]\', \'[prob_token_2]\', \'[prob_token_3]\', \'[prob_token_4]\', \'[prob_token_5]\', \'[prob_token_6]\', \'[prob_token_7]\']}, clean_up_tokenization_spaces=True, added_tokens_decoder={\n\t0: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t50264: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),\n\t50265: AddedToken("[prob_token_0]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50266: AddedToken("[prob_token_1]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50267: AddedToken("[prob_token_2]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50268: AddedToken("[prob_token_3]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50269: AddedToken("[prob_token_4]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50270: AddedToken("[prob_token_5]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50271: AddedToken("[prob_token_6]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50272: AddedToken("[prob_token_7]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n}\n), \'device\': \'cuda\', \'number_of_bins\': 8, \'edges\': [6.197085156145477e-08, 0.48822552817208426, 0.8300329021045139, 0.9682016628129141, 0.9969601631164551, 0.9998409748077393, 0.9999935626983643, 1.0], \'model_for_entailment\': None, \'tokenizer_for_entailment\': None}', 'truth_values': [0.9956139326095581, 0.9314244389533997, 0.9946297407150269, 0.9754387140274048, 0.9439488053321838], 'normalized_truth_values': [1.0, 1.0, 1.0, 1.0, 1.0]}, 'truth_method_1': {'name': "Confidence with {'normalizer': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6b23d1ac0>, 'scoring_function': <TruthTorchLM.scoring_methods.length_normalized_scoring.LengthNormalizedScoring object at 0x7ff6ee7fd6a0>}", 'truth_values': [-0.0049440693414889774, -0.012483988724128673, -0.11414005253523385, -0.013905867143561425, -0.16729742338490497], 'normalized_truth_values': [1.0, 1.0, 1.0, 1.0, 1.0]}, 'truth_method_2': {'name': 'SelfDetection with {\'normalizer\': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6a781ed20>, \'tokenizer_for_entailment\': DebertaTokenizer(name_or_path=\'microsoft/deberta-large-mnli\', vocab_size=50265, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side=\'right\', truncation_side=\'right\', special_tokens={\'bos_token\': \'[CLS]\', \'eos_token\': \'[SEP]\', \'unk_token\': \'[UNK]\', \'sep_token\': \'[SEP]\', \'pad_token\': \'[PAD]\', \'cls_token\': \'[CLS]\', \'mask_token\': \'[MASK]\'}, clean_up_tokenization_spaces=False, added_tokens_decoder={\n\t0: AddedToken("[PAD]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t1: AddedToken("[CLS]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t2: AddedToken("[SEP]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t3: AddedToken("[UNK]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50264: AddedToken("[MASK]", rstrip=False, lstrip=True, single_word=False, normalized=True, special=True),\n}\n), \'model_for_entailment\': DebertaForSequenceClassification(\n  (deberta): DebertaModel(\n    (embeddings): DebertaEmbeddings(\n      (word_embeddings): Embedding(50265, 1024, padding_idx=0)\n      (LayerNorm): DebertaLayerNorm()\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): DebertaEncoder(\n      (layer): ModuleList(\n        (0-23): 24 x DebertaLayer(\n          (attention): DebertaAttention(\n            (self): DisentangledSelfAttention(\n              (in_proj): Linear(in_features=1024, out_features=3072, bias=False)\n              (pos_dropout): Dropout(p=0.1, inplace=False)\n              (pos_proj): Linear(in_features=1024, out_features=1024, bias=False)\n              (pos_q_proj): Linear(in_features=1024, out_features=1024, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): DebertaSelfOutput(\n              (dense): Linear(in_features=1024, out_features=1024, bias=True)\n              (LayerNorm): DebertaLayerNorm()\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): DebertaIntermediate(\n            (dense): Linear(in_features=1024, out_features=4096, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output): DebertaOutput(\n            (dense): Linear(in_features=4096, out_features=1024, bias=True)\n            (LayerNorm): DebertaLayerNorm()\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n      (rel_embeddings): Embedding(1024, 1024)\n    )\n  )\n  (pooler): ContextPooler(\n    (dense): Linear(in_features=1024, out_features=1024, bias=True)\n    (dropout): Dropout(p=0, inplace=False)\n  )\n  (classifier): Linear(in_features=1024, out_features=3, bias=True)\n  (dropout): Dropout(p=0.1, inplace=False)\n), \'number_of_questions\': 5, \'prompt_for_generating_question\': \'Given a question, paraphrase it to have different words and expressions but have the same meaning as the original question. Please note that you should not answer the question, but rather provide a re-phrased. These paraphrased questions should be different from each other. Previous paraphrased questions: {previous_questions}. Only output a single paraphrased question, nothing else. Question: {question}\', \'system_prompt\': \'You are a helpful assistant. Give short and precise answers.\', \'prompt_for_entailment\': "Determine whether the answer {seq1} is Contradicted or Same with the answer {seq2} for the question {context}. You need to check whether the two answers exactly describe the same thing such as the same entity, digit, or arithmetical results. If the two answers are the same, give \'Same\', otherwise give \'Contradicted\' as the result.", \'system_prompt_for_entailment\': \'You are a helpful assistant. Give short and precise answers.\', \'batch_generation\': True, \'question_max_new_tokens\': 128, \'question_temperature\': 1.0, \'generation_kwargs\': {}, \'user_prompt\': \'{question}?\', \'output_type\': \'entropy\', \'method_for_similarity\': \'semantic\'}', 'truth_values': [np.float64(-0.0), np.float64(-0.0), np.float64(-0.0), np.float64(-0.0), np.float64(-0.0)], 'normalized_truth_values': [1.0, 1.0, 1.0, 1.0, 1.0]}}}

7. 長文生成における真実性評価

長文テキスト全体に単一の真実性値を割り当てることは実用的ではありません。TruthTorchLMは、生成されたテキストを短い単一文の主張（claim）に分解し、これらの主張に「主張チェックメソッド（claim check methods）」を使用して真実性値を割り当てます。

import TruthTorchLM.long_form_generation as LFG
from transformers import DebertaForSequenceClassification, DebertaTokenizer

# テキストを主張に分解するメソッドを定義する
# APIモデルを使用するか、ローカルのHuggingfaceモデルを使用できる
decomposition_method = LFG.decomposition_methods.StructuredDecompositionAPI(model="gpt-4o-mini", decomposition_depth=1)
# decomposition_method = LFG.decomposition_methods.StructuredDecompositionLocal(model, tokenizer, decomposition_depth=1) # HFモデルを使用する場合

# いくつかのTruth MethodやClaim Check Methodで使用される合意モデルを定義する
model_for_entailment = DebertaForSequenceClassification.from_pretrained('microsoft/deberta-large-mnli').to('cuda:0')
tokenizer_for_entailment = DebertaTokenizer.from_pretrained('microsoft/deberta-large-mnli')

# Claim Check Methodを定義する (Truth Methodを利用するClaim Check Method)
qa_generation = LFG.claim_check_methods.QuestionAnswerGeneration(
    model="gpt-4o-mini",
    tokenizer=None,
    num_questions=2,
    max_answer_trials=2,
    truth_methods=[confidence, lars], # ここで短文用のTruth Methodを適用できる
    seed=0,
    entailment_model=model_for_entailment,
    entailment_tokenizer=tokenizer_for_entailment
)

# 特定の目的のために設計されたClaim Check Method
ac_entailment = LFG.claim_check_methods.AnswerClaimEntailment(
    model="gpt-4o-mini",
    tokenizer=None,
    num_questions=3,
    num_answers_per_question=2,
    entailment_model=model_for_entailment,
    entailment_tokenizer=tokenizer_for_entailment
)

# チャット履歴を定義する
chat = [{"role": "system", "content": 'You are a helpful assistant. Give brief and precise answers.'},
        {"role": "user", "content": f'Who is Ryan Reynolds?'}]

# 長文生成と真実性値の付与を行う (Huggingfaceモデルの場合)
# output_hf_model_long = LFG.long_form_generation_with_truth_value(
#    model=model,
#    tokenizer=tokenizer,
#    messages=chat,
#    decomp_method=decomposition_method,
#    claim_check_methods=[qa_generation, ac_entailment],
#    generation_seed=0
# )
# print("Long-Form Huggingface Model Output:", output_hf_model_long)

# 長文生成と真実性値の付与を行う (APIモデル)
output_api_model_long = LFG.long_form_generation_with_truth_value(
    model="gpt-4o-mini",
    messages=chat,
    decomp_method=decomposition_method,
    claim_check_methods=[qa_generation, ac_entailment],
    generation_seed=0,
    seed=0
)
print("Long-Form API Model Output:", output_api_model_long)
出力例 …
Long-Form API Model Output: {'generated_text': 'Ryan Reynolds is a Canadian-American actor, producer, and entrepreneur, known for his roles in films such as "Deadpool," "The Proposal," and "Detective Pikachu." He has gained acclaim for his comedic talent and charm and is involved in various business ventures, including the spirits company Aviation Gin.', 'claims': ['Ryan Reynolds is a Canadian-American actor.', 'Ryan Reynolds is a producer.', 'Ryan Reynolds is an entrepreneur.', "Ryan Reynolds is known for his role in the film 'Deadpool.'", "Ryan Reynolds is known for his role in the film 'The Proposal.'", "Ryan Reynolds is known for his role in the film 'Detective Pikachu.'", 'Ryan Reynolds has gained acclaim for his comedic talent.', 'Ryan Reynolds has gained acclaim for his charm.', 'Ryan Reynolds is involved in various business ventures.', 'Ryan Reynolds is involved in the spirits company Aviation Gin.'], 'normalized_truth_values': [[[np.int64(0), np.int64(0)], [np.float64(0.9220338983050848), np.float64(0.9763313609467456)], [np.int64(0), np.int64(0)], [np.float64(0.9887640449438202), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.int64(0), np.int64(0)], [np.float64(0.9220338983050848), np.float64(0.9763313609467456)], [np.float64(0.8055555555555556), np.float64(0.8297872340425532)], [np.float64(0.8243243243243243), np.float64(0.8297872340425532)], [np.float64(0.9691119691119691), np.float64(0.9489795918367347)]], [0.7310585786300049, 0.6970592839654074, 0.7310585786300049, 0.7310585786300049, 0.5825702064623147, 0.5, 0.5, 0.6224593312018546, 0.7310585786300049, 0.7310585786300049]], 'unnormalized_truth_values': [[[np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.05209214542482085), np.float64(0.99284428358078)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.003418909216793509), np.float64(0.995142936706543)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.05522248082532651), np.float64(0.9909861087799072)], [np.float64(-0.2631640745980274), np.float64(0.9273071885108948)], [np.float64(-0.2054076709699011), np.float64(0.925814688205719)], [np.float64(-0.01688575710081371), np.float64(0.9875125288963318)]], [1.0, 0.8333333333333334, 1.0, 1.0, 0.3333333333333333, 0.0, 0.0, 0.5, 1.0, 1.0]], 'claim_check_method_details': [{'claim_check_method_name': 'QuestionAnswerGeneration', 'details': [{'claim': 'Ryan Reynolds is a Canadian-American actor.', 'normalized_truth_values': [np.int64(0), np.int64(0)], 'truth_values': [np.float64(-10000000000.0), np.float64(-10000000000.0)], 'questions': [], 'answers': [], 'non_entailment_tuples': [("What is Ryan Reynolds' nationality?", 'Ryan Reynolds is Canadian.'), ("What is Ryan Reynolds' nationality?", 'Ryan Reynolds is Canadian.')], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{}]}]}, {'claim': 'Ryan Reynolds is a producer.', 'normalized_truth_values': [np.float64(0.9220338983050848), np.float64(0.9763313609467456)], 'truth_values': [np.float64(-0.05209214542482085), np.float64(0.99284428358078)], 'questions': ["What is Ryan Reynolds' role in the film industry?", 'What role does Ryan Reynolds have in addition to being an actor?'], 'answers': ['Ryan Reynolds is an actor and producer in the film industry.', 'Ryan Reynolds is also a producer and entrepreneur.'], 'non_entailment_tuples': [], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{'truth_value': -0.07031655229852163, 'normalized_truth_value': 0.9220338983050848}, {'truth_value': -0.05209214542482085, 'normalized_truth_value': 0.9220338983050848}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{'truth_value': 0.99284428358078, 'normalized_truth_value': 0.9763313609467456}, {'truth_value': 0.9391509294509888, 'normalized_truth_value': 0.8297872340425532}]}]}, {'claim': 'Ryan Reynolds is an entrepreneur.', 'normalized_truth_values': [np.int64(0), np.int64(0)], 'truth_values': [np.float64(-10000000000.0), np.float64(-10000000000.0)], 'questions': [], 'answers': [], 'non_entailment_tuples': [("What is one of Ryan Reynolds' roles outside of acting?", 'Ryan Reynolds is known for being a co-owner of the aviation gin brand.'), ("What is one of Ryan Reynolds' roles outside of acting?", 'Ryan Reynolds is a co-owner of the Aviation American Gin brand.')], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{}]}]}, {'claim': "Ryan Reynolds is known for his role in the film 'Deadpool.'", 'normalized_truth_values': [np.float64(0.9887640449438202), np.float64(1.0)], 'truth_values': [np.float64(-0.003418909216793509), np.float64(0.995142936706543)], 'questions': ["What role is Ryan Reynolds famously known for in the film 'Deadpool'?", 'What film is Ryan Reynolds best known for?'], 'answers': ["Ryan Reynolds is famously known for the role of Deadpool in the film 'Deadpool'.", 'Ryan Reynolds is best known for "Deadpool."'], 'non_entailment_tuples': [], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{'truth_value': -0.06472894202420004, 'normalized_truth_value': 0.9220338983050848}, {'truth_value': -0.003418909216793509, 'normalized_truth_value': 0.9887640449438202}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{'truth_value': 0.9808505773544312, 'normalized_truth_value': 0.9489795918367347}, {'truth_value': 0.995142936706543, 'normalized_truth_value': 1.0}]}]}, {'claim': "Ryan Reynolds is known for his role in the film 'The Proposal.'", 'normalized_truth_values': [np.int64(0), np.int64(0)], 'truth_values': [np.float64(-10000000000.0), np.float64(-10000000000.0)], 'questions': [], 'answers': [], 'non_entailment_tuples': [('What film is Ryan Reynolds known for his role in?', 'Ryan Reynolds is known for his role in "Deadpool."'), ('What film is Ryan Reynolds known for his role in?', 'Ryan Reynolds is known for his role in "Deadpool."')], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{}]}]}, {'claim': "Ryan Reynolds is known for his role in the film 'Detective Pikachu.'", 'normalized_truth_values': [np.int64(0), np.int64(0)], 'truth_values': [np.float64(-10000000000.0), np.float64(-10000000000.0)], 'questions': [], 'answers': [], 'non_entailment_tuples': [('What film is Ryan Reynolds known for starring in?', 'Ryan Reynolds is known for starring in "Deadpool."'), ('What film is Ryan Reynolds known for starring in?', 'Ryan Reynolds is known for starring in "Deadpool."'), ('What film is Ryan Reynolds known for his role in?', 'Ryan Reynolds is known for his role in "Deadpool."'), ('What film is Ryan Reynolds known for his role in?', 'Ryan Reynolds is known for his role in "Deadpool."')], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{}]}]}, {'claim': 'Ryan Reynolds has gained acclaim for his comedic talent.', 'normalized_truth_values': [np.float64(0.9220338983050848), np.float64(0.9763313609467456)], 'truth_values': [np.float64(-0.05522248082532651), np.float64(0.9909861087799072)], 'questions': ['What is Ryan Reynolds known for in his acting career?', 'What talent has Ryan Reynolds gained acclaim for?'], 'answers': ['Ryan Reynolds is known for his roles in comedy and action films, particularly as Deadpool in the Marvel franchise.', 'Ryan Reynolds has gained acclaim for his acting, particularly in comedic roles.'], 'non_entailment_tuples': [('What is Ryan Reynolds known for in his acting career?', 'Ryan Reynolds is known for his role as Deadpool in the Marvel film series.'), ('What talent has Ryan Reynolds gained acclaim for?', 'Ryan Reynolds has gained acclaim for his acting, particularly in comedy and superhero roles.')], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{'truth_value': -0.16287591693432546, 'normalized_truth_value': 0.8571428571428571}, {'truth_value': -0.05522248082532651, 'normalized_truth_value': 0.9220338983050848}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{'truth_value': 0.9736630916595459, 'normalized_truth_value': 0.8666666666666667}, {'truth_value': 0.9909861087799072, 'normalized_truth_value': 0.9763313609467456}]}]}, {'claim': 'Ryan Reynolds has gained acclaim for his charm.', 'normalized_truth_values': [np.float64(0.8055555555555556), np.float64(0.8297872340425532)], 'truth_values': [np.float64(-0.2631640745980274), np.float64(0.9273071885108948)], 'questions': ['What is Ryan Reynolds known for in his public persona?', "What qualities have contributed to Ryan Reynolds' acclaim?"], 'answers': ['Ryan Reynolds is known for his charming humor and wit.', "Ryan Reynolds' acclaim stems from his charm, comedic talent, versatility in roles, and strong social media presence."], 'non_entailment_tuples': [], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{'truth_value': -0.2631640745980274, 'normalized_truth_value': 0.8055555555555556}, {'truth_value': -0.4084716224473775, 'normalized_truth_value': 0.7764705882352941}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{'truth_value': 0.7806310057640076, 'normalized_truth_value': 0.8297872340425532}, {'truth_value': 0.9273071885108948, 'normalized_truth_value': 0.8297872340425532}]}]}, {'claim': 'Ryan Reynolds is involved in various business ventures.', 'normalized_truth_values': [np.float64(0.8243243243243243), np.float64(0.8297872340425532)], 'truth_values': [np.float64(-0.2054076709699011), np.float64(0.925814688205719)], 'questions': ['What business ventures is Ryan Reynolds involved in?', 'What types of business ventures is Ryan Reynolds involved in?'], 'answers': ['Ryan Reynolds is involved in business ventures such as Aviation American Gin and Mint Mobile.', 'Ryan Reynolds is involved in ventures such as aviation gin, a mobile phone company called Mint Mobile, and investments in various technology and entertainment businesses.'], 'non_entailment_tuples': [], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{'truth_value': -0.2054076709699011, 'normalized_truth_value': 0.8243243243243243}, {'truth_value': -0.5844489765475421, 'normalized_truth_value': 0.7407407407407407}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{'truth_value': 0.925814688205719, 'normalized_truth_value': 0.8297872340425532}, {'truth_value': 0.8245497345924377, 'normalized_truth_value': 0.8297872340425532}]}]}, {'claim': 'Ryan Reynolds is involved in the spirits company Aviation Gin.', 'normalized_truth_values': [np.float64(0.9691119691119691), np.float64(0.9489795918367347)], 'truth_values': [np.float64(-0.01688575710081371), np.float64(0.9875125288963318)], 'questions': ['What company is Ryan Reynolds involved with in the spirits industry?'], 'answers': ['Ryan Reynolds is involved with Aviation American Gin.'], 'non_entailment_tuples': [], 'truth_method_spec_outputs': [{'Truth method name': 'Confidence', 'detailed_outputs': [{'truth_value': -0.01688575710081371, 'normalized_truth_value': 0.9691119691119691}]}, {'Truth method name': 'LARS', 'detailed_outputs': [{'truth_value': 0.9875125288963318, 'normalized_truth_value': 0.9489795918367347}]}]}]}, {'claim_check_method_name': 'AnswerClaimEntailment', 'details': [{'claim': 'Ryan Reynolds is a Canadian-American actor.', 'normalized_truth_values': 0.7310585786300049, 'truth_values': 1.0, 'questions': ["What is Ryan Reynolds' nationality and profession?"], 'answers': ['Ryan Reynolds is Canadian and a professional actor.', 'Ryan Reynolds is Canadian and an actor.'], 'entailment': [1, 1]}, {'claim': 'Ryan Reynolds is a producer.', 'normalized_truth_values': 0.6970592839654074, 'truth_values': 0.8333333333333334, 'questions': ['What role does Ryan Reynolds have in addition to being an actor?', 'What role does Ryan Reynolds have as a producer?', "What is Ryan Reynolds' role in film production?"], 'answers': ['Ryan Reynolds is also a producer and entrepreneur.', 'Ryan Reynolds is also a businessman and entrepreneur.', 'Ryan Reynolds is a producer on various film and television projects, often involved in creative decisions and project development.', 'Ryan Reynolds serves as a producer on various film and television projects, often involved in developing and overseeing production.', 'Ryan Reynolds is a producer and actor in film production.', 'Ryan Reynolds is a producer and actor in film production.'], 'entailment': [1, 0, 1, 1, 1, 1]}, {'claim': 'Ryan Reynolds is an entrepreneur.', 'normalized_truth_values': 0.7310585786300049, 'truth_values': 1.0, 'questions': ['What business ventures is Ryan Reynolds involved in as an entrepreneur?', 'What is Ryan Reynolds known for in the business world as an entrepreneur?', "What is Ryan Reynolds' role as an entrepreneur?"], 'answers': ['Ryan Reynolds is involved in business ventures such as Aviation American Gin and Mint Mobile.', 'Ryan Reynolds is involved in business ventures such as Aviation American Gin, Mint Mobile, and the Maximum Effort marketing agency.', 'Ryan Reynolds is known for co-owning the aviation gin brand.', 'Ryan Reynolds is known for his successful investments, particularly in the aviation gin brand and Mint Mobile.', 'Ryan Reynolds is an entrepreneur known for investing in and founding various businesses, including Aviation American Gin and Mint Mobile.', 'Ryan Reynolds is a co-owner and investor in multiple businesses, including Aviation American Gin and Mint Mobile.'], 'entailment': [1, 1, 1, 1, 1, 1]}, {'claim': "Ryan Reynolds is known for his role in the film 'Deadpool.'", 'normalized_truth_values': 0.7310585786300049, 'truth_values': 1.0, 'questions': ['What is Ryan Reynolds best known for?', 'What role is Ryan Reynolds best known for in film?', 'What film is Ryan Reynolds best known for?'], 'answers': ['Ryan Reynolds is best known for his role as Deadpool in the Marvel film franchise.', 'Ryan Reynolds is best known for his role as Deadpool in the Marvel film series.', 'Ryan Reynolds is best known for his role as Deadpool in the film series.', 'Ryan Reynolds is best known for his role as Deadpool in the "Deadpool" films.', 'Ryan Reynolds is best known for his role in "Deadpool."', 'Ryan Reynolds is best known for "Deadpool."'], 'entailment': [1, 1, 1, 1, 1, 1]}, {'claim': "Ryan Reynolds is known for his role in the film 'The Proposal.'", 'normalized_truth_values': 0.5825702064623147, 'truth_values': 0.3333333333333333, 'questions': ['What is Ryan Reynolds best known for?', "What role is Ryan Reynolds known for in the film 'The Proposal'?", "What role does Ryan Reynolds play in the film 'The Proposal'?"], 'answers': ['Ryan Reynolds is best known for his role as Deadpool in the Marvel film franchise.', 'Ryan Reynolds is best known for his role as Deadpool in the Marvel film franchise.', "Ryan Reynolds is known for his role as Andrew Paxton in the film 'The Proposal'.", "Ryan Reynolds is known for the role of Andrew Paxton in the film 'The Proposal'.", "Ryan Reynolds plays the role of Andrew Paxton in the film 'The Proposal'.", "Ryan Reynolds plays the role of Andrew Paxton in the film 'The Proposal.'"], 'entailment': [0, 0, 1, 1, 0, 0]}, {'claim': "Ryan Reynolds is known for his role in the film 'Detective Pikachu.'", 'normalized_truth_values': 0.5, 'truth_values': 0.0, 'questions': ['What film is Ryan Reynolds known for his role in?'], 'answers': ['Ryan Reynolds is known for his role in "Deadpool."', 'Ryan Reynolds is known for his role in "Deadpool."'], 'entailment': [0, 0]}, {'claim': 'Ryan Reynolds has gained acclaim for his comedic talent.', 'normalized_truth_values': 0.5, 'truth_values': 0.0, 'questions': ['What is Ryan Reynolds known for in the entertainment industry?', 'What talent has Ryan Reynolds gained acclaim for in the entertainment industry?', 'What type of talent has Ryan Reynolds gained acclaim for in the entertainment industry?'], 'answers': ['Ryan Reynolds is known for his roles in films such as "Deadpool" and "The Proposal."', 'Ryan Reynolds is known for his roles in films, particularly as Deadpool in the Marvel franchise.', 'Ryan Reynolds has gained acclaim for his acting, particularly in comedy and action films.', 'Ryan Reynolds has gained acclaim for his acting, particularly in comedic and action roles.', 'Ryan Reynolds has gained acclaim for his acting talent, particularly in comedy and action roles.', 'Ryan Reynolds has gained acclaim for his acting talent.'], 'entailment': [0, 0, 0, 0, 0, 0]}, {'claim': 'Ryan Reynolds has gained acclaim for his charm.', 'normalized_truth_values': 0.6224593312018546, 'truth_values': 0.5, 'questions': ["What qualities have contributed to Ryan Reynolds' acclaim?", "What quality has contributed to Ryan Reynolds' acclaim?"], 'answers': ["Ryan Reynolds' acclaim is attributed to his charisma, comedic talent, versatility, and strong work ethic.", "Ryan Reynolds' acclaim can be attributed to his versatility, comedic timing, and charisma.", "Ryan Reynolds' versatility in both comedic and dramatic roles has contributed to his acclaim.", "Ryan Reynolds' versatility in both comedic and dramatic roles has contributed to his acclaim."], 'entailment': [1, 1, 0, 0]}, {'claim': 'Ryan Reynolds is involved in various business ventures.', 'normalized_truth_values': 0.7310585786300049, 'truth_values': 1.0, 'questions': ['What types of business ventures is Ryan Reynolds involved in?', 'What business ventures is Ryan Reynolds involved in?'], 'answers': ['Ryan Reynolds is involved in ventures such as the aviation gin brand, Mint Mobile, and his production company Maximum Effort.', 'Ryan Reynolds is involved in various business ventures including aviation, alcohol, and media production.', 'Ryan Reynolds is involved in ventures such as Aviation American Gin, Mint Mobile, and various film and television productions.', 'Ryan Reynolds is involved in ventures such as Aviation Gin and Mint Mobile.'], 'entailment': [1, 1, 1, 1]}, {'claim': 'Ryan Reynolds is involved in the spirits company Aviation Gin.', 'normalized_truth_values': 0.7310585786300049, 'truth_values': 1.0, 'questions': ['What company is Ryan Reynolds associated with in the spirits industry?', 'Which spirits company is Ryan Reynolds associated with?'], 'answers': ['Ryan Reynolds is associated with Aviation American Gin.', 'Ryan Reynolds is associated with Aviation American Gin.', 'Ryan Reynolds is associated with Aviation American Gin.', 'Ryan Reynolds is associated with Aviation American Gin.'], 'entailment': [1, 1, 1, 1]}]}]}

8. 長文生成におけるTruth Methodの評価

長文生成におけるTruth Methodの評価は、evaluate_truth_method_long_form関数を使用して行われます。各主張の正確性をGoogle検索で評価し、ラベルを割り当てます。

# 主張にラベルを割り当てるSAFEオブジェクトを作成する
safe = LFG.ClaimEvaluator(rater='gpt-4o-mini', tokenizer=None, max_steps=5, max_retries=10, num_searches=3)

# 評価指標を定義する
sample_level_eval_metrics = ['f1'] # 質問の主張全体で指標を計算し、平均をとる
dataset_level_eval_metrics = ['auroc', 'prr'] # データセット全体の主張で指標を計算する

# 長文生成のTruth Methodを評価する
results_long_form = LFG.evaluate_truth_method_long_form(
    dataset='longfact_objects', # 長文ファクト評価用のデータセット
    model='gpt-4o-mini',
    tokenizer=None,
    sample_level_eval_metrics=sample_level_eval_metrics,
    dataset_level_eval_metrics=dataset_level_eval_metrics,
    decomp_method=decomposition_method,
    claim_check_methods=[qa_generation],
    claim_evaluator=safe,
    size_of_data=3,
    previous_context=[{'role': 'system', 'content': 'You are a helpful assistant. Give precise answers.'}],
    user_prompt="Question: {question}",
    seed=41,
    return_method_details=False,
    return_calim_eval_details=False,
    wandb_run=None,
    add_generation_prompt=True,
    continue_final_message=False
)
print("Long-Form Evaluation Results:", results_long_form)
出力例 …
Long-Form Evaluation Results: {'sample_level_eval_list': {'claim_check_methods_0_truth_method_0': {'f1': {'values': [0.8181818181818182, 0.8, 0.8695652173913043], 'mean': np.float64(0.8292490118577075), 'max': np.float64(0.8695652173913043), 'min': np.float64(0.8), 'std': np.float64(0.029458354214830004)}}, 'claim_check_methods_0_truth_method_1': {'f1': {'values': [0.8181818181818182, 0.8, 0.8695652173913043], 'mean': np.float64(0.8292490118577075), 'max': np.float64(0.8695652173913043), 'min': np.float64(0.8), 'std': np.float64(0.029458354214830004)}}}, 'dataset_level_eval_list': {'claim_check_methods_0_truth_method_0': {'auroc': 0.7426470588235294, 'prr': np.float64(0.6608616036163603)}, 'claim_check_methods_0_truth_method_1': {'auroc': 0.5808823529411765, 'prr': np.float64(0.27191222357499767)}}, 'output_dict': {'previous_context': [{'role': 'system', 'content': 'You are a helpful assistant. Give precise answers.'}], 'user_prompt': 'Question: {question}', 'generation': ['Danica Patrick is a retired American race car driver and automotive ambassador, known for her groundbreaking achievements in motorsport. Born on March 25, 1982, in Beloit, Wisconsin, she gained prominence as one of the first women to compete full-time in both the IndyCar Series and NASCAR. Patrick became the first woman to win an IndyCar Series race in 2008 and is also known for her strong performances in various competitions, including a notable third-place finish in the 2009 Indianapolis 500. After retiring from racing, she transitioned into a role as an entrepreneur and media personality, including work in television and sports commentary.', 'Max Weber was a German sociologist, philosopher, and political economist, regarded as one of the founding figures of modern sociology. He was born on April 21, 1864, and died on June 14, 1920. Weber is best known for his analysis of the relationship between religion and economic behavior, particularly in his work "The Protestant Ethic and the Spirit of Capitalism." He also contributed significantly to the development of concepts such as bureaucracy, authority, and social action. His ideas have had a lasting impact on social theory and research methods in the social sciences.', 'IBM Watson is an artificial intelligence system developed by IBM that uses natural language processing and machine learning to analyze large amounts of data, understand complex questions, and provide answers. Initially famous for its performance on the quiz show "Jeopardy!" in 2011, Watson has since evolved to assist in various fields, including healthcare, customer service, and business analytics. It provides tools and APIs for building AI applications, enabling organizations to leverage data for improved decision-making and insights.'], 'claims': [['Danica Patrick is a retired American race car driver.', 'Danica Patrick is an automotive ambassador.', 'Danica Patrick is known for her groundbreaking achievements in motorsport.', 'Danica Patrick was born on March 25, 1982.', 'Danica Patrick was born in Beloit, Wisconsin.', 'Danica Patrick gained prominence as one of the first women to compete full-time in the IndyCar Series.', 'Danica Patrick gained prominence as one of the first women to compete full-time in NASCAR.', 'Danica Patrick became the first woman to win an IndyCar Series race in 2008.', 'Danica Patrick is known for her strong performances in various competitions.', 'Danica Patrick achieved a notable third-place finish in the 2009 Indianapolis 500.', 'After retiring from racing, Danica Patrick transitioned into a role as an entrepreneur.', 'After retiring from racing, Danica Patrick became a media personality.', 'Danica Patrick has worked in television and sports commentary.'], ['Max Weber was a German sociologist.', 'Max Weber was a German philosopher.', 'Max Weber was a political economist.', 'Max Weber is regarded as one of the founding figures of modern sociology.', 'Max Weber was born on April 21, 1864.', 'Max Weber died on June 14, 1920.', 'Max Weber is best known for his analysis of the relationship between religion and economic behavior.', "Max Weber is particularly known for his work 'The Protestant Ethic and the Spirit of Capitalism.'", 'Max Weber contributed significantly to the development of concepts such as bureaucracy, authority, and social action.', "Max Weber's ideas have had a lasting impact on social theory and research methods in the social sciences."], ['IBM Watson is an artificial intelligence system developed by IBM.', 'IBM Watson uses natural language processing to analyze large amounts of data.', 'IBM Watson uses machine learning to analyze large amounts of data.', 'IBM Watson can understand complex questions.', 'IBM Watson can provide answers.', "IBM Watson was initially famous for its performance on the quiz show 'Jeopardy!' in 2011.", 'IBM Watson has evolved to assist in various fields, including healthcare.', 'IBM Watson has evolved to assist in customer service.', 'IBM Watson has evolved to assist in business analytics.', 'IBM Watson provides tools for building AI applications.', 'IBM Watson provides APIs for building AI applications.', 'IBM Watson enables organizations to leverage data for improved decision-making.', 'IBM Watson enables organizations to gain insights.']], 'claim_correctness': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]), array([1, 0, 1, 1, 1, 1, 1, 1, 1, 1]), array([1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1])], 'question_text': ['Who is Danica Patrick?', 'Who is Max Weber?', 'What is the IBM Watson system?'], 'context': ['', '', ''], 'claim_check_methods': ['QuestionAnswerGeneration'], 'claim_check_methods_0': {'name': 'Claim Check Method by Generating Questions and Answers.\nQuestion generation model: gpt-4o-mini\nNumber of questions to be generated for a stament: 2\nNumber of trials to generate an answer that entails with the claim: 2\nEntailment check model: <class \'transformers.models.deberta.modeling_deberta.DebertaForSequenceClassification\'>\n\nQuestion generation instruction for the first claim:\n  [{\'role\': \'system\', \'content\': "You are an expert assistant skilled at generating focused and contextually relevant questions from claims. Your task is to create a question such that the answer would align closely with the provided claim. To ensure the question is precise and relevant, consider the context provided by the original question. Study the examples below from a variety of topics and follow the same pattern.\\n\\nOriginal Question: What themes are commonly explored in 20th-century dystopian literature?\\nClaim: George Orwell\'s novel 1984 explores the theme of government surveillance. \\nQuestion: What theme does George Orwell\'s novel 1984 explore?   \\n\\nOriginal Question: What themes are commonly explored in 20th-century dystopian literature?\\nClaim: George Orwell\'s novel 1984 portrays a totalitarian regime that monitors every aspect of citizens\' lives. \\nQuestion: How does George Orwell\'s novel 1984 reflect the theme of totalitarian control, as commonly explored in 20th-century dystopian literature?\\n\\nOriginal Question: What themes are commonly explored in 20th-century dystopian literature?\\nClaim: The novel 1984 is written by George Orwell.\\nQuestion: Who has written the novel 1984?\\n\\nOriginal Question: How has artificial intelligence influenced industries in the 21st century?\\nClaim: Artificial intelligence enables better decision-making through data analysis.\\nQuestion: How does artificial intelligence enhance the decision-making process in modern businesses?\\n\\nOriginal Question: What factors contributed to the Great Depression, and how did governments respond?\\nClaim: Stock market speculation contributed to the Great Depression.\\nQuestion: Did stock market speculation contribute to the Great Depression?\\n\\nOriginal Question: Who is Abraham Lincoln?\\nClaim: Abraham Lincoln is best known for leading the country through the Civil War.\\nQuestion: What is Abraham Lincoln\'s most significant historical contribution?\\n\\nOriginal Question: Who is Abraham Lincoln?\\nClaim: Abraham Lincoln served from 1861 to 1865 as the president of the US.\\nQuestion: When did Abraham Lincoln serve as the president of the United States?\\n\\nNow, follow the pattern demonstrated in the examples to generate a question for the given claim,without adding explanations, introductions, or conversational responses."}, {\'role\': \'user\', \'content\': \'Original question: {question}\\nClaim: {claim}\\nQuestion: \'}]\n\nQuestion generation instruction for claims with preceeding text:\n  [{\'role\': \'system\', \'content\': "You are an expert assistant skilled at generating focused and contextually relevant questions from claims. Your task is to create a question such that the answer would align closely with the provided claim. To ensure the question is precise and relevant, consider the context provided by the original question. Study the examples below from a variety of topics and follow the same pattern.\\n\\nOriginal Question: What themes are commonly explored in 20th-century dystopian literature?\\nClaim: George Orwell\'s novel 1984 explores the theme of government surveillance. \\nQuestion: What theme does George Orwell\'s novel 1984 explore?   \\n\\nOriginal Question: What themes are commonly explored in 20th-century dystopian literature?\\nClaim: George Orwell\'s novel 1984 portrays a totalitarian regime that monitors every aspect of citizens\' lives. \\nQuestion: How does George Orwell\'s novel 1984 reflect the theme of totalitarian control, as commonly explored in 20th-century dystopian literature?\\n\\nOriginal Question: What themes are commonly explored in 20th-century dystopian literature?\\nClaim: The novel 1984 is written by George Orwell.\\nQuestion: Who has written the novel 1984?\\n\\nOriginal Question: How has artificial intelligence influenced industries in the 21st century?\\nClaim: Artificial intelligence enables better decision-making through data analysis.\\nQuestion: How does artificial intelligence enhance the decision-making process in modern businesses?\\n\\nOriginal Question: What factors contributed to the Great Depression, and how did governments respond?\\nClaim: Stock market speculation contributed to the Great Depression.\\nQuestion: Did stock market speculation contribute to the Great Depression?\\n\\nOriginal Question: Who is Abraham Lincoln?\\nClaim: Abraham Lincoln is best known for leading the country through the Civil War.\\nQuestion: What is Abraham Lincoln\'s most significant historical contribution?\\n\\nOriginal Question: Who is Abraham Lincoln?\\nClaim: Abraham Lincoln served from 1861 to 1865 as the president of the US.\\nQuestion: When did Abraham Lincoln serve as the president of the United States?\\n\\nNow, follow the pattern demonstrated in the examples to generate a question for the given claim,without adding explanations, introductions, or conversational responses."}, {\'role\': \'user\', \'content\': \'Original question: {question}\\nClaim: {claim}\\nQuestion: \'}]\n\nAnswer generation instruction:\n    [{\'role\': \'system\', \'content\': \'You are a helpful assistant. Give a single claim answer to given question. Don\\\\‘t provide any additional information. Just answer the question with a brief sentence in a single claim.\'}, {\'role\': \'user\', \'content\': \'{question}\'}]\n\nTruth methods to assign a score the question(s):\n   [<TruthTorchLM.truth_methods.confidence.Confidence object at 0x7ff6a7ea4110>, <TruthTorchLM.truth_methods.lars.LARS object at 0x7ff6b9b0bf20>]', 'truth_methods': ['Confidence', 'LARS'], 'truth_methods_name': ["Confidence with {'normalizer': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6b23d1ac0>, 'scoring_function': <TruthTorchLM.scoring_methods.length_normalized_scoring.LengthNormalizedScoring object at 0x7ff6ee7fd6a0>}", 'LARS with {\'normalizer\': <TruthTorchLM.normalizers.isotonic_regression.IsotonicRegression object at 0x7ff6b3a2c710>, \'ue_type\': \'confidence\', \'number_of_generations\': 0, \'batch_generation\': True, \'lars_model\': RobertaForSequenceClassification(\n  (roberta): RobertaModel(\n    (embeddings): RobertaEmbeddings(\n      (word_embeddings): Embedding(50273, 768, padding_idx=1)\n      (position_embeddings): Embedding(514, 768, padding_idx=1)\n      (token_type_embeddings): Embedding(1, 768)\n      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): RobertaEncoder(\n      (layer): ModuleList(\n        (0-11): 12 x RobertaLayer(\n          (attention): RobertaAttention(\n            (self): RobertaSdpaSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): RobertaSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): RobertaIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n            (intermediate_act_fn): GELUActivation()\n          )\n          (output): RobertaOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n    )\n  )\n  (classifier): RobertaClassificationHead(\n    (dense): Linear(in_features=768, out_features=768, bias=True)\n    (dropout): Dropout(p=0.1, inplace=False)\n    (out_proj): Linear(in_features=768, out_features=1, bias=True)\n  )\n), \'lars_tokenizer\': RobertaTokenizerFast(name_or_path=\'duygunuryldz/LARS\', vocab_size=50265, model_max_length=512, is_fast=True, padding_side=\'right\', truncation_side=\'right\', special_tokens={\'bos_token\': \'<s>\', \'eos_token\': \'</s>\', \'unk_token\': \'<unk>\', \'sep_token\': \'</s>\', \'pad_token\': \'<pad>\', \'cls_token\': \'<s>\', \'mask_token\': \'<mask>\', \'additional_special_tokens\': [\'[prob_token_0]\', \'[prob_token_1]\', \'[prob_token_2]\', \'[prob_token_3]\', \'[prob_token_4]\', \'[prob_token_5]\', \'[prob_token_6]\', \'[prob_token_7]\']}, clean_up_tokenization_spaces=True, added_tokens_decoder={\n\t0: AddedToken("<s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t1: AddedToken("<pad>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t2: AddedToken("</s>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t3: AddedToken("<unk>", rstrip=False, lstrip=False, single_word=False, normalized=True, special=True),\n\t50264: AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False, special=True),\n\t50265: AddedToken("[prob_token_0]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50266: AddedToken("[prob_token_1]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50267: AddedToken("[prob_token_2]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50268: AddedToken("[prob_token_3]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50269: AddedToken("[prob_token_4]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50270: AddedToken("[prob_token_5]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50271: AddedToken("[prob_token_6]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n\t50272: AddedToken("[prob_token_7]", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),\n}\n), \'device\': \'cuda\', \'number_of_bins\': 8, \'edges\': [6.197085156145477e-08, 0.48822552817208426, 0.8300329021045139, 0.9682016628129141, 0.9969601631164551, 0.9998409748077393, 0.9999935626983643, 1.0], \'model_for_entailment\': None, \'tokenizer_for_entailment\': None}'], 'truth_values': [[[np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.13477927940546977), np.float64(0.9396624565124512)], [np.float64(-0.2587355322490888), np.float64(0.867489218711853)], [np.float64(-1.2271258579435847e-06), np.float64(0.9756694436073303)], [np.float64(-0.0019039125377896032), np.float64(0.9711687564849854)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.16384524288218433), np.float64(0.9710964560508728)], [np.float64(-0.0021771447475833097), np.float64(0.9931827187538147)], [np.float64(-0.12174047855608477), np.float64(0.9046770334243774)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.15295889579295666), np.float64(0.4658990800380707)], [np.float64(-0.24578257505822876), np.float64(0.5608422756195068)]], [[np.float64(-0.007338265075556397), np.float64(0.9924321174621582)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.39422451889486754), np.float64(0.9805521368980408)], [np.float64(-4.0801459551158376e-07), np.float64(0.9736986756324768)], [np.float64(-1.1398384138772144e-06), np.float64(0.9822611212730408)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.04104600753682556), np.float64(0.9945863485336304)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.28761962679996383), np.float64(0.8052077889442444)]], [[np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.45116458593229025), np.float64(0.9013420343399048)], [np.float64(-0.19805144095101324), np.float64(0.9866961240768433)], [np.float64(-0.3208595231661687), np.float64(0.9800649285316467)], [np.float64(-0.2317622641851102), np.float64(0.9547934532165527)], [np.float64(-0.18177153621315623), np.float64(0.9849340915679932)], [np.float64(-0.0640968275823397), np.float64(0.9904605746269226)], [np.float64(-0.513157139782237), np.float64(0.977510929107666)], [np.float64(-0.7552012589315726), np.float64(0.918754518032074)], [np.float64(-0.13085216040589512), np.float64(0.9905827045440674)], [np.float64(-10000000000.0), np.float64(-10000000000.0)], [np.float64(-0.05997878056184547), np.float64(0.9863292574882507)], [np.float64(-0.1785628160822161), np.float64(0.9543466567993164)]]], 'normalized_truth_values': [[[np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)]], [[np.float64(1.0), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)]], [[np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)], [np.int64(0), np.int64(0)], [np.float64(1.0), np.float64(1.0)], [np.float64(1.0), np.float64(1.0)]]]}}}

おわりに

本記事では、大規模言語モデル（LLM）のハルシネーション問題に対処するための強力なオープンソースライブラリ「TruthTorchLM」について、その詳細と具体的な使用方法を解説しました。TruthTorchLMは、多様な真実性予測手法とシームレスな互換性を提供することで、LLMの信頼性を飛躍的に向上させる可能性を秘めています。

高リスクなドメインにおけるLLMの活用が広がるにつれて、その出力の正確性を保証する技術の重要性はますます高まります。TruthTorchLMは、研究者や開発者がLLMの「真実性」を探求し、より堅牢で信頼性の高いAIシステムを構築するための強力なツールとなるでしょう。ぜひ本ライブラリを活用し、次世代のAI開発に貢献してください。

More Information

  • arXiv:2507.08203, Duygu Nur Yaldiz et al., 「TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs」, https://arxiv.org/abs/2507.08203

関連記事

Qwen3-TTS: 思い通りの声を創り出すAI音声合成

Qwenチーム（Alibaba Cloud）は、最新の多言語対応音声合成（TTS）モデルシリーズ「Qwen3-TTS」を公開しました。本モデルは500万時間以上という圧倒的な規模の音声データで学習され、日本語を含む10言 […]

深層学習モデルの軽量化: PyTorchによる知識蒸留の実践

近年、ディープラーニングは画像認識、自然言語処理など、様々な分野で目覚ましい成果を上げています。特に、大規模なニューラルネットワークは、大量のデータを学習し、高度なタスクをこなす能力を持っています。しかし、これらのモデル […]

OpenVINOによる深層学習モデルのパフォーマンス改善

機械学習モデルの社会実装が加速する現代において、その推論速度は、アプリケーションの応答性、ひいてはユーザー体験を左右する重要な要素となっています。リアルタイム性が求められるエッジAIの現場から、大量データを処理するクラウ […]