2 Small Subset: GPT-5 Vs Rule-Based

We analyze LLM scraping results for three randomly selected PDFs, comparing them against our current rule-based scraper and the original PDF files.

Key Takeaway

Results show promise for significant value add via improved and flexible scraping via LLM intelligence. While the LLM outperforms the rule-based scraper, it produces one singular, systematic error type: tabular gap-filling errors where the model interpolates values in blank PDF table cells rather than preserving null states. These errors appear predictable and easily identifiable, making them likely fixable through post-processing. In addition to automatic value corrections made the LLM, based on our testing we see examples when rule-based scraper misses large amounts of records, it’s not yet clear how common this is, but it is corrected by the LLM.

2.1 Comparisons

2.1.1 2025 Week 28

Total Records:

The LLM (GPT-5) extracts the same number of records as the rule-based scraper
This matches the correct count as verified against the PDF

Value Discrepancies:

9 discrepancies exist between the two datasets
In 7 cases, the rule-based scraper is actually WRONG due to errors in the PDF table itself. The LLM corrects these by leveraging contextual narrative text and identifying misplaced commas
In the remaining 2 cases, the LLM is wrong while the rule-based scraper is correct. This represents the tabular gap-filling error pattern
- Errors occur when blank cells lack placeholders (0, “-”, etc.)
- Pattern is consistent but not universal
- Aggressive prompt engineering has not resolved this issue

	Country	Event	Parameter	LLM	Baseline	Verification	pdf
0	Angola	Cholera	TotalCases	27160.0	2716.0	✅	Week_28__7_-_13_July_2025.pdf
5	Burundi	Mpox	TotalCases	4010.0	401.0	✅	Week_28__7_-_13_July_2025.pdf
6	Burundi	Mpox	CasesConfirmed	4010.0	401.0	✅	Week_28__7_-_13_July_2025.pdf
10	Democratic Republic of the Congo	Measles	TotalCases	30690.0	3069.0	✅	Week_28__7_-_13_July_2025.pdf
15	Madagascar	Malnutrition crisis	TotalCases	357900.0	3579.0	✅	Week_28__7_-_13_July_2025.pdf
20	South Sudan	Cholera	TotalCases	82890.0	8289.0	✅	Week_28__7_-_13_July_2025.pdf
21	South Sudan	Cholera	CasesConfirmed	82890.0	8289.0	✅	Week_28__7_-_13_July_2025.pdf
26	Tanzania, United Republic of	Cholera	CasesConfirmed	159.0	0.0	❌	Week_28__7_-_13_July_2025.pdf
27	Tanzania, United Republic of	Cholera	Deaths	0.0	159.0	❌	Week_28__7_-_13_July_2025.pdf

2.1.2 2025 Week 4

Total Records:

The LLM extracts 110 records (correct per PDF)
The rule-based scraper only extracts 34 (significant undercount)

Value Discrepancies:

	Country	Event	Parameter	LLM	Baseline	Verification	pdf
1	Burundi	Cholera	CasesConfirmed	12.0	0.0	❌	Week_4__20_-_26_January_2025.pdf
2	Burundi	Cholera	Deaths	0.0	12.0	❌	Week_4__20_-_26_January_2025.pdf
6	Central African Republic	Measles	CasesConfirmed	1.0	0.0	❌	Week_4__20_-_26_January_2025.pdf
7	Central African Republic	Measles	Deaths	0.0	1.0	❌	Week_4__20_-_26_January_2025.pdf
11	Chad	Measles	CasesConfirmed	20.0	0.0	❌	Week_4__20_-_26_January_2025.pdf
12	Chad	Measles	Deaths	0.0	20.0	❌	Week_4__20_-_26_January_2025.pdf
16	Comoros	Cholera	CasesConfirmed	152.0	0.0	❌	Week_4__20_-_26_January_2025.pdf
17	Comoros	Cholera	Deaths	0.0	152.0	❌	Week_4__20_-_26_January_2025.pdf
21	Comoros	Cyclone Chido	CasesConfirmed	5.0	0.0	❌	Week_4__20_-_26_January_2025.pdf
22	Comoros	Cyclone Chido	Deaths	0.0	5.0	❌	Week_4__20_-_26_January_2025.pdf

2.1.3 2020 Week 49

Total Records:

LLM extracts 117 records vs rule-based scraper’s 119
The 2 missing LLM records are “closed events” - likely a solvable via prompting, but need to decide how to properly deal with these

Value Discrepancies:

Common records show the same tabular gap-filling error pattern

	Country	Event	Parameter	LLM	Baseline	Verification	pdf
1	Nigeria	Measles	CasesConfirmed	14.0	0.0	❌	OEW49-291106122020.pdf
2	Nigeria	Measles	Deaths	0.0	14.0	❌	OEW49-291106122020.pdf

--- echo: false code-folding: false --- ## Small Subset: GPT-5 Vs Rule-Based We analyze LLM scraping results for three randomly selected PDFs, comparing them against our current rule-based scraper and the original PDF files. **Key Takeaway** Results show promise for significant value add via improved and flexible scraping via LLM intelligence. While the LLM outperforms the rule-based scraper, it produces one singular, systematic error type: **tabular gap-filling errors** where the model interpolates values in blank PDF table cells rather than preserving null states. These errors appear predictable and easily identifiable, making them likely fixable through post-processing. In addition to automatic value corrections made the LLM, based on our testing we see examples when rule-based scraper misses large amounts of records, it's not yet clear how common this is, but it is corrected by the LLM. ```{python} import os import sys from pathlib import Path # Add repository root to Python path for imports repo_root = Path(os.getcwd()).parent sys.path.insert(0, str(repo_root)) # Store original working directory original_cwd = os.getcwd() # Change working directory to repo root for file access os.chdir(str(repo_root)) import pandas as pd from src.reporting.prompt_comparison_utils import ( get_model_discrepancies, get_model_discrepancies_complete, ) from src.compare import perform_discrepancy_analysis # gpt5_v144_complete = get_model_discrepancies_complete( # "v1.4.4", "openai_gpt_5_pdf_upload" # ) # # worked worse! I don't get it! # gpt5_v146_complete = get_model_discrepancies_complete( # "v1.4.6", "openai_gpt_5_pdf_upload" # ) # gpt5_v147_complete = get_model_discrepancies_complete( # "v1.4.7", "openai_gpt_5_pdf_upload" # ) # Change back to original directory os.chdir(original_cwd) ``` ```{python} # | warning: false # | message: false # | output: false baseline_df = pd.read_csv("../data/final_data_for_powerbi_with_kpi.csv") fp_w28b = "../outputs/extraction_178_prompt_v1.4.7_model_openai_gpt_5_pdf_upload.csv" # week 28 fp_w28 = "../outputs/extraction_172_prompt_v1.4.5_model_openai_gpt_5_pdf_upload.csv" # week 28 fp_w4 = "../outputs/extraction_173_prompt_v1.4.5_model_openai_gpt_5_pdf_upload.csv" # week 4 fp_q = "../outputs/extraction_174_prompt_v1.4.5_model_openai_gpt_5_pdf_upload.csv" # week 4 gpt5_v145_w28 = pd.read_csv(fp_w28) gpt5_v145_w28 = pd.read_csv(fp_w28b) gpt5_v145_w4 = pd.read_csv(fp_w4) gpt5_v145_y2020w49 = pd.read_csv(fp_q) # Filter baseline to Week 28, 2025 baseline_week28 = baseline_df[ (baseline_df["Year"] == 2025) & (baseline_df["WeekNumber"] == 28) ].copy() baseline_week4 = baseline_df[ (baseline_df["Year"] == 2025) & (baseline_df["WeekNumber"] == 4) ].copy() baseline_week49 = baseline_df[ (baseline_df["Year"] == 2020) & (baseline_df["WeekNumber"] == 49) ].copy() discrepancies_28, llm_common_28, llm_only_28, baseline_only_28 = ( perform_discrepancy_analysis(gpt5_v145_w28, baseline_week28) ) # discrepancies_28b, llm_common_28b, llm_only_28b, baseline_only_28b = ( # perform_discrepancy_analysis(gpt5_v145_w28, baseline_week28) # ) discrepancies_4, llm_common_4, llm_only_4, baseline_only_4 = ( perform_discrepancy_analysis(gpt5_v145_w4, baseline_week4) ) discrepancies_49, llm_common_49, llm_only_49, baseline_only_49 = ( perform_discrepancy_analysis(gpt5_v145_y2020w49, baseline_week49) ) ``` ```{python} # | eval: false # | echo: false gpt5_v145_y2020w49[gpt5_v145_y2020w49["Country"].str.startswith("D")] ``` ```{python} def pivot_discrepancies_long(discrepancies_df): """ Pivot discrepancies dataframe to long format with columns: Country, Event, Parameter, LLM, Baseline, Discrepancy """ if discrepancies_df is None or len(discrepancies_df) == 0: return pd.DataFrame( columns=["Country", "Event", "Parameter", "LLM", "Baseline", "Discrepancy"] ) # Identify the parameters by finding discrepancy columns discrepancy_cols = [ col for col in discrepancies_df.columns if col.endswith("_discrepancy") ] parameters = [col.replace("_discrepancy", "") for col in discrepancy_cols] # Create the long format dataframe long_data = [] for _, row in discrepancies_df.iterrows(): country = row["Country"] event = row["Event"] for param in parameters: # Get the values for this parameter discrepancy_col = f"{param}_discrepancy" llm_col = f"llm_{param}" baseline_col = f"baseline_{param}" if all( col in discrepancies_df.columns for col in [discrepancy_col, llm_col, baseline_col] ): long_data.append( { "Country": country, "Event": event, "Parameter": param, "LLM": row[llm_col], "Baseline": row[baseline_col], "Discrepancy": row[discrepancy_col], } ) return pd.DataFrame(long_data) ``` ```{python} discrepancies_long_28 = pivot_discrepancies_long(discrepancies_28) # discrepancies_long_28b = pivot_discrepancies_long(discrepancies_28b) discrepancies_long_4 = pivot_discrepancies_long(discrepancies_4) discrepancies_long_49 = pivot_discrepancies_long(discrepancies_49) discrepancies_long_28["pdf"] = gpt5_v145_w28.SourceDocument.unique()[0] discrepancies_long_4["pdf"] = gpt5_v145_w4.SourceDocument.unique()[0] discrepancies_long_49["pdf"] = gpt5_v145_y2020w49.SourceDocument.unique()[0] # gpt5_v145_y2020w49 # discrepancies_49.sort_values(["comparison_key", "TotalCases_discrepancy"]) d28_verified = discrepancies_long_28[discrepancies_long_28.Discrepancy].copy() # d28b_verified = discrepancies_long_28b[discrepancies_long_28b.Discrepancy].copy() d4_verified = discrepancies_long_4[discrepancies_long_4.Discrepancy].copy() d49_verified = discrepancies_long_49[discrepancies_long_49.Discrepancy].copy() d49_verified["Verification"] = "❌" d4_verified["Verification"] = "❌" d28_verified["Verification"] = d28_verified["Country"].apply( lambda country: "❌" if country == "Tanzania, United Republic of" else "✅" ) ``` ## Comparisons ### 2025 Week 28 **Total Records:** - The LLM (GPT-5) extracts the same number of records as the rule-based scraper - This matches the correct count as verified against the PDF **Value Discrepancies:** - 9 discrepancies exist between the two datasets - In 7 cases, the rule-based scraper is actually WRONG due to errors in the PDF table itself. The LLM corrects these by leveraging contextual narrative text and identifying misplaced commas - In the remaining 2 cases, the LLM is wrong while the rule-based scraper is correct. This represents the **tabular gap-filling error** pattern + Errors occur when blank cells lack placeholders (0, "-", etc.) + Pattern is consistent but not universal + Aggressive prompt engineering has not resolved this issue ```{python} d28_verified[ ["Country", "Event", "Parameter", "LLM", "Baseline", "Verification", "pdf"] ] ``` ### 2025 Week 4 **Total Records:** - The LLM extracts 110 records (correct per PDF) - The rule-based scraper **only extracts 34** (significant undercount) **Value Discrepancies:** ```{python} d4_verified[["Country", "Event", "Parameter", "LLM", "Baseline", "Verification", "pdf"]] ``` ### 2020 Week 49 **Total Records:** - LLM extracts 117 records vs rule-based scraper's 119 - The 2 missing LLM records are "closed events" - likely a solvable via prompting, but need to decide how to properly deal with these **Value Discrepancies:** - Common records show the same tabular gap-filling error pattern ```{python} d49_verified[["Country", "Event", "Parameter", "LLM", "Baseline", "Verification", "pdf"]] ```