How The Core Platform Features Were Implemented

1a. Data Ingestion Agent

The ingestion layer runs one coordinated retrieval pipeline that writes dated JSON snapshots for each country and source. Those snapshots follow the pattern country_source_dataset_YYYYMMDD.json, so later report-generation runs can use a stable evidence date instead of relying on live external APIs at runtime.

GDELT event and media monitoring

GDELT is a global event and news-monitoring dataset. The agent retrieves events and gkg streams for the selected country, then uses them as narrative evidence for topic identification, section analysis, and grounded report writing.

HDX HAPI structured humanitarian indicators

HDX HAPI provides the structured datasets behind the quantitative side of the report. The pipeline fetches idps, refugees, returnees, humanitarian_needs, operational_presence, funding, conflict_events, national_risk, food_security, food_prices, poverty_rate, rainfall, and baseline_population for indicators, key figures, tables, and factual grounding.

ReliefWeb reports and disaster feeds

ReliefWeb is the project's operational document source. The ingestion run stores reports and disasters records, which are later summarised into report sections and bibliography entries so narrative claims can be tied back to humanitarian publications.

Copernicus rapid mapping activations and AOIs

Copernicus EMS Rapid Mapping publishes emergency geospatial products derived from satellite imagery. The system stores the selected activation and cached AOI layers, then reuses them in the map view so analysts can see crisis footprints and mapped areas inside the UI.

The result is a dated evidence pack: structured humanitarian records from HDX, operational reports from ReliefWeb, narrative event streams from GDELT, and geospatial activation layers from Copernicus, all serialised as reusable JSON snapshots for orchestration and admin status checks.

1b. Agent Pipeline Nodes and Language Verification

The runtime graph is defined as a LangGraph workflow using a typed WorkflowState and explicit nodes: load_data, extract_data, identify_topics, generate_indicators, analyze_sections, write_report, score_confidence, analyze_gaps, generate_bibliography, generate_dual_view, verify_language, and generate_crisis_overview_description.

The LLM-backed nodes run on Azure-hosted gpt-4o through LangChain's AzureChatOpenAI. That deployment was suitable here because the same model can alternate between strict JSON extraction, topic identification, indicator generation, and longer section synthesis without switching models mid-pipeline. The orchestrator pairs it with JSON and string parsers, retry logic, and bounded temperature settings so each node can stay predictable for its task.

def create_workflow() -> StateGraph:
    """Create LangGraph workflow for agent orchestration.
    
    Returns:
        Configured StateGraph workflow
    """
    # Create workflow
    workflow = StateGraph(WorkflowState)
    
    # Add nodes
    workflow.add_node("load_data", load_data_node)
    workflow.add_node("extract_data", data_extraction_node)
    workflow.add_node("identify_topics", topic_identification_node)
    workflow.add_node("generate_indicators", smart_indicator_node)
    workflow.add_node("analyze_sections", section_analysis_nodes)
    workflow.add_node("write_report", report_writing_node)
    workflow.add_node("score_confidence", confidence_scoring_node)
    workflow.add_node("analyze_gaps", gap_analysis_node)
    workflow.add_node("generate_bibliography", bibliography_node)
    workflow.add_node("generate_dual_view", dual_view_node)
    workflow.add_node("verify_language", language_verification_node)
    workflow.add_node("generate_crisis_overview_description", crisis_overview_description_node)

return AzureChatOpenAI(
    azure_deployment=settings.azure_openai_deployment_name,
    openai_api_version=settings.azure_openai_api_version,
    azure_endpoint=settings.azure_openai_endpoint,
    api_key=settings.azure_openai_api_key,
    temperature=temperature or settings.temperature,
    max_tokens=settings.max_tokens,
    max_retries=6,
    request_timeout=120,
)

The language verification stage applies structured rating and confidence checks and can trigger rewrite loops before output is accepted. This is important because the system does not stop at first-pass generation; it explicitly re-checks terminology and style against IFRC reporting expectations.

The most implementation-specific part of the pipeline is the section-generation stage: it does not send all raw content to one prompt. Instead, it separates structured humanitarian datasets from article-style sources, converts GDELT and ReliefWeb entries into compact excerpts, chunks content into bounded batches, performs section-level extraction, and then runs a synthesis pass to assemble the final narrative. This staged approach was chosen to keep prompts within model context limits and to make section failures easier to debug.

CHUNK_SIZE = 200
content_chunks = [all_content[i:i + CHUNK_SIZE] for i in range(0, len(all_content), CHUNK_SIZE)]

for section in narrative_sections:
    chunk_analyses = []
    for chunk_idx, chunk in enumerate(content_chunks, 1):
        chunk_response = await call_llm_with_prompt(
            system_prompt="You extract key facts from news articles and humanitarian reports, grounding analysis in structured data.",
            user_prompt=chunk_user_prompt,
            temperature=0.3
        )

        if chunk_response:
            chunk_analyses.append(chunk_response)
            if chunk_idx < len(content_chunks):
                await asyncio.sleep(5)

    if chunk_analyses:
        final_response = await call_llm_with_prompt(
            system_prompt="You are a humanitarian analyst writing comprehensive reports.",
            user_prompt=synthesis_prompt,
            temperature=0.6
        )

    await asyncio.sleep(30)

1c and 1d. Agent Workflow Integration and Database Persistence

The Django API layer reads and writes report runs and metric-report rows that are later exposed through retrieval endpoints for analyst views, operations views, references, /report/whats-new, and admin deletion workflows. Those rows carry the final report payload as JSON/JSONB-style structures such as sections, dual views, bibliography, and metadata, which become the persistence backbone used by the main pages, admin tools, and chatbot report-context retrieval.

Persistence is implemented through direct PostgreSQL access rather than only relying on the Django ORM. Ingestion-side writes use pooled psycopg2 connections, while the chatbot service opens its own asyncpg.create_pool(...) for asynchronous report retrieval. This split matters because the system has both synchronous admin/orchestrator workflows and asynchronous chat traffic, but both operate over the same report and crisis records stored in Azure PostgreSQL.