Trading Sentiment Aggregator - Project Documentation

This repository contains an advanced financial data aggregator designed to scrape, compile, and structure market sentiment and fundamental analytics from diverse online sources. The application exposes a localized API framework built on FastAPI, incorporates a persistent SQLite database, and is containerized for seamless deployment via a dedicated CI/CD pipeline.

1. Software Architecture

The project is built around an automated multi-source ingestion pattern:

• Data Scrapers: Custom modular scraping routines designed to parse live history, analysis, and fundamental metrics from financial portals.
• Database Layer: A local SQLite database (aggregator.db) orchestrated through an asynchronous SQLAlchemy object-relational mapping architecture.
• API Framework: FastAPI endpoints providing organized read/write downstream access to consolidated ticker maps and aggregated market indexes.
• Containerization: Docker and Docker Compose definitions ensuring exact environment parity across development, staging, and final staging execution.

2. Directory Structure

trading-sentiment-aggregator/
├── api/
│   ├── endpoints.py
│   ├── __init__.py
│   └── main.py
├── config/
│   └── general_config.json
├── core/
│   ├── config.py
│   ├── __init__.py
│   ├── security.py
├── data/
│   └── aggregator.db
├── database/
│   ├── base_model.py
│   ├── __init__.py
│   └── session.py
├── models
│   ├── __init__.py
│   ├── history.py
│   ├── instrument_cache.py
│   └── user.py
├── schemas
│   ├── __init__.py
│   ├── payload.py
│   ├── request.py
│   └── response.py
├── scripts
│   ├── manage_users.py
│   ├── update_config.py
├── services
│   ├── __init__.py
│   ├── aggregator.py
│   ├── analyzer_client.py
│   ├── cache_warmer.py
│   ├── global_rate_limiter.py
│   ├── history_service.py
│   ├── news_fetcher.py
│   ├── news_provider.py
│   └── sentiment_service.py
├── tests
│   ├── __init__.py
│   ├── conftest.py
│   └── test_api.py
├── .gitignore
├── .gitlab-ci.yml
├── docker-compose.yml
├── Dockerfile
├── pytest.ini
├── README.md
└── requirements.txt

3. Technology Stack & Key Libraries

• Language: Python 3.11+
• Web Framework: FastAPI (Uvicorn server)
• Database & ORM: SQLite + SQLAlchemy (AsyncEngine implementation)
• Data Protocols: Protocol Buffers (pricing.proto) for structured payload operations
• Scraping Framework: Asynchronous requests and structured parsing libraries

4. Sources for Headlines

Trading Sentiment Aggregator API lists Yahoo Finance RSS Feed as one of the sources for headlines. Before calling the API, you can check that you will get a meaningful response by using this URL:

https://feeds.finance.yahoo.com/rss/2.0/headline?s="INSTRUMENT"&region=US&lang=en-US

Replace "INSTRUMENT" with your instrument. For example, to evaluate headlines for EURUSD use: https://feeds.finance.yahoo.com/rss/2.0/headline?s=EURUSD=X&region=US&lang=en-US

You can search for instruments supported by Yahoo here: https://finance.yahoo.com/lookup/

5. API Endpoints

The API is served under the local application server root path and provides programmatic access to the ingestion network.

Core Endpoints:

• GET /: Root endpoint for connectivity checks.
• GET /health: Health check and basic API information.
• GET /docs: Interactive Swagger UI documentation
• POST /analyze: Analyze instrument

5. How it Works

Send a request via POST to the /analyze endpoint: *

curl --request POST \
	--url https://trading-sentiment-aggregator-ai.p.rapidapi.com/analyze \
	--header 'Content-Type: application/json' \
	--header 'x-rapidapi-host: trading-sentiment-aggregator-ai.p.rapidapi.com' \
	--header 'x-rapidapi-key: <your-api-key>' \
	--data '{"instrument":"USD","lookback": 3}'

*you can get your free api-key on RapidAPI

You can expect a response such as

{
  "instrument": "USD",
  "current": {
    "headline": "RBC Capital Markets Says USD/CAD Bias Remains Skewed Toward Top End Of 1.3500-1.3900 Range",
    "prediction": "bearish",
    "confidence": 0.904,
    "published":"2026-05-12T20:30:12",
    "source": "live"
  },
  "remainingCredits": 999999
}

6. Sources for Headlines

Trading Sentiment Aggregator AI lists Yahoo Finance RSS Feed as one of the sources for headlines. Before calling the API, you can check that you will get a meaningful response by using this URL:

https://feeds.finance.yahoo.com/rss/2.0/headline?s="INSTRUMENT"&region=US&lang=en-US

Replacing "INSTRUMENT" with your instrument. For example, to evaluate headlines for EURUSD use: https://feeds.finance.yahoo.com/rss/2.0/headline?s=EURUSD=X&region=US&lang=en-US

You can search for instruments supported by Yahoo here: https://finance.yahoo.com/lookup/