Overview

Insight is a passive web threat scanner. Submit any URL and it fetches all public resources — HTML, scripts, HTTP headers, and TLS certificates — then analyses them entirely on content. There are no calls to reputation databases or external threat intelligence APIs.

Because detection is content-based, Insight catches zero-day campaigns, freshly registered phishing domains, and newly injected skimmers that reputation feeds haven't yet indexed. It is a companion tool to vault1337.com and shares the same design system.

The result of each scan is a prioritised findings report covering JavaScript threats, phishing indicators, domain intelligence, security misconfigurations, and the full detected technology stack — each finding categorised by severity (CRITICAL / HIGH / MEDIUM / LOW / INFO) with supporting evidence extracted directly from the page content.

Requirements

1. Start Redis

Redis must be running before the backend or Celery worker will start successfully.

docker run -d -p 6379:6379 redis:7-alpine

redis-server

2. Backend

git clone https://github.com/DanDreadless/Insight.git
cd Insight/backend

pip install -r requirements.txt

cp ../.env.sample ../.env
# Edit ../.env — set SECRET_KEY to a long random string

python manage.py migrate
python manage.py runserver

3. Celery Worker

Scans are executed as asynchronous Celery tasks. The worker must be running in a separate terminal — scans will queue but never run without it.

celery -A insight worker -l info

4. Frontend

The React frontend runs on its own dev server and proxies all /api/ requests to Django on port 8000.

npm install
npm run dev

5. Verify

Run the test suite

# All tests
python manage.py test scanner

# Single module
python manage.py test scanner.tests.test_validators

Docker (Alternative)

If you prefer not to install Python and Node.js locally, the full stack can be started with Docker Compose.

cp .env.sample .env   # edit SECRET_KEY
docker-compose up --build

# Stop and remove volumes
docker-compose down -v

# Full clean (removes images too)
docker-compose down --rmi all --volumes --remove-orphans

Environment Variables

All configuration lives in .env at the repo root. Copy from .env.sample — the only required change for local development is setting SECRET_KEY.

How It Works

When a URL is submitted, Insight fetches the page and all linked external scripts using an SSRF-safe HTTP fetcher. No external threat intelligence APIs are called — all analysis is performed against the raw content returned by the target server.

Five analysis modules run against every scan:

• JavaScript analyser — 59 checks against all scripts collected from the page
• HTML analyser — 33+ structural checks against the page markup
• Domain intelligence — 14 checks on the hostname and TLD
• Header analyser — 15 checks on HTTP response headers
• SSL analyser — 6 checks on the TLS certificate

Each check emits zero or more findings. A context-collapse engine fires additional synthetic findings when combinations of signals indicate coordinated attack infrastructure.

Verdict & Scoring

Verdict derivation

Context collapse rules

Synthetic findings are generated when multiple signals combine to indicate a coordinated attack pattern:

• High-risk TLD + external form action + missing security headers → HIGH "phishing infrastructure"
• DGA domain + hidden iframe + obfuscated JS → CRITICAL "drive-by malware delivery"
• Brand impersonation + phishing form (± new certificate) → CRITICAL "active phishing campaign"
• Keylogger / skimmer + DevTools evasion → CRITICAL "sophisticated targeted malware"
• Fake CAPTCHA / ClickFix UI + clipboard write → CRITICAL "ClickFix malware delivery"
• Injected unknown external script + ClickFix / clipboard payload → CRITICAL "compromised site delivering ClickFix malware"
• Newly registered domain (≤ 30 days) + high-risk TLD → HIGH "newly registered high-risk domain" — purpose-built attack infrastructure signal

JavaScript Analysis

All scripts are run through jsbeautifier before analysis (files ≤ 256 KB). Both the original and beautified forms are checked. Base64-encoded strings are decoded and the plaintext payload is appended to the evidence block where printable.

CRITICAL

HIGH

MEDIUM

HTML Analysis

Domain Intelligence

Domain checks run against the hostname of the scanned URL. No WHOIS or DNS lookups are made — detection is based on the domain string itself.

Brands monitored include: PayPal, Google, Microsoft, Apple, Amazon, Facebook, Instagram, Netflix, Steam, Coinbase, Binance, MetaMask, Ledger, Trezor, Trust Wallet, OpenSea, Roblox, Discord, Twitch, Spotify, Chase, Barclays, and more.

Header Analysis

15 checks on HTTP response headers. All checks are passive — no additional requests are made.

Severity philosophy: Missing defensive headers are configuration debt, not threat indicators. Industry consensus (Cobalt, OWASP, Invicti, pentest report standards) rates them LOW/INFO. They only escalate in meaning when combined with active threat signals — handled by the context collapse engine.

SSL Analysis

6 checks on the TLS certificate. The certificate is retrieved directly — no third-party certificate transparency APIs are used.

Technology Detection

Identifies the technology stack from HTML, script sources, HTTP headers, and cookies. Displayed as colour-coded badges with logos on the results page.

Visual Renderer (Carapace)

Carapace is an optional companion service that provides a Chromium-headless visual screenshot of each scanned URL. It renders the page with JavaScript fully enabled but all network requests intercepted and blocked — allowing dynamic overlays (ClickFix, SocGholish, ClearFake, drainers) to execute and render visibly in the screenshot, revealing the actual attack UI. A verdict badge is composited onto every screenshot.

Carapace runs as a separate Docker service alongside the Insight backend. When CARAPACE_URL is configured, every scan automatically calls the POST /render endpoint and the screenshot is stored with the scan result. If Carapace is unavailable the scan continues normally — it is entirely best-effort.

In addition to the screenshot, Carapace returns a threat report with:

• A risk score (0–100) derived from renderer-level observations
• Threat flags — findings from the renderer converted directly into Insight findings under the Renderer category
• Technology detections — a DOM-parsed tech stack that is merged with Insight's own BeautifulSoup-based detections, catching things a static parser may miss

Renderer Findings

Threat flags from Carapace are converted to Insight findings in the Renderer category. Sanitisation-behaviour codes that fire on almost every real page (e.g. BLOCKED_ELEMENT_SCRIPT, NETWORK_ATTEMPT_BLOCKED) are suppressed — only signals with independent threat value are surfaced.

API Endpoints

All endpoints are under /api/. No authentication is required — rate limiting is enforced per IP address.

Tech Stack

Acknowledgements

Insight is built on a number of excellent open-source libraries.

Backend & Analysis

Frontend

Infrastructure

Visual Renderer