Job Hunter Agent: Data and Integration

Most of Job Hunter Agent’s data can be thrown away and regenerated by running the pipeline again. Two things can’t: my review decisions and the signed-in LinkedIn session. That asymmetry drives the retention, backup and migration choices below.

Data ownership and classification

Data	Owner	Class	Regenerable?
Resumes (master and derived)	Me	Personal, confidential	No (keep my own copies)
Search profiles	Me	Internal	Easy to recreate
Job postings	The employer, public on LinkedIn	Public	Yes, while the posting is live
Scores, tailored resumes, cover letters	Me (generated)	Personal	Yes, at the cost of LLM time
Applications and review notes	Me	Personal	No
Company research, financials, news	Public sources	Public	Yes
Contact suggestions	Third parties’ public profile data	Personal data of others	Yes
Run history, agent results	Me	Internal	No, but low value
LinkedIn browser session	Me	Secret	No (needs an interactive sign-in)

Data retention and deletion

A daily upkeep job (03:30) deletes what’s past retention. The period is one setting, RETENTION_DAYS (180, six months; 0 keeps everything):

Data	Retention
Contact suggestions (other people’s names and titles)	6 months, with the model outputs that produced them
Rejected applications	6 months after the rejection, with their tailored resume, letter and model outputs
The posting behind a rejected application	Kept, so the same posting isn’t found and drafted again
Approved and applied applications	Kept
Postings never drafted, run history	Kept for now: small, and the run history feeds the SLOs

Deleted rows still exist in the nightly exports until those expire (at most 12 weeks, below). Each upkeep run appears in the run history with what it deleted.

Database schema and migrations

Twelve tables in Supabase Postgres: users, search_profiles, resume_templates, jobs, job_scores, optimized_resumes, cover_letters, applications, company_research, contacts, agent_results, runs. Lists are text[], structured values (resume changes, revenue series, news, run stats) are jsonb. Every table has row-level security with an own_rows policy on user_id, which the server’s service-role key bypasses; it’s there for the day the app authenticates users against Supabase directly.

Migrations live in supabase/migrations/, timestamped, all additive (add column if not exists):

Migration	Adds
…000000_init	The core tables and RLS
…010000_research_contacts	Company research cache, contacts, apply links
…020000_letter_warnings, …030000_company_slug	Letter warnings; LinkedIn company slug
20261022000000_profiles_and_screening	Profile name, resume, mode, tone; jobs’ profile; screening score
20261022010000_runs	Run history
20261022020000_company_facts	Company size, HQ, revenue, headcount, news
20261022030000_profile_schedule	Each profile’s cron

CI applies all of them to a fresh local Supabase and runs an integration test. In production I paste each one into the Supabase SQL editor before merging the code that needs it, because the app only holds the REST service key, not a database password.

Data validation and quality

• LLM output: validated against Pydantic schemas (scores 0–100, at most 25 resume changes, exactly three letter paragraphs, at most three contacts, no placeholders in notes). A failed validation gets one repair retry.
• Job pages: the header (company, title, location, posting age) is parsed deterministically; the LLM extracts it only when LinkedIn’s layout is unfamiliar. An earlier version let the model read the header and it sometimes named the wrong company.
• Company facts: a Wikidata item is used only if its official website matches the company’s LinkedIn website, or, without a website, its label matches exactly. Otherwise: no figures.

Import and export requirements

• Import: resumes as PDF, DOCX (including tables, read in document order), TXT or Markdown. Postings by hand: title, company, location, link and pasted description.
• Export: each tailored resume as DOCX, Markdown or plain text; the cover letter by copy.
• API: everything is JSON over the REST API, so a bulk export is a few curl calls.

External systems and third-party integrations

System	How	Volume
LinkedIn	MCP tools search_jobs, get_job_details, get_company_profile, get_company_employees, apply info	≤5 postings per keyword per run
LiteLLM	OpenAI-compatible chat completions with JSON schema	~5 calls per drafted job
Supabase	supabase-py (PostgREST) in a thread pool	Low
Wikidata	wbsearchentities, wbgetentities, with a descriptive User-Agent	2–3 requests per company per week
Google News	RSS search, last 180 days	1 request per company per week
Slack	Incoming webhook	1 message per scheduled run
Indeed	HTML scraping, off by default	—

Wikidata refused the first requests with HTTP 403 until the User-Agent named the tool and a contact URL, as its policy requires.

Event and messaging contracts

There’s no message bus; the only “events” are outbound Slack messages, and their contract is plain text:

⚠️ *Job Hunter*: scheduled full run for *Cloud Architect* errors in 7 min
5 found · 3 new · 2 skipped · 3 analyzed · 2 drafted
<https://jobs.platform.home.cloudutsuk.com/ui/|4 waiting for your review>
Errors: Cloud Architect: linkedin 'Cloud Architect': TimeoutError

The status emoji, counts and error lines come from the run record, so the message and the run history never disagree.

Backup and restore strategy

What	How it’s protected
LinkedIn session (PVC)	Velero file-system backup of the linkedin volume, nightly
Supabase data	The app’s own nightly export (below); the free plan has no backups
Resumes	My own files (the master resume lives outside the app too)
Code and config	Git

Supabase’s free plan takes no backups (daily backups start on the paid plans), and Supabase’s own docs tell free projects to export their data themselves. So the upkeep job writes every table to one gzipped JSON file on a second volume, backups, at 03:30 IST every day, and keeps the newest seven. Velero’s nightly file-system backup then copies that volume to Azure Blob storage off the host, encrypted by Kopia. Nightly copies are kept 30 days and weekly ones 12 weeks.

Why an export through the app rather than pg_dump:

• It uses the same service key the app already has, so the cluster never holds the database password.
• python -m hunter.maintenance restore <file> loads it back, and CI tests that round trip against a real Postgres.

The cost is that it’s a logical export: restoring needs the migrations applied first, and the RPO is a day.

Writing the export also exposed a bug that had been there all along. Supabase’s REST API returns at most 1,000 rows per request, so any list longer than that was silently cut short. Lists now page through the whole table.

Restoring the LinkedIn session from a backup is possible but LinkedIn may still ask for a fresh sign-in; the in-cluster login procedure is in the deployment runbook.

Data migration and reconciliation

There was no legacy data. The only migrations so far were schema changes on live data, and they were all additive with defaults, so existing rows stayed valid: the first profile became “Default”, existing jobs got no profile, existing research got its new facts the next time it was refreshed. When I edited the truthfulness rules I re-ran them over the two stored drafts and updated their warnings, which is the reconciliation pattern I’d reuse: recompute derived data, never hand-edit it.

Checklist

• Retention periods confirmed (6 months) and a clean-up job scheduled
• Supabase backup coverage confirmed (none on the free plan), with a tested export and restore
• Velero restore of the linkedin volume tried once
• Every migration additive and applied before its code merges
• Contact suggestions deleted after six months

• Data classification table reviewed when a new source is added