GitSiteSyncer

Fetches all pages from a sitemap.xml, rewrites internal links, and commits the results to a GitHub Pages repository. Also syncs additional raw files (e.g. llms.txt, robots.txt) directly from the host domain. Runs as a systemd timer on a Linux VPS.

---

What it does

1. Pulls the latest state of the target GitHub Pages repo via LibGit2Sharp.
2. Downloads sitemap.xml from the configured URL and saves it to the repo root.
3. Fetches every HTML page listed in the sitemap, rewrites app-link / no-app-link anchors to point to the GitHub Pages domain, and saves each page as a static .html file.
4. Downloads any additional files listed in AdditionalFiles (e.g. llms.txt, robots.txt) as raw text — no rewriting applied.
5. Deletes .html files in the repo that are no longer in the sitemap (skipped if any download failed that run).
6. Stages, commits, and force-pushes all changes.

---

Configuration

Copy appsettings.template.json to appsettings.json in the same directory and fill in real values. appsettings.json is gitignored and never committed.

{
  "GitDirectory": "/path/to/local/pages-repo",
  "LockFileDirectory": "/var/lock/gitsitesyncer",
  "GitCredentials": {
    "Email": "you@example.com",
    "Username": "your-github-username",
    "Password": "YOUR_GITHUB_TOKEN"
  },
  "SitemapUrl": "https://yourapp.com/sitemap.xml",
  "MinutesToConsider": 60,
  "AppHostDomain": "https://yourapp.com",
  "NoAppHostDomain": "https://yourorg.github.io",
  "Exclusions": [
    "404.html",
    "offline.html"
  ],
  "AdditionalFiles": [
    "llms.txt",
    "robots.txt"
  ]
}

Key	Description
GitDirectory	Absolute path to the local clone of the GitHub Pages repo
LockFileDirectory	Directory for the run lock file (prevents overlapping runs)
GitCredentials	GitHub username + PAT with repo scope
SitemapUrl	Full URL to the sitemap, e.g. https://yourapp.com/sitemap.xml
MinutesToConsider	Currently unused; reserved for future incremental-sync logic
AppHostDomain	The live app domain — links with class app-link are rewritten to this
NoAppHostDomain	The GitHub Pages domain — links with class no-app-link are rewritten to this
Exclusions	HTML files to keep even when absent from the sitemap
AdditionalFiles	Root-relative paths fetched raw from AppHostDomain and committed as-is

---

Deploying to a Linux VPS

1. Publish

From the repo root on your local machine:

dotnet publish GitSiteSyncer/GitSiteSyncer.csproj \
  -c Release \
  -r linux-x64 \
  --self-contained true \
  -o ./publish

2. Copy to the server

scp -r ./publish user@yourserver:/var/www/gitsitesyncer

3. Configure

On the server, copy the template and fill in values:

cp /var/www/gitsitesyncer/appsettings.template.json \
   /var/www/gitsitesyncer/appsettings.json
nano /var/www/gitsitesyncer/appsettings.json

Make sure GitDirectory points to a local clone of your GitHub Pages repo that already exists on the server:

git clone https://github.com/yourorg/your-pages-repo.git /path/to/pages-repo

4. Create the systemd service unit

/etc/systemd/system/gitsitesyncer.service:

[Unit]
Description=GitSiteSyncer — sync site to GitHub Pages
After=network.target

[Service]
Type=oneshot
User=www-data
ExecStart=/var/www/gitsitesyncer/GitSiteSyncer
WorkingDirectory=/var/www/gitsitesyncer
StandardOutput=journal
StandardError=journal

5. Create the systemd timer unit

/etc/systemd/system/gitsitesyncer.timer:

[Unit]
Description=Run GitSiteSyncer hourly

[Timer]
OnCalendar=hourly
Persistent=true

[Install]
WantedBy=timers.target

Adjust OnCalendar to taste — *:0/30 for every 30 minutes, daily for once a day, etc. See man systemd.time.

6. Enable and start

sudo systemctl daemon-reload
sudo systemctl enable --now gitsitesyncer.timer

7. Verify

# Check the timer is scheduled
systemctl list-timers gitsitesyncer.timer

# Trigger a manual run immediately
sudo systemctl start gitsitesyncer.service

# Watch the output
journalctl -u gitsitesyncer.service -f

---

Redeploying after a code change

# Local
dotnet publish GitSiteSyncer/GitSiteSyncer.csproj \
  -c Release -r linux-x64 --self-contained true -o ./publish

scp -r ./publish user@yourserver:/var/www/gitsitesyncer

# Remote — no service restart needed; next timer tick picks up the new binary

appsettings.json on the server is not touched by a redeploy (it isn't in the publish output). If you need to update config, edit it directly on the server.

---

Observability

# Last run result
journalctl -u gitsitesyncer.service --since "1 hour ago" --no-pager

# All recent runs
journalctl -u gitsitesyncer.service -n 200 --no-pager

# Timer next-run / last-trigger
systemctl status gitsitesyncer.timer

---

Lock file

A lock file under LockFileDirectory prevents two instances from running simultaneously (e.g. if a run takes longer than the timer interval). It is created at start and deleted on exit. If the process is killed hard, delete app.lock manually before the next run.

License

MIT — see the LICENSE file.