Automation model

End-to-end GitLab automation between workflow repositories and docs-portal.

Pipeline sources

Source	Portal behavior
Push to develop / master	Lint → validate mirrors → build → Pages
Downstream trigger from workflow repo	Sync → validate → commit mirror → build → Pages
Manual SYNC_MODE=all (web)	Full sync all active workflows

Workflow repo pipeline (example: superbuild)

Stages: validate → trigger

1. workflow-docs:contract

Runs on develop when docs/**/* changes.

• Confirms docs/workflow-manifest.yaml, portal-integration.md, navigation-meta.md
• Validates manifest workflow_id and version

2. trigger:docs-portal

Triggers multi-project pipeline:

trigger:
  project: libesys/ai-workflows/docs-portal
  branch: develop
  strategy: depend
  forward:
    pipeline_variables: true
variables:
  WORKFLOW_SYNC_ID: superbuild
  WORKFLOW_SYNC_REF: $CI_COMMIT_SHA
  WORKFLOW_SYNC_PROJECT: $CI_PROJECT_PATH
  SYNC_MODE: single
  PORTAL_SYNC_COMMIT: "true"

strategy: depend blocks the workflow pipeline until the portal pipeline completes — failures are visible upstream.

Portal pipeline (docs-portal)

Defined in .gitlab-ci.yml + .gitlab/workflow-sync.yml.

Sync stage (sync:workflow)

When WORKFLOW_SYNC_ID is set:

1. Clone workflow repo at WORKFLOW_SYNC_REF using CI_JOB_TOKEN
2. When the manifest declares ci_artifact_overlay, download artifacts from WORKFLOW_SYNC_PARENT_PIPELINE_ID / WORKFLOW_SYNC_ARTIFACT_JOB and overlay CI-generated docs assets (for example extension screenshots) onto the clone
3. python scripts/sync_workflow.py <id> --commit
4. Update sidebars*.ts from manifest sidebar_order
5. Commit and push mirror changes to develop when PORTAL_SYNC_COMMIT=true

Validate stage

Job	When
validate:import	After sync trigger — full manifest/registry validation
validate:mirrored	Every push — portal-owned files and target dirs exist

Build and Pages

Unchanged Docusaurus flow:

• develop → https://docs-portal-1c6c85.gitlab.io
• master → https://esys-ai.org

Numbered end-to-end flow

1. Author edits superbuild-workflow/docs/ on develop
2. Workflow CI validates portal contract files
3. Workflow CI triggers docs-portal downstream pipeline (strategy: depend)
4. Portal clones superbuild at WORKFLOW_SYNC_REF
5. sync_workflow.py updates docs-superbuild/ (preserves intro.md)
6. Sidebar updated from manifest order
7. validate_workflow_import.py checks contract coherence
8. Automated chore(sync): … commit lands on develop (if diff); triggered runs push with ci.skip so GitLab does not auto-cancel the downstream pipeline
9. npm run build with branch-appropriate DOCS_SITE_URL
10. pages publishes preview site
11. Workflow pipeline marks trigger job pass/fail based on portal result

Required GitLab configuration

docs-portal project

Setting	Value
Settings → CI/CD → Pipeline triggers	Inbound triggers from workflow projects allowed
Settings → CI/CD → Variables	GITLAB_SYNC_TOKEN (masked) — see Group access token below
Settings → CI/CD → Job token permissions → Outbound	Optional fallback if not using GITLAB_SYNC_TOKEN for clone
Settings → CI/CD → Job token permissions → Allow Git push	Enable if automated chore(sync): commits push via CI_JOB_TOKEN
Pages custom domain	esys-ai.org on production (master) only

On GitLab.com Free, cross-project artifact download (ci_artifact_overlay) requires GITLAB_SYNC_TOKEN. Job tokens and needs:project are insufficient (needs:project also requires Premium).

Group access token (Free tier)

Create one token for the whole libesys/ai-workflows namespace:

1. Open group libesys/ai-workflows → Settings → Access tokens → Add new token
2. Token name: e.g. docs-portal-sync
3. Role: Developer (or Maintainer)
4. Scopes: enable read_api, read_repository, and write_repository
   • read_api — list jobs and download CI artifacts (extension screenshots)
   • read_repository — clone workflow repos during sync
   • write_repository — push automated chore(sync): commits (optional if push stays on CI_JOB_TOKEN)
5. Set an expiration date, create the token, and copy the value (shown once).

Then on docs-portal → Settings → CI/CD → Variables → Add variable:

Key	Value	Flags
GITLAB_SYNC_TOKEN	paste token	Masked, Protected (recommended)

No job-token allowlist changes are required for artifact download when GITLAB_SYNC_TOKEN is set. The sync scripts use the PRIVATE-TOKEN header for API calls and oauth2:<token> for git clone/push.

Token rotation: create a new group access token, update the CI variable, revoke the old token.

Contributor repos with ci_artifact_overlay (optional)

If you rely on CI_JOB_TOKEN instead of GITLAB_SYNC_TOKEN, the contributor project must add docs-portal to its inbound allowlist with Read jobs (READ_JOBS). Not needed when GITLAB_SYNC_TOKEN is configured.

Each workflow project

Setting	Value
CI/CD variables	DOCS_PORTAL_PROJECT, WORKFLOW_SYNC_ID (or hardcode in .gitlab-ci.yml)
Multi-project pipelines	Permission to trigger docs-portal

Token scopes (group access token)

• read_api — job artifacts API (ci_artifact_overlay)
• read_repository — clone workflow repos
• write_repository — push sync commits (if not using CI_JOB_TOKEN for push)

Security and governance

• Workflow repos remain source of truth — portal tokens only clone and mirror
• Automated commits use a dedicated bot identity (docs-portal sync)
• Manual edits to mirrored files are discouraged; sync overwrites on next trigger
• Downstream triggers are scoped to docs/** changes on develop

Local parity

Developers run the same sync without CI:

pip install -r requirements-sync.txt
python scripts/sync_workflow.py superbuild --source ../superbuild-workflow
python scripts/validate_workflow_import.py superbuild --source ../superbuild-workflow
npm run build

Related documents

• Sync architecture
• GitLab Pages
• Workflow onboarding