Engineering Blog

Using Forgejo as a Hot Standby for GitHub Outages

A second control plane that keeps CI running through Forgejo, a NAS, and a MacBook during a GitHub outage

Some time has passed since the last GitHub outage, but it made me think that I should not rely on GitHub alone. So I decided to look for an alternative. The resources I have are a MacBook and one U5825 NAS, and the goal is to make the most of them.

When GitHub goes down, it is not just the repository that becomes unavailable. In reality, PRs, GitHub Actions, self-hosted runner scheduling, Releases, and OIDC-based deployments can all be affected at once. Even if runners are alive on the NAS and MacBook, they cannot receive work when the GitHub Actions control plane is down.

So what is needed is not a simple Git mirror, but a second control plane that can keep at least Git and CI running without GitHub.

Forgejo seemed like the best option for that.

Forgejo is a self-hosted Git forge that provides repositories, pull requests, issues, releases, a package registry, and Actions-based CI. It is completely free to self-host, and its workflow syntax is similar to GitHub Actions, making it relatively easy to move existing projects over.

At first, I considered moving everything to Forgejo.

Forgejo
├─ Git
├─ PR
├─ CI
├─ Release
└─ Deploy
   ↓ mirror
GitHub
└─ backup / public mirror

But I ultimately decided against it.

The reason is Codex Security. When I have actually run Codex Security on Myven, it has often gone much deeper than ordinary code review. Rather than simply looking at the current diff, it finds vulnerable paths based on the whole codebase and its threat model, and the results have been quite good.

Codex automatic Code Review also fits naturally into GitHub PRs.

If I changed Forgejo into the canonical repository, it would be difficult to retain these GitHub-based capabilities as they are. The Git repository itself can be mirrored, but Forgejo PRs, review comments, and approval state are not replicated to GitHub.

So this is the structure I am currently considering.

                   GitHub
                   ├─ canonical repository
                   ├─ PR / Review
                   ├─ Codex Code Review
                   ├─ Codex Security
                   └─ Release
Developer
   │
   └──────────────► Forgejo
                    ├─ Git mirror
                    ├─ emergency PR
                    └─ Forgejo Actions
                         │
                 ┌───────┴────────┐
                 ▼                ▼
             U5825 NAS         MacBook
             max 3 jobs        max 2 jobs

In other words, GitHub remains the normal platform, while Forgejo serves as a hot standby that can be used during a GitHub outage.

The key is not simply copying the repository to Forgejo. CI must also run independently of GitHub.

For example, if GitHub Actions contains a lot of CI logic like this:

- name: test
  run: |
    # long test logic

It is better to move the actual work into repository scripts whenever possible.

scripts/ci/
├─ lint.sh
├─ test-api.sh
├─ test-web.sh
├─ test-browser.sh
└─ build-images.sh

In GitHub Actions:

- run: ./scripts/ci/test-api.sh

Forgejo Actions calls the same thing:

- run: ./scripts/ci/test-api.sh

This makes GitHub Actions and Forgejo Actions simple schedulers, while the actual CI logic is no longer tied to a particular service.

GitHub API dependencies also need attention.

For example, if the current CI runs:

gh api /repos/.../actions/runs

to see whether all three NAS runners are occupied before handing work to the Mac runner, the same setup cannot work on Forgejo during a GitHub outage.

On the Forgejo side, it is simpler to remove as much of this dynamic selection as possible and use runner labels and concurrency instead.

NAS

  • linux-amd64
  • docker
  • browser
  • concurrency 3

MacBook

  • linux-arm64
  • docker
  • browser
  • concurrency 2

Being precise about preferring the NAS is less important than making the system work simply during an outage.

Another easy-to-miss part is uses: actions/....

Suppose a Forgejo workflow has the following:

- uses: actions/checkout@...
- uses: actions/setup-node@...

Even if Forgejo itself is alive, it is not fully independent of a GitHub outage if it still has to retrieve those actions from GitHub.

Likewise, if CI images only exist in GHCR, a GHCR outage will also have an impact.

In the long run, it is better to mirror required actions in Forgejo, replace setup actions with scripts, or prepare a CI toolchain image containing the tools needed in advance.

For example:

myven-ci-toolchain
├─ Node
├─ pnpm
├─ Python
├─ uv
├─ Terraform
├─ gitleaks
└─ browser dependencies

If it is built for both amd64 and arm64, both the NAS and MacBook can be used.

Forgejo itself will use Docker on the NAS. I use Unraid, which provides a Docker UI.

U5825 NAS
├─ Unraid docker
│   └─ Forgejo
│
└─ k3s
    └─ CI workloads

This keeps the control plane and workloads separate.

If a GitHub outage actually occurs, it can be used roughly like this.

branch
 ↓
Forgejo
 ├─ temporary PR
 └─ Forgejo Actions
      ├─ lint
      ├─ unit test
      ├─ E2E
      └─ build

When GitHub recovers, push the same branch to GitHub again, create a normal GitHub PR, then go through Codex Review and Codex Security before the final merge.

The point of introducing Forgejo is not to abandon GitHub.

Rather:

GitHub
= the main development platform in normal times
Forgejo
= an independent Git + CI alternative during a GitHub outage

This is how the roles are divided.

Originally, I thought moving completely to Forgejo would be cleaner. But for now, the results from Codex Security are good enough that there is not much reason to give up GitHub.

Instead, I plan to keep Forgejo on the NAS and use the U5825 NAS and MacBook as runners so that testing and builds can continue even when GitHub is unavailable.

The most important verification criterion is simple.

Can Forgejo CI complete successfully even when access to GitHub domains, APIs, and GHCR is blocked?

If this can actually be checked regularly, it becomes a much more practical preparation for GitHub outages than a simple Git mirror.

https://x.com/studiojin_dev/status/2091818813876768896

Join the Investigation

Get the latest updates on my projects and indie hacking journey directly in your inbox.

No spam. Unsubscribe anytime.