CI & JUnit reports¶

Horizon-QA CI runs are normal dedicated-server runs with the Horizon-QA mode set on the Minecraft server JVM:

./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci"

--mcJvmArgs is provided by Retrofuturagradle (RFG). Do not pass -Dhorizonqa.mode=ci directly to Gradle; that sets the property on the Gradle daemon, where the Minecraft server cannot read it.

In horizonqa.mode=ci, Horizon-QA discovers tests, runs the selected batch automatically after the server is ready, writes reports, and exits the process with a deterministic status code. Local authoring should use horizonqa.mode=interactive or omit the mode property, because interactive is the default.

Use horizonqa.mode=ci -Dhorizonqa.autoRun=false when you want report files from a manually-started non-interactive batch without CI lifetime management:

./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.autoRun=false -Dhorizonqa.reportDir=build/horizonqa"

Manual reported batches use the same report formats as automatic CI and default to the same void world policy. Then run /horizonqa run <testId>, /horizonqa runall [namespace], or /horizonqa runfailed. The selected batch writes JUnit XML and status JSON when it finishes, but the server does not auto-run tests at startup and does not exit afterward. horizonqa.tests and horizonqa.allowNoTests only affect automatic execution; for manual reported batches, use the command arguments to choose tests.

Modes are presets. Override specific behavior when the workflow needs it:

# Use the configured or existing world instead of Horizon-QA's void world
./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.world=normal"

# Run automatically but keep the server up afterward
./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.stopServer=false"

# Place the test grid at Y=128
./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.autoRun=false -Dhorizonqa.world=normal -Dhorizonqa.gridOrigin=0,128,0"

# Manual reported batches with CI overrides
./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.autoRun=false -Dhorizonqa.reportDir=build/horizonqa"

Report files¶

By default reports are written in the server process working directory:

TEST-horizonqa.xml
horizonqa-result.json

For CI, send them to a predictable artifact directory:

./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.reportDir=build/horizonqa"
./gradlew runServer --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.autoRun=false -Dhorizonqa.reportDir=build/horizonqa"

Report path flags:

Relative paths resolve from the server process working directory. When horizonqa.reportDir is set and horizonqa.statusFile is not set, the status JSON is written to horizonqa-result.json in that same directory.

JUnit XML¶

TEST-horizonqa.xml uses a standard JUnit-style <testsuite>:

<testsuite name="horizonqa" tests="…" failures="…" errors="…" skipped="…" …>
  <testcase name="methodName" classname="namespace:ClassName" time="…">
    <!-- failure, error, skipped, and system-out elements as appropriate -->
  </testcase>
</testsuite>

Required assertion failures and timeouts are emitted as <failure>. Infrastructure problems such as cleanup, template, configuration, selection, report-path, and reporting failures are emitted as <error>. Optional failures are emitted as <skipped> so JUnit publishers can show them without failing the suite aggregate.

When event recording is enabled, each <testcase> may include ordered [t=NNN] [category] summary lines in <system-out>. The server console also prints a compact failure tail.

Disable event recording only for performance investigations:

-Dhorizonqa.events=off

Status JSON schema¶

horizonqa-result.json is the compact automation surface. Schema version 1 has this top-level shape:

{
  "schemaVersion": 1,
  "status": "passed",
  "exitCode": 0,
  "configuration": {
    "mode": "ci",
    "rawMode": "ci",
    "world": "void",
    "rawWorld": null,
    "autoRun": true,
    "rawAutoRun": null,
    "stopServer": true,
    "rawStopServer": null,
    "gridOrigin": "0,64,0",
    "rawGridOrigin": null,
    "tests": null,
    "selectsAllTests": true,
    "allowNoTests": false,
    "eventsEnabled": true,
    "reportFile": null,
    "reportDir": "build/horizonqa",
    "statusFile": null
  },
  "counts": {
    "selectedTests": 1,
    "passed": 1,
    "failed": 0,
    "timedOut": 0,
    "incomplete": 0,
    "requiredFailures": 0,
    "optionalFailures": 0,
    "issues": 0,
    "diagnosticErrors": 0,
    "junitFailures": 0,
    "junitErrors": 0,
    "junitSkipped": 0
  },
  "reports": {
    "junit": "build/horizonqa/TEST-horizonqa.xml",
    "status": "build/horizonqa/horizonqa-result.json"
  },
  "issues": [],
  "tests": []
}

Each issues[] entry contains id, kind, source, name, message, fatalInCi, and optional details / stackTrace. Each tests[] entry contains id, classname, name, status, required, ticks, timeSeconds, optional blockedByIssueId, and optional failure details.

Status values are:

Selectors¶

Use horizonqa.tests to limit automatic execution:

-Dhorizonqa.tests=mymod
-Dhorizonqa.tests=mymod:AssemblerTests.processesOneRecipe
-Dhorizonqa.tests=mymod,compatmod:BridgeTests.basic

Selector grammar:

selectors := selector ("," selector)*
selector  := namespace | exact-test-id
namespace := token-without-colon
exact-test-id := namespace ":" class-and-method

Rules:

• unset or empty horizonqa.tests selects all valid tests,
• a namespace selector matches every valid test id that starts with namespace:,
• an exact selector must contain exactly one : and match the full test id,
• whitespace around comma-separated tokens is trimmed,
• empty tokens such as a,,b are invalid,
• * is not supported; omit the property or set it to an empty value to run everything,
• duplicate selections are de-duplicated while preserving discovery order.

For automatic execution, invalid selector syntax aborts before tests run and exits 2. A syntactically valid selector that matches no valid tests is reported as a CI infrastructure issue; if other selectors match valid tests, those tests still run and the final result still includes the selector issue.

If no valid tests are selected automatically, CI still writes TEST-horizonqa.xml and horizonqa-result.json. By default this exits 2. Set -Dhorizonqa.allowNoTests=true only for jobs where an empty selection is expected and there are no selector infrastructure issues. Manual reported batches ignore these selector properties and use /horizonqa run, /horizonqa runall [namespace], or /horizonqa runfailed arguments instead.

Optional tests¶

@GameTest(required = false) marks a test as optional. Optional tests still run, appear in JUnit XML, and appear in tests[] in the status JSON with required: false.

An optional failure or timeout:

• increments counts.optionalFailures,
• is represented as <skipped> in JUnit XML,
• does not make the process exit non-zero by itself.

Use optional tests for genuinely quarantined, experimental, or environment-specific coverage. Required tests should gate merges.

GitHub Actions handling¶

Always upload reports with if: always() so failed tests still leave artifacts. Publish JUnit XML from a later always() step, then let the original runServer exit code fail the job.

name: Horizon-QA

on:
  pull_request:
  push:
    branches: [ master, main ]

jobs:
  gametest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version-file: .java-version

      - uses: gradle/actions/setup-gradle@v4

      - name: Run Horizon-QA
        run: >
          ./gradlew runServer
          --mcJvmArgs="-Dhorizonqa.mode=ci -Dhorizonqa.reportDir=build/horizonqa"

      - name: Upload Horizon-QA reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: horizonqa-reports
          path: |
            build/horizonqa/TEST-horizonqa.xml
            build/horizonqa/horizonqa-result.json

If your workflow uses a JUnit publishing action, run it after the upload step with if: always() and point it at build/horizonqa/TEST-horizonqa.xml.

When CI fails¶

Work from the artifacts before relaunching anything: the <failure> message, the event trace in <system-out>, and issues[] in the status JSON usually identify the cause on their own. The triage workflow, including a failure-signature table and the in-game reproduction loop, is in Debugging failed tests.

read TEST-horizonqa.xml → runServer (interactive) → /horizonqa runfailed
                        → fix → /horizonqa runthis → push