test: real shader/terrain tests + headless render check

npm test was a brace counter: it checked each file was >10 chars and that {
and } counts matched. It passed on a build whose shader would not compile.

- tests/shaders.test.js compiles the game's real GLSL with glslangValidator
  (Khronos' reference compiler) under a three.js-style attribute prefix, and
  includes a canary that asserts a known-bad vec4 construction is REJECTED, so
  the suite cannot pass vacuously.
- tests/terrain.test.js covers heightfield determinism (the "persistent world"
  claim), airport flattening inside the 200 m radius with no leakage outside it,
  out-of-bounds behaviour, plausible height range, and vertex colour counts.
- scripts/headless-check.mjs (npm run test:render) serves dist/, drives the
  game over raw CDP with SwiftShader software WebGL, and fails if the canvas is
  black, the shaders fail to compile, or the scene does not change after input.
  No new dependencies: Node 22 built-in fetch/WebSocket. 120 s watchdog so it
  cannot hang a runner.
- npm test -> node --test "tests/*.test.js"; old checker kept as npm run lint:syntax.
- CI installs glslang-tools and runs the real tests; render job is
  continue-on-error because not every runner has a Chrome binary.

Verified non-vacuous: re-breaking the shader makes the unit tests fail 2/4 with
glslang's own "constructor: too many arguments" and makes the render check fail
four ways (1 distinct colour, 0/64000 lit pixels, shader error, frozen scene).
This commit is contained in:
Hermes Cron Job committed 2026-10-07 13:31:32 +00:00
1 parent 04e3400ab9
commit ee24276fd0
6 files changed
+528 -13

No files matched your search

+32 -1
View File
@@ -16,10 +16,41 @@ jobs:
with:
node-version: 22
# glslangValidator is Khronos' reference GLSL compiler. The shader tests compile
# the game's real shaders with it; without it they skip, so install it explicitly.
- name: Install GLSL compiler
run: sudo apt-get update && sudo apt-get install -y glslang-tools
- run: npm ci
- name: Syntax and structure checks
- name: Unit tests (shaders, terrain)
run: npm test
- name: Build
run: npm run build
- name: Basic syntax lint
run: npm run lint:syntax
# Needs a Chrome/Chromium binary and CPU-rendered WebGL (SwiftShader). Not every
# runner has one, so this reports without blocking the pipeline.
render:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- name: Install Chromium
run: sudo apt-get update && sudo apt-get install -y chromium
- run: npm ci
- name: Build
run: npm run build
- name: Headless render check
run: CHROME=$(command -v chromium || command -v chromium-browser) npm run test:render
+26 -11
View File
@@ -67,16 +67,31 @@ shove a warning in your face.
**Fix:** stall flag on the flight model → HUD warning + stick shaker / audio cue.
## T4 — The test suite is theatre
## T4 — The test suite is theatre — FIXED on branch `fix/crt-shader-and-real-tests`
`scripts/check-syntax.js` checks that each file is >10 chars and that `{` and `}` counts
match. It cannot catch T1. CI (`.gitea/workflows/*.yml`) runs it on every push, so the
green tick currently means nothing.
match. It cannot catch T1 or T8. CI (`.gitea/workflows/ci.yml`) runs it on every push, so
the green tick meant nothing.
**Fix:** real unit tests on `FlightModel` (takeoff profile, stall, ground contact) and
`Terrain` (determinism, airport flattening radius). Node has `node:test` built in — no
new dependency needed. Add a headless smoke test that asserts the game reaches the
`FLIGHT` state.
Now: `npm test` runs `node --test "tests/*.test.js"` (no new dependency) with
`tests/shaders.test.js` and `tests/terrain.test.js`; the old brace-counter survives as
`npm run lint:syntax`. `npm run test:render` runs `scripts/headless-check.mjs`, a
dependency-free CDP check that renders the built game in software WebGL and fails if the
canvas is black or the shaders fail to compile.
Both new tests were proven **non-vacuous** by re-breaking the shader and watching them
fail: the shader tests fail 2/4 with glslang's own `'constructor' : too many arguments`,
and the render check fails four ways (1 distinct colour, 0/64,000 lit pixels, shader
error, scene frozen).
Still outstanding: no `FlightModel` tests yet — those belong with T1, where the expected
behaviour has to be decided first.
Note on CI: this instance currently has **no registered Actions runners**
(`GET /repos/kevinhermes/retro-flight-sim/actions/runners` → `total_count: 0`), and
upstream's only run (id 36) ended `cancelled`, not `success`. So CI has never actually
validated this repo. The workflow is updated to install `glslang-tools` and run the real
tests, but treat CI as unproven until a runner exists.
## T5 — No LICENSE
@@ -91,7 +106,7 @@ not the same thing.
**Fix:** persist the world seed + aircraft state; add a save/load menu entry.
## T8 — The CRT shader does not compile, so the 3D canvas is black
## T8 — The CRT shader does not compile, so the 3D canvas is black — FIXED on branch `fix/crt-shader-and-real-tests`
Found by running the game under software WebGL, not by reading the code. `js/retro.js:29`:
@@ -115,10 +130,10 @@ colour, 0/64,000 non-black pixels**. Verified in isolation: `vec4(vec3, float)`
**Fix (verified):** `gl_Position = vec4(position.xy, 0.0, 1.0);` — compiles clean, and the
canvas then reports 143 distinct colours, 63,792/64,000 non-black pixels, scanline banding
visible every 25 rows, and the scene hash changing frame to frame once the aircraft moves.
Applied locally during the audit to confirm it, then reverted — not committed.
Committed on this branch.
Note this is the kind of bug the current test suite cannot see (see T4): braces balance
perfectly.
Note this is exactly the class of bug the old test suite could not see: braces balanced
perfectly. `tests/shaders.test.js` now compiles these shaders with glslang on every run.
## T7 — Minor / housekeeping
+4 -1
View File
@@ -12,6 +12,9 @@
"scripts": {
"dev": "vite",
"build": "vite build",
"test": "node scripts/check-syntax.js"
"preview": "vite preview",
"test": "node --test \"tests/*.test.js\"",
"test:render": "node scripts/headless-check.mjs",
"lint:syntax": "node scripts/check-syntax.js"
}
}
+291
View File
@@ -0,0 +1,291 @@
#!/usr/bin/env node
/*
* Headless render check — proves the 3D canvas actually renders.
*
* No dependencies: uses Chrome's DevTools protocol over Node's built-in fetch/WebSocket.
* Works without a GPU via SwiftShader (CPU rasterisation).
*
* node scripts/headless-check.mjs
* CHROME=/path/to/chrome node scripts/headless-check.mjs
*
* Exits non-zero if the canvas renders black or the shaders fail to compile.
*/
import { spawn, execSync } from 'node:child_process';
import { createServer } from 'node:http';
import { readFile } from 'node:fs/promises';
import { rmSync, mkdtempSync, existsSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join, extname, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
const root = join(dirname(fileURLToPath(import.meta.url)), '..');
const dist = join(root, 'dist');
const PORT = Number(process.env.CHECK_PORT || 4176);
const CDP_PORT = Number(process.env.CDP_PORT || 9333);
function findChrome() {
if (process.env.CHROME) return process.env.CHROME;
const candidates = [
'/root/.cache/ms-playwright/chromium-1234/chrome-linux64/chrome',
'/usr/bin/google-chrome',
'/usr/bin/chromium',
'/usr/bin/chromium-browser',
];
for (const c of candidates) {
if (existsSync(c)) return c;
}
// last resort: ask the shell
try {
return execSync('command -v google-chrome chromium chromium-browser 2>/dev/null | head -1').toString().trim() || null;
} catch {
return null;
}
}
const MIME = { '.html': 'text/html', '.js': 'text/javascript', '.css': 'text/css', '.ico': 'image/x-icon' };
function serveStatic() {
return new Promise((resolve) => {
const server = createServer(async (req, res) => {
const path = req.url === '/' ? '/index.html' : req.url.split('?')[0];
try {
const body = await readFile(join(dist, '.' + path));
res.writeHead(200, { 'content-type': MIME[extname(path)] || 'application/octet-stream' });
res.end(body);
} catch {
res.writeHead(404).end('not found');
}
});
server.listen(PORT, '127.0.0.1', () => resolve(server));
});
}
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function waitForCDP(port, attempts = 60) {
for (let i = 0; i < attempts; i++) {
try {
const r = await fetch(`http://127.0.0.1:${port}/json/version`);
if (r.ok) return r.json();
} catch {}
await sleep(500);
}
throw new Error('Chrome CDP endpoint never became available');
}
class CDP {
constructor(wsUrl) {
this.ws = new WebSocket(wsUrl);
this.id = 0;
this.pending = new Map();
this.listeners = new Map();
}
open() {
return new Promise((resolve, reject) => {
this.ws.addEventListener('open', resolve);
this.ws.addEventListener('error', reject);
this.ws.addEventListener('message', (ev) => {
let parsed;
try { parsed = JSON.parse(typeof ev.data === 'string' ? ev.data : ev.data.toString()); }
catch { return; }
this.handle(parsed);
});
});
}
send(method, params = {}) {
const id = ++this.id;
return new Promise((resolve, reject) => {
const timer = setTimeout(() => {
this.pending.delete(id);
reject(new Error(`CDP call '${method}' timed out after 20s`));
}, 20000);
this.pending.set(id, {
resolve: (v) => { clearTimeout(timer); resolve(v); },
reject: (e) => { clearTimeout(timer); reject(e); },
});
this.ws.send(JSON.stringify({ id, method, params }));
});
}
on(method, fn) {
if (!this.listeners.has(method)) this.listeners.set(method, []);
this.listeners.get(method).push(fn);
}
handle(event) {
if (event.id && this.pending.has(event.id)) {
const { resolve, reject } = this.pending.get(event.id);
this.pending.delete(event.id);
event.error ? reject(new Error(event.error.message)) : resolve(event.result);
return;
}
if (event.method && this.listeners.has(event.method)) {
for (const fn of this.listeners.get(event.method)) fn(event.params);
}
}
async evaluate(expression) {
const r = await this.send('Runtime.evaluate', {
expression,
awaitPromise: true,
returnByValue: true,
});
if (r.exceptionDetails) {
throw new Error(r.exceptionDetails.exception?.description || r.exceptionDetails.text);
}
return r.result.value;
}
close() { try { this.ws.close(); } catch {} }
}
const chrome = findChrome();
if (!chrome) {
console.error('No Chrome/Chromium found. Set CHROME=/path/to/chrome');
process.exit(2);
}
// Watchdog: a render check must never hang a CI runner.
const watchdog = setTimeout(() => {
console.error('HEADLESS CHECK FAILED: watchdog expired after 120s');
process.exit(1);
}, 120000);
watchdog.unref();
const server = await serveStatic();
const profile = mkdtempSync(join(tmpdir(), 'headless-check-'));
const args = [
'--headless=new',
'--no-sandbox',
'--disable-gpu-sandbox',
'--use-angle=swiftshader',
'--enable-unsafe-swiftshader',
`--remote-debugging-port=${CDP_PORT}`,
`--user-data-dir=${profile}`,
'--no-first-run',
'--no-default-browser-check',
'about:blank',
];
const proc = spawn(chrome, args, { stdio: ['ignore', 'ignore', 'pipe'] });
let stderr = '';
proc.stderr.on('data', (d) => { stderr += d.toString(); });
let exitCode = 1;
try {
await waitForCDP(CDP_PORT);
const targets = await (await fetch(`http://127.0.0.1:${CDP_PORT}/json/list`)).json();
const page = targets.find((t) => t.type === 'page');
if (!page) throw new Error('no page target');
const cdp = new CDP(page.webSocketDebuggerUrl);
await cdp.open();
await cdp.send('Page.enable');
await cdp.send('Runtime.enable');
// Hook getContext before the page's own scripts run, so we can read back the
// game's real WebGL context.
await cdp.send('Page.addScriptToEvaluateOnNewDocument', {
source: `
(function () {
const orig = HTMLCanvasElement.prototype.getContext;
window.__gl = null;
HTMLCanvasElement.prototype.getContext = function (type, ...rest) {
const c = orig.call(this, type, ...rest);
if (c && (type === 'webgl2' || type === 'webgl') && this.id === 'game-canvas') window.__gl = c;
return c;
};
window.__shaderErrors = [];
window.addEventListener('error', (e) => window.__shaderErrors.push(String(e.message)));
})();
`,
});
const consoleErrors = [];
cdp.on('Runtime.consoleAPICalled', (p) => {
if (p.type === 'error') consoleErrors.push(p.args.map((a) => a.value ?? a.description ?? '').join(' '));
});
cdp.on('Runtime.exceptionThrown', (p) => consoleErrors.push(p.exceptionDetails.exception?.description || p.exceptionDetails.text));
await cdp.send('Page.navigate', { url: `http://127.0.0.1:${PORT}/` });
await sleep(9000);
const sample = () => cdp.evaluate(`(async () => {
await new Promise(r => requestAnimationFrame(() => requestAnimationFrame(r)));
const gl = window.__gl;
if (!gl) return { err: 'no webgl context captured' };
const w = gl.drawingBufferWidth, h = gl.drawingBufferHeight;
const px = new Uint8Array(w * h * 4);
gl.readPixels(0, 0, w, h, gl.RGBA, gl.UNSIGNED_BYTE, px);
const colors = new Map();
let nonBlack = 0;
for (let i = 0; i < px.length; i += 4) {
if (px[i] || px[i+1] || px[i+2]) nonBlack++;
const k = (px[i] >> 4) + ',' + (px[i+1] >> 4) + ',' + (px[i+2] >> 4);
colors.set(k, (colors.get(k) || 0) + 1);
}
let hash = 0;
for (let i = 0; i < px.length; i += 16) hash = (hash * 31 + px[i] * 3 + px[i+1] * 7 + px[i+2] * 11) >>> 0;
return { buffer: w + 'x' + h, distinctColors: colors.size, nonBlack, total: w * h, hash,
topColors: [...colors.entries()].sort((a,b) => b[1]-a[1]).slice(0,5).map(([k,v]) => 'rgb(' + k + ') x' + v) };
})()`);
const menu = await cdp.evaluate(`document.body.innerText.replace(/\\s+/g,' ').slice(0,80)`);
const idle = await sample();
// drive the aircraft: full throttle, then rotate
const VK = { Enter: 13, ArrowUp: 38, ArrowDown: 40, ArrowLeft: 37, ArrowRight: 39, KeyW: 87, KeyS: 83, KeyC: 67, Escape: 27 };
const key = async (type, code) => {
await cdp.send('Input.dispatchKeyEvent', {
type,
key: code === 'Enter' ? 'Enter' : code.replace(/^Key/, '').toLowerCase().replace(/^Arrow/, 'Arrow'),
code,
windowsVirtualKeyCode: VK[code],
nativeVirtualKeyCode: VK[code],
});
};
const press = async (code) => { await key('keyDown', code); await key('keyUp', code); };
await press('Enter');
await sleep(2500);
await press('Enter');
await sleep(5000);
for (let i = 0; i < 15; i++) { await press('KeyW'); await sleep(120); }
await key('keyDown', 'ArrowUp');
await sleep(3000);
await key('keyUp', 'ArrowUp');
await sleep(1500);
const flown = await sample();
const shaderErrors = consoleErrors.filter((e) => /Shader Error|not compiled|WebGLProgram/i.test(e));
const report = {
chrome: chrome.replace(/^.*\/chrome-linux64\//, ''),
menuText: menu,
idle,
flown,
shaderErrors,
consoleErrors: consoleErrors.filter((e) => !/Shader Error|not compiled|WebGLProgram/i.test(e)).slice(0, 5),
};
console.log(JSON.stringify(report, null, 2));
const failures = [];
if (shaderErrors.length) failures.push(`shader compile errors: ${shaderErrors.length}`);
if (idle.err) failures.push(idle.err);
if (!idle.err && idle.distinctColors < 8) failures.push(`canvas looks blank: only ${idle.distinctColors} distinct colours`);
if (!idle.err && idle.nonBlack / idle.total < 0.5) failures.push(`canvas mostly black: ${idle.nonBlack}/${idle.total} lit pixels`);
if (!flown.err && flown.hash === idle.hash) failures.push('scene did not change after throttle + pitch input');
if (failures.length) {
console.error('\nHEADLESS CHECK FAILED:\n ' + failures.join('\n '));
exitCode = 1;
} else {
console.log('\nHEADLESS CHECK PASSED');
exitCode = 0;
}
cdp.close();
} catch (e) {
console.error('headless check errored:', e.message);
if (stderr) console.error('chrome stderr:', stderr.slice(-800));
exitCode = 1;
} finally {
proc.kill();
server.close();
try { rmSync(profile, { recursive: true, force: true }); } catch {}
}
process.exit(exitCode);
+95
View File
@@ -0,0 +1,95 @@
import { test, describe } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync, writeFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import { tmpdir } from 'node:os';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
// Compiles the game's real GLSL with Khronos' reference compiler (glslangValidator).
// This is the test that would have caught the vec4(vec3, float, float) bug: the shader
// failed to compile in the browser and the 3D canvas rendered pure black.
const root = join(dirname(fileURLToPath(import.meta.url)), '..');
const retroSrc = readFileSync(join(root, 'js', 'retro.js'), 'utf8');
function glslangAvailable() {
try {
execFileSync('glslangValidator', ['--version'], { stdio: ['ignore', 'ignore', 'ignore'] });
return true;
} catch {
return false;
}
}
// three.js prepends these declarations for a GLSL ES 1.00 ShaderMaterial.
const VERTEX_PREFIX = `precision highp float;
attribute vec3 position;
attribute vec3 normal;
attribute vec2 uv;
uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;
uniform mat3 normalMatrix;
`;
const FRAGMENT_PREFIX = 'precision highp float;\n';
function extractShader(source, name) {
const re = new RegExp(`${name}:\\s*\`([\\s\\S]*?)\``, 'm');
const m = source.match(re);
if (!m) throw new Error(`could not extract ${name} from js/retro.js`);
return m[1];
}
function compile(source, stage, label) {
// glslang infers the shader stage from the file extension, so the extension matters.
const ext = stage === 'vert' ? 'vert' : 'frag';
const file = join(tmpdir(), `shader-test-${label}.${ext}`);
writeFileSync(file, `#version 100\n${source}`);
try {
execFileSync('glslangValidator', [file], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] });
return { ok: true, log: '' };
} catch (e) {
const log = `${e.stdout || ''}${e.stderr || ''}`;
// A usage banner means glslang never read the file: that is a harness fault, not a
// shader fault, and must not be mistaken for a compile error.
if (/^Usage: glslang/m.test(log.trim())) {
throw new Error(`glslang invoked incorrectly (no stage recognised) for ${label}.${ext}`);
}
return { ok: false, log };
}
}
describe('CRT post-processing shaders', () => {
const vertex = extractShader(retroSrc, 'vertexShader');
const fragment = extractShader(retroSrc, 'fragmentShader');
test('both shaders are present in js/retro.js', () => {
assert.ok(vertex.includes('gl_Position'), 'vertex shader has no gl_Position assignment');
assert.ok(fragment.includes('gl_FragColor'), 'fragment shader has no gl_FragColor assignment');
});
test('shaders compile as GLSL ES 1.00', { skip: glslangAvailable() ? false : 'glslangValidator not installed (apt: glslang-tools)' }, () => {
const v = compile(VERTEX_PREFIX + vertex, 'vert', 'vertex');
assert.ok(v.ok, `vertex shader failed to compile:\n${v.log}`);
const f = compile(FRAGMENT_PREFIX + fragment, 'frag', 'fragment');
assert.ok(f.ok, `fragment shader failed to compile:\n${f.log}`);
});
// Canary: proves the harness can actually reject a bad shader, so the test above
// cannot pass vacuously.
test('harness rejects a known-bad vec4 construction', { skip: glslangAvailable() ? false : 'glslangValidator not installed (apt: glslang-tools)' }, () => {
const bad = `${VERTEX_PREFIX}varying vec2 vUv;\nvoid main(){ vUv = uv; gl_Position = vec4(position, 0.0, 1.0); }`;
const r = compile(bad, 'vert', 'canary');
assert.equal(r.ok, false, 'canary shader unexpectedly compiled — the test harness is not validating anything');
assert.match(r.log, /constructor/, 'canary rejected for an unexpected reason');
});
test('vertex shader does not over-fill gl_Position', () => {
// position is a vec3 in three.js, so vec4(position, x, y) is 5 components into a vec4.
const offenders = vertex
.split('\n')
.map((line, i) => ({ line: line.trim(), i: i + 1 }))
.filter(l => /vec4\(\s*position\s*,\s*[\d.]+\s*,\s*[\d.]+\s*\)/.test(l.line));
assert.deepEqual(offenders, [], `vec4(vec3, float, float) is invalid GLSL ES: ${JSON.stringify(offenders)}`);
});
});
+80
View File
@@ -0,0 +1,80 @@
import { test, describe } from 'node:test';
import assert from 'node:assert/strict';
import { Terrain } from '../js/terrain.js';
describe('Terrain', () => {
test('heightfield is deterministic across instances (persistent worlds)', () => {
const a = new Terrain(2000, 100);
const b = new Terrain(2000, 100);
a.createMesh();
b.createMesh();
for (const [x, z] of [[0, 0], [317, -812], [-999, 999], [123.5, -45.25]]) {
assert.equal(a.getHeightAt(x, z), b.getHeightAt(x, z), `height differs at ${x},${z}`);
}
});
test('heightfield is deterministic across rebuilds of the same instance', () => {
const t = new Terrain(2000, 100);
t.createMesh();
const first = t.getHeightAt(317, -812);
t.createMesh();
assert.equal(t.getHeightAt(317, -812), first);
});
test('airport apron is flattened inside the radius and untouched outside it', () => {
const rough = new Terrain(2000, 100);
rough.buildHeightData(); // heightfield only, no flattening
const flat = new Terrain(2000, 100);
flat.createMesh(); // createMesh() flattens a 200 m radius
// The apron centre happens to be naturally flat in this world, so assert on the
// strongest reduction found anywhere inside the radius rather than on one point.
let best = null;
for (let x = -190; x <= 190; x += 5) {
for (let z = -190; z <= 190; z += 5) {
if (Math.hypot(x, z) > 190) continue;
const delta = rough.getHeightAt(x, z) - flat.getHeightAt(x, z);
if (!best || delta > best.delta) best = { x, z, delta };
}
}
assert.ok(best && best.delta > 1, `createMesh() did not flatten inside the 200 m radius (best delta: ${best && best.delta})`);
// Flattening must stop at the radius, not bleed into the wider world.
for (const [x, z] of [[210, 0], [0, 210], [300, 300], [-400, 250]]) {
assert.equal(flat.getHeightAt(x, z), rough.getHeightAt(x, z), `flattening leaked outside the radius at ${x},${z}`);
}
});
test('height outside the heightfield bounds is zero, not garbage', () => {
const t = new Terrain(2000, 100);
t.createMesh();
for (const [x, z] of [[1500, 0], [0, 1500], [-1500, -1500]]) {
assert.equal(t.getHeightAt(x, z), 0, `expected 0 outside bounds at ${x},${z}`);
}
});
test('heights stay within a sane range for a 2000 m world', () => {
const t = new Terrain(2000, 100);
t.createMesh();
let min = Infinity;
let max = 0;
for (let x = -990; x <= 990; x += 30) {
for (let z = -990; z <= 990; z += 30) {
const h = t.getHeightAt(x, z);
min = Math.min(min, h);
max = Math.max(max, h);
}
}
assert.ok(min >= 0, `negative terrain height: ${min}`);
assert.ok(max > 10 && max < 200, `implausible max terrain height: ${max}`);
});
test('mesh has one vertex colour per vertex', () => {
const t = new Terrain(2000, 50);
const mesh = t.createMesh();
const pos = mesh.geometry.attributes.position;
const col = mesh.geometry.attributes.color;
assert.equal(col.count, pos.count, 'colour attribute count does not match position count');
assert.equal(col.itemSize, 3);
});
});