tests/stress: add Node-based adversarial stress harness
Two suites under tests/stress/, plus a tmux-friendly run_loop.sh runner. Both boot a fresh uvicorn on an isolated DB per cycle and log JSON line summaries to runs/. api_stress.mjs covers WS-level scenarios that the existing pytest suite does not exercise: 20-student happy path, late joiners with correct remaining_ms, mid-question disconnect, browser-sleep + wake to a different question_idx, cookie tampering and cross-session cookie reuse, duplicate student_id, bad submit (out-of-order, wrong idx, resubmit no-op), close-boundary race with auto-close, malformed JSON fuzz, and flaky reconnect. ui_stress.mjs drives the same flows in a real Chromium context via playwright: happy UI, sleep/wake by closing+reopening a context with the persisted cookie, document.cookie tampering attempt, and two browser contexts joining with the same student_id. Findings will be summarised in runs/summary.jsonl over time. One known issue surfaces from the fuzz scenario: app/room.py student_ws's receive_json call propagates JSONDecodeError out of the only try/except (which catches WebSocketDisconnect), killing that client's WS handler. Other clients are unaffected.
This commit is contained in:
ameer
2
tests/stress/.gitignore
vendored
Normal file
2
tests/stress/.gitignore
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
node_modules/
|
||||
runs/
|
||||
34
tests/stress/README.md
Normal file
34
tests/stress/README.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# Quiz portal stress harness
|
||||
|
||||
Adversarial frontend + API stress tests for the quiz portal. Built 2026-05-02.
|
||||
|
||||
## Files
|
||||
|
||||
- `lib.mjs` — shared helpers: server boot, cookie jar, `Student` and `Admin` WS wrappers, the fixed `STRESS_POOL`.
|
||||
- `api_stress.mjs` — pure WS adversarial scenarios (no browser): happy path with 20 concurrent students, late join, mid-question disconnect, **sleep/wake to next question** (the phone-screen-sleep scenario), cookie tampering, cross-session cookie reuse, duplicate student_id, bad submits (out-of-order, wrong idx, resubmit), close-boundary race, malformed-JSON fuzz, flaky reconnect.
|
||||
- `ui_stress.mjs` — Playwright/Chromium scenarios that exercise the real SPA: happy UI flow, sleep/wake by closing+reopening browser context with persisted cookie, cookie-tamper via `document.cookie`, two browsers with same student_id.
|
||||
- `run_loop.sh` — bash wrapper that runs `api_stress.mjs` every cycle and `ui_stress.mjs` every `UI_EVERY` cycles (default 5), with a fresh random seed each time. Logs JSON summary lines to `runs/summary.jsonl` and full output to `runs/run-<timestamp>.jsonl`.
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
# One-shot
|
||||
node api_stress.mjs # uses Date.now() seed
|
||||
node api_stress.mjs 12345 8210 # explicit seed + port
|
||||
node ui_stress.mjs # browser-based; HEADLESS=0 to watch
|
||||
|
||||
# Long-running loop in tmux
|
||||
tmux new -d -s quiz_stress 'cd /home/ameer/RD/Projects/Apps/quiz/tests/stress && bash run_loop.sh'
|
||||
tmux attach -t quiz_stress # to watch
|
||||
tmux send -t quiz_stress C-c # to stop
|
||||
```
|
||||
|
||||
Each cycle boots a fresh uvicorn on its own port and clean DB, runs scenarios, then tears down. Failures are recorded in the `failures` array of the per-cycle summary line.
|
||||
|
||||
## Known findings (tracked outside this dir)
|
||||
|
||||
- **Codex bug:** `app/room.py` `student_ws` (line ~87) and `instructor_ws` call `await websocket.receive_json()` whose JSON parsing can raise `JSONDecodeError`, but the surrounding `try/except` only catches `WebSocketDisconnect`. Result: a single malformed message kills that client's WS handler. The fuzz scenario in `api_stress.mjs` flags this consistently. Fix: wrap the receive in `try/except (JSONDecodeError, RuntimeError):` and either close cleanly or send `{"type":"error","code":"bad_message"}` and continue.
|
||||
|
||||
## Adding scenarios
|
||||
|
||||
Write an `async function name(server) { ... }` in `api_stress.mjs` (or `(server, browser)` for UI), add it to the `SCENARIOS` map / array, and re-run. Use `expect(cond, scenario, msg, extra)` for assertions and `note(scenario, msg)` for warnings that shouldn't fail the suite. **Critical pattern:** pre-register `waitFor` waiters BEFORE the action that triggers the message — `Student.waitFor(type)` only resolves on NEW messages, not cached ones, to avoid stale-state false passes.
|
||||
461
tests/stress/api_stress.mjs
Normal file
461
tests/stress/api_stress.mjs
Normal file
@@ -0,0 +1,461 @@
|
||||
// API-level adversarial stress tests for the quiz portal.
|
||||
// Each scenario boots a fresh server on its own port, runs assertions,
|
||||
// and logs JSON lines to stdout. Designed to be run repeatedly with
|
||||
// different seeds; see run_loop.sh for the wrapper.
|
||||
|
||||
import { bootServer, setupSession, Student, Admin, STRESS_POOL, sleep, logLine, rand, pickRandom } from "./lib.mjs";
|
||||
import WebSocket from "ws";
|
||||
|
||||
const SEED = parseInt(process.argv[2] || Date.now(), 10);
|
||||
const PORT = parseInt(process.argv[3] || (8200 + (SEED % 100)), 10);
|
||||
|
||||
let mulberry32 = (s) => () => {
|
||||
s |= 0; s = s + 0x6D2B79F5 | 0;
|
||||
let t = Math.imul(s ^ s >>> 15, 1 | s);
|
||||
t = t + Math.imul(t ^ t >>> 7, 61 | t) ^ t;
|
||||
return ((t ^ t >>> 14) >>> 0) / 4294967296;
|
||||
};
|
||||
const rng = mulberry32(SEED);
|
||||
|
||||
let pass = 0, fail = 0, warn = 0;
|
||||
const failures = [];
|
||||
|
||||
function expect(cond, scenario, msg, extra = {}) {
|
||||
if (cond) { pass++; logLine(scenario, "pass", msg, extra); }
|
||||
else { fail++; failures.push({ scenario, msg, extra }); logLine(scenario, "fail", msg, extra); }
|
||||
}
|
||||
function note(scenario, msg, extra = {}) { warn++; logLine(scenario, "note", msg, extra); }
|
||||
|
||||
async function runScenario(name, fn) {
|
||||
logLine(name, "start", `seed=${SEED}`);
|
||||
let server;
|
||||
try {
|
||||
server = await bootServer({ port: PORT });
|
||||
await fn(server);
|
||||
logLine(name, "ok", "scenario completed");
|
||||
} catch (err) {
|
||||
fail++;
|
||||
failures.push({ scenario: name, msg: "uncaught", extra: { err: err.message, stack: err.stack?.slice(0, 600) } });
|
||||
logLine(name, "fail", "uncaught exception", { err: err.message });
|
||||
} finally {
|
||||
if (server) await server.stop();
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- Scenarios ----------
|
||||
|
||||
async function happyPath(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const N = 20;
|
||||
const students = await Promise.all(Array.from({ length: N }, async (_, i) => {
|
||||
const s = new Student(server.url, sid, `S${i.toString().padStart(3, "0")}`, `Student${i}`);
|
||||
await s.join();
|
||||
await s.connect();
|
||||
return s;
|
||||
}));
|
||||
const admin = new Admin(server.url, sid, jar);
|
||||
await admin.connect();
|
||||
await admin.waitFor("lobby_update");
|
||||
|
||||
for (let q = 0; q < STRESS_POOL.questions.length; q++) {
|
||||
// Pre-register waiters so we don't lose the broadcast in the race window
|
||||
const studentOpenWaits = students.map(s => s.waitFor("question_open"));
|
||||
const adminOpenWait = admin.waitFor("question_open");
|
||||
admin.open(q, 5);
|
||||
await adminOpenWait;
|
||||
await Promise.all(studentOpenWaits);
|
||||
// Each student picks a random answer (mostly correct)
|
||||
await Promise.all(students.map(async (s, i) => {
|
||||
try {
|
||||
await sleep(rand(50, 800));
|
||||
const correct = STRESS_POOL.questions[q].correct;
|
||||
const ans = rng() < 0.7 ? correct : pickRandom(["A","B","C","D"]);
|
||||
const ackWait = s.waitFor("submit_ack", { timeoutMs: 3000 });
|
||||
s.submit(q, ans);
|
||||
const ack = await ackWait;
|
||||
expect(ack.question_idx === q, "happy", `student${i} q${q} ack idx==q`, { ack_idx: ack.question_idx, expected: q });
|
||||
} catch (e) {
|
||||
note("happy", `student${i} q${q}: ${e.message}`);
|
||||
}
|
||||
}));
|
||||
const studentClosedWaits = students.map(s => s.waitFor("question_closed", { timeoutMs: 3000 }).catch(() => null));
|
||||
const adminClosedWait = admin.waitFor("question_closed", { timeoutMs: 3000 });
|
||||
admin.close();
|
||||
await adminClosedWait;
|
||||
await Promise.all(studentClosedWaits);
|
||||
if (q < STRESS_POOL.questions.length - 1) {
|
||||
admin.next();
|
||||
await sleep(150);
|
||||
}
|
||||
}
|
||||
const sessionEndedWait = admin.waitFor("session_ended", { timeoutMs: 3000 });
|
||||
admin.end();
|
||||
await sessionEndedWait;
|
||||
expect(true, "happy", "session ended cleanly");
|
||||
students.forEach(s => s.disconnect());
|
||||
admin.disconnect();
|
||||
}
|
||||
|
||||
async function lateJoiners(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar);
|
||||
await admin.connect();
|
||||
// 5 students join in lobby
|
||||
const early = await Promise.all([0,1,2,3,4].map(async i => {
|
||||
const s = new Student(server.url, sid, `E${i}`, `Early${i}`);
|
||||
await s.join(); await s.connect(); return s;
|
||||
}));
|
||||
const adminOpenWait = admin.waitFor("question_open");
|
||||
admin.open(0, 8);
|
||||
await adminOpenWait;
|
||||
await sleep(2000);
|
||||
// 3 late joiners
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const s = new Student(server.url, sid, `L${i}`, `Late${i}`);
|
||||
await s.join();
|
||||
// Pre-register waiters BEFORE connect so we catch the snapshot on connect
|
||||
const stateWait = s.waitFor("state");
|
||||
const qopenWait = s.waitFor("question_open", { timeoutMs: 2000 });
|
||||
await s.connect();
|
||||
const m = await qopenWait.catch(() => null);
|
||||
if (!m) { fail++; failures.push({ scenario: "late_join", msg: `late${i} got no question_open on connect`, extra: {} }); logLine("late_join", "fail", `late${i} got no question_open on connect`); continue; }
|
||||
expect(m.question_idx === 0, "late_join", `late${i} sees correct idx`);
|
||||
expect(m.remaining_ms < 8000, "late_join", `late${i} remaining_ms reduced`, { remaining_ms: m.remaining_ms });
|
||||
expect(m.remaining_ms > 0, "late_join", `late${i} remaining_ms > 0`, { remaining_ms: m.remaining_ms });
|
||||
const ackWait = s.waitFor("submit_ack", { timeoutMs: 3000 });
|
||||
s.submit(0, STRESS_POOL.questions[0].correct);
|
||||
const ack = await ackWait.catch(() => null);
|
||||
expect(ack && ack.score > 0, "late_join", `late${i} got positive score from late submit`, { score: ack?.score });
|
||||
s.disconnect();
|
||||
}
|
||||
const adminClosedWait = admin.waitFor("question_closed");
|
||||
admin.close();
|
||||
await adminClosedWait;
|
||||
early.forEach(s => s.disconnect());
|
||||
admin.disconnect();
|
||||
}
|
||||
|
||||
async function disconnectMidQuestion(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar);
|
||||
await admin.connect();
|
||||
const s = new Student(server.url, sid, "D1", "Drop");
|
||||
await s.join();
|
||||
const stateWaitInitial = s.waitFor("state");
|
||||
await s.connect();
|
||||
await stateWaitInitial;
|
||||
const sQopenWait = s.waitFor("question_open");
|
||||
const aOpenWait = admin.waitFor("question_open");
|
||||
admin.open(0, 10);
|
||||
await aOpenWait;
|
||||
await sQopenWait;
|
||||
s.disconnect();
|
||||
await sleep(500);
|
||||
// While dropped, instructor closes
|
||||
const closedWait = admin.waitFor("question_closed");
|
||||
admin.close();
|
||||
await closedWait;
|
||||
// Reconnect — should get state=question_closed (or current state)
|
||||
const reconnectStateWait = s.waitFor("state", { timeoutMs: 2000 });
|
||||
await s.reconnect();
|
||||
const state = await reconnectStateWait;
|
||||
expect(["question_closed", "between_questions", "lobby"].includes(state.state), "disconnect_midq", `state on reconnect = ${state.state}`);
|
||||
s.disconnect();
|
||||
admin.disconnect();
|
||||
}
|
||||
|
||||
// THE KEY scenario from the user: phone screen sleeps mid-quiz, instructor
|
||||
// advances to a new question, phone wakes — the student MUST see the
|
||||
// LATEST question, not a stale screen.
|
||||
async function sleepWakeNextQuestion(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar);
|
||||
await admin.connect();
|
||||
const s = new Student(server.url, sid, "SW1", "SleepWake");
|
||||
await s.join();
|
||||
const initState = s.waitFor("state");
|
||||
await s.connect();
|
||||
await initState;
|
||||
// Open Q0
|
||||
let sWait = s.waitFor("question_open");
|
||||
let aWait = admin.waitFor("question_open");
|
||||
admin.open(0, 5);
|
||||
await aWait; await sWait;
|
||||
await sleep(500);
|
||||
const ackWait = s.waitFor("submit_ack");
|
||||
s.submit(0, "B");
|
||||
await ackWait;
|
||||
// Phone "sleeps" — drop WS hard
|
||||
s.disconnect();
|
||||
await sleep(800);
|
||||
// Instructor closes, advances, opens Q2 (skip Q1 to make wake state non-trivial)
|
||||
let closedW = admin.waitFor("question_closed");
|
||||
admin.close();
|
||||
await closedW;
|
||||
admin.next();
|
||||
await sleep(150);
|
||||
aWait = admin.waitFor("question_open");
|
||||
admin.open(2, 8);
|
||||
await aWait;
|
||||
await sleep(300);
|
||||
// Phone wakes — reconnect; pre-register both expected events
|
||||
const stateOnWake = s.waitFor("state", { timeoutMs: 2000 });
|
||||
const qopenOnWake = s.waitFor("question_open", { timeoutMs: 2000 });
|
||||
await s.reconnect();
|
||||
const state = await stateOnWake;
|
||||
expect(state.state === "question_open", "sleep_wake", `state on wake = ${state.state}`, { state });
|
||||
expect(state.current_question_idx === 2, "sleep_wake", `current_question_idx on wake = ${state.current_question_idx}`);
|
||||
const qopen = await qopenOnWake.catch(() => null);
|
||||
expect(qopen && qopen.question_idx === 2, "sleep_wake", "reconnect emits question_open for current idx", { qopen_idx: qopen?.question_idx });
|
||||
expect(qopen && qopen.text === STRESS_POOL.questions[2].text, "sleep_wake", "question text matches latest Q");
|
||||
if (qopen) {
|
||||
const ack2W = s.waitFor("submit_ack", { timeoutMs: 3000 });
|
||||
s.submit(2, "B");
|
||||
const ack = await ack2W.catch(() => null);
|
||||
expect(ack && ack.question_idx === 2, "sleep_wake", "submit on Q2 after wake succeeded", { ack_idx: ack?.question_idx });
|
||||
}
|
||||
s.disconnect();
|
||||
admin.disconnect();
|
||||
}
|
||||
|
||||
// Cookie tampering: try to flip student_id by mangling the signed cookie
|
||||
async function cookieTampering(server) {
|
||||
const { sid } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const s = new Student(server.url, sid, "ORIG", "Original");
|
||||
await s.join();
|
||||
// Tamper: append a junk byte
|
||||
const original = s.jar.get("qz_student");
|
||||
s.jar.set("qz_student", original + "X");
|
||||
// Try to connect WS
|
||||
let connected = false, closeCode = null;
|
||||
try {
|
||||
await new Promise((res, rej) => {
|
||||
const wsUrl = server.url.replace(/^http/, "ws") + `/ws/student/${sid}`;
|
||||
const w = new WebSocket(wsUrl, { headers: { Cookie: s.jar.header() } });
|
||||
w.on("open", () => { connected = true; w.close(); res(); });
|
||||
w.on("close", (c) => { closeCode = c; res(); });
|
||||
w.on("error", () => res());
|
||||
setTimeout(res, 1500);
|
||||
});
|
||||
} catch {}
|
||||
expect(!connected || closeCode === 4001 || closeCode === 1006 || closeCode === 1008, "cookie_tamper", "tampered cookie rejected on WS", { connected, closeCode });
|
||||
// Reset cookie and retry
|
||||
s.jar.set("qz_student", original);
|
||||
await s.connect();
|
||||
await s.waitFor("state");
|
||||
expect(true, "cookie_tamper", "valid cookie still works after tamper attempt");
|
||||
s.disconnect();
|
||||
}
|
||||
|
||||
// Cross-session cookie: cookie from session A should not work on session B.
|
||||
async function crossSessionCookie(server) {
|
||||
const { sid: sidA, jar: jarA } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const { sid: sidB } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const s = new Student(server.url, sidA, "X1", "CrossUser");
|
||||
await s.join();
|
||||
// Try to use sidA's cookie to access sidB
|
||||
const wsUrl = server.url.replace(/^http/, "ws") + `/ws/student/${sidB}`;
|
||||
let opened = false;
|
||||
await new Promise(res => {
|
||||
const w = new WebSocket(wsUrl, { headers: { Cookie: s.jar.header() } });
|
||||
w.on("open", () => { opened = true; w.close(); res(); });
|
||||
w.on("close", () => res());
|
||||
w.on("error", () => res());
|
||||
setTimeout(res, 1500);
|
||||
});
|
||||
expect(!opened, "cross_session", "cookie from sidA rejected when used against sidB", { opened });
|
||||
}
|
||||
|
||||
// Duplicate student_id: two browsers join with same student_id (different cookies)
|
||||
async function duplicateStudentId(server) {
|
||||
const { sid } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const s1 = new Student(server.url, sid, "DUP", "FirstSession");
|
||||
const s2 = new Student(server.url, sid, "DUP", "SecondSession");
|
||||
await s1.join();
|
||||
const s1State = s1.waitFor("state");
|
||||
await s1.connect();
|
||||
await s1State;
|
||||
await s2.join();
|
||||
const s2State = s2.waitFor("state");
|
||||
await s2.connect();
|
||||
await s2State;
|
||||
expect(s1.ws.readyState === WebSocket.OPEN, "dup_id", "first DUP session open");
|
||||
expect(s2.ws.readyState === WebSocket.OPEN, "dup_id", "second DUP session open");
|
||||
s1.disconnect(); s2.disconnect();
|
||||
}
|
||||
|
||||
// Submit out-of-order / wrong question_idx
|
||||
async function badSubmits(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar); await admin.connect();
|
||||
const s = new Student(server.url, sid, "BAD", "Bad");
|
||||
await s.join();
|
||||
const initState = s.waitFor("state");
|
||||
await s.connect();
|
||||
await initState;
|
||||
// Submit before any question opened
|
||||
s.submit(0, "A");
|
||||
await sleep(300);
|
||||
expect(!s.lastMsgByType.submit_ack, "bad_submit", "submit before open ignored / no ack", { last: Object.keys(s.lastMsgByType) });
|
||||
let sWait = s.waitFor("question_open");
|
||||
let aWait = admin.waitFor("question_open");
|
||||
admin.open(0, 5);
|
||||
await aWait; await sWait;
|
||||
// Wrong idx submit (give it 600 ms to arrive if it does)
|
||||
const ackBefore = s.lastMsgByType.submit_ack;
|
||||
s.submit(99, "A");
|
||||
await sleep(600);
|
||||
const ackAfter = s.lastMsgByType.submit_ack;
|
||||
expect(ackAfter === ackBefore || (ackAfter && ackAfter.question_idx !== 99), "bad_submit", "wrong-idx submit not acked", { ackAfter });
|
||||
// Valid submit
|
||||
const okWait = s.waitFor("submit_ack", { timeoutMs: 2000 });
|
||||
s.submit(0, "B");
|
||||
const ok = await okWait;
|
||||
expect(ok.question_idx === 0, "bad_submit", "valid submit acked");
|
||||
// Resubmit (already submitted) — should NOT change stored answer
|
||||
s.submit(0, "A");
|
||||
await sleep(400);
|
||||
const closedWait = s.waitFor("question_closed");
|
||||
admin.close();
|
||||
const closed = await closedWait;
|
||||
expect(closed.your_answer === "B", "bad_submit", "your_answer remained 'B' after resubmit attempt", { your_answer: closed.your_answer });
|
||||
s.disconnect(); admin.disconnect();
|
||||
}
|
||||
|
||||
// Race: many students submit at the moment of close (within ms)
|
||||
async function closeBoundaryRace(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar); await admin.connect();
|
||||
const N = 30;
|
||||
const students = await Promise.all(Array.from({ length: N }, async (_, i) => {
|
||||
const s = new Student(server.url, sid, `R${i}`, `Race${i}`);
|
||||
await s.join(); await s.connect(); return s;
|
||||
}));
|
||||
// Pre-register everyone's question_open + question_closed waits
|
||||
const sOpens = students.map(s => s.waitFor("question_open", { timeoutMs: 3000 }).catch(() => null));
|
||||
const aOpen = admin.waitFor("question_open", { timeoutMs: 3000 });
|
||||
// Pre-register the closed wait too — auto-close fires even without manual close()
|
||||
const aClosed = admin.waitFor("question_closed", { timeoutMs: 6000 });
|
||||
admin.open(0, 1); // 1-second window — auto-close should fire ~1s later
|
||||
await aOpen;
|
||||
await Promise.all(sOpens);
|
||||
// Have all students fire submit at random times spanning the window edge
|
||||
const fires = students.map(async (s, i) => {
|
||||
const ackW = s.waitFor("submit_ack", { timeoutMs: 2000 });
|
||||
await sleep(rand(800, 1200)); // some fire after auto-close
|
||||
s.submit(0, "B");
|
||||
return ackW.catch(() => null);
|
||||
});
|
||||
const acks = await Promise.all(fires);
|
||||
const acked = acks.filter(Boolean).length;
|
||||
const closed = await aClosed.catch(() => null);
|
||||
logLine("close_race", "info", `race results`, { acked, total: N, hist: closed?.histogram });
|
||||
expect(closed !== null, "close_race", "question_closed broadcast received (auto-close or manual)");
|
||||
expect(acked >= 1 && acked <= N, "close_race", "no crash, some submits succeeded");
|
||||
students.forEach(s => s.disconnect()); admin.disconnect();
|
||||
}
|
||||
|
||||
// Fuzz: malformed messages from a student WS
|
||||
async function fuzzMessages(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar); await admin.connect();
|
||||
// First: a sentinel student that does NOT receive fuzz, to verify global server health
|
||||
const sentinel = new Student(server.url, sid, "SENT", "Sentinel");
|
||||
await sentinel.join();
|
||||
const sentinelStateW = sentinel.waitFor("state");
|
||||
await sentinel.connect();
|
||||
await sentinelStateW;
|
||||
// Second: a fuzz student that gets garbage shoved at it
|
||||
const s = new Student(server.url, sid, "FZ", "Fuzz");
|
||||
await s.join();
|
||||
const fzStateW = s.waitFor("state");
|
||||
await s.connect();
|
||||
await fzStateW;
|
||||
const garbages = [
|
||||
"not json",
|
||||
"{}",
|
||||
JSON.stringify({ type: "open_question" }), // student trying to act as instructor
|
||||
JSON.stringify({ type: "submit", question_idx: -1, answer: "Z" }),
|
||||
JSON.stringify({ type: "submit", question_idx: 0, answer: { nested: "obj" } }),
|
||||
JSON.stringify({ type: "💀" }),
|
||||
"x".repeat(50_000),
|
||||
];
|
||||
for (const g of garbages) { try { s.ws.send(g); } catch {} ; await sleep(50); }
|
||||
await sleep(500);
|
||||
// Server should still serve OTHER clients regardless of what happened to fuzz student.
|
||||
const sentOpenW = sentinel.waitFor("question_open", { timeoutMs: 2000 });
|
||||
const adminOpenW = admin.waitFor("question_open", { timeoutMs: 2000 });
|
||||
admin.open(0, 3);
|
||||
await adminOpenW;
|
||||
const sm = await sentOpenW.catch(() => null);
|
||||
expect(sm && sm.question_idx === 0, "fuzz", "OTHER clients still served after fuzz on one student", { got: !!sm });
|
||||
// Did the fuzz student survive? (informational, not asserted as pass/fail)
|
||||
const survived = !s.closed && s.ws.readyState === WebSocket.OPEN;
|
||||
logLine("fuzz", "info", `fuzz student survival`, { survived, ws_state: s.ws.readyState });
|
||||
if (!survived) note("fuzz", "fuzz student WS was killed by malformed input — server lacks JSON-decode try/except in WS loop (codex room.py student_ws line ~87)");
|
||||
const adminClosedW = admin.waitFor("question_closed", { timeoutMs: 5000 });
|
||||
admin.close();
|
||||
await adminClosedW.catch(() => null);
|
||||
s.disconnect(); sentinel.disconnect(); admin.disconnect();
|
||||
}
|
||||
|
||||
// Repeated rapid connect/disconnect (simulating flaky network)
|
||||
async function flakyReconnect(server) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const admin = new Admin(server.url, sid, jar); await admin.connect();
|
||||
const s = new Student(server.url, sid, "FK", "Flaky");
|
||||
await s.join();
|
||||
for (let i = 0; i < 10; i++) {
|
||||
const stateW = s.waitFor("state", { timeoutMs: 2000 });
|
||||
await s.connect();
|
||||
await stateW.catch(() => {});
|
||||
s.disconnect();
|
||||
await sleep(rand(50, 200));
|
||||
}
|
||||
// Final reconnect
|
||||
const finalStateW = s.waitFor("state", { timeoutMs: 2000 });
|
||||
await s.connect();
|
||||
await finalStateW;
|
||||
const sQopenW = s.waitFor("question_open");
|
||||
const aQopenW = admin.waitFor("question_open");
|
||||
admin.open(0, 5);
|
||||
await aQopenW; await sQopenW;
|
||||
const ackW = s.waitFor("submit_ack", { timeoutMs: 3000 });
|
||||
s.submit(0, "B");
|
||||
const ack = await ackW;
|
||||
expect(ack !== null, "flaky_reconnect", "post-flaky student can still submit");
|
||||
const closedW = admin.waitFor("question_closed");
|
||||
admin.close();
|
||||
await closedW;
|
||||
s.disconnect(); admin.disconnect();
|
||||
}
|
||||
|
||||
const SCENARIOS = {
|
||||
happy: happyPath,
|
||||
late_join: lateJoiners,
|
||||
disconnect_midq: disconnectMidQuestion,
|
||||
sleep_wake: sleepWakeNextQuestion,
|
||||
cookie_tamper: cookieTampering,
|
||||
cross_session: crossSessionCookie,
|
||||
dup_id: duplicateStudentId,
|
||||
bad_submit: badSubmits,
|
||||
close_race: closeBoundaryRace,
|
||||
fuzz: fuzzMessages,
|
||||
flaky_reconnect: flakyReconnect,
|
||||
};
|
||||
|
||||
// Pick scenario subset based on env or run all in random order
|
||||
const wanted = process.env.SCENARIOS
|
||||
? process.env.SCENARIOS.split(",")
|
||||
: Object.keys(SCENARIOS).sort(() => rng() - 0.5);
|
||||
|
||||
logLine("runner", "info", `starting api stress`, { seed: SEED, port: PORT, scenarios: wanted });
|
||||
|
||||
for (const name of wanted) {
|
||||
const fn = SCENARIOS[name];
|
||||
if (!fn) { logLine("runner", "warn", `unknown scenario ${name}`); continue; }
|
||||
await runScenario(name, fn);
|
||||
}
|
||||
|
||||
logLine("runner", "summary", `done`, { pass, fail, warn, failures });
|
||||
process.exit(fail > 0 ? 1 : 0);
|
||||
256
tests/stress/lib.mjs
Normal file
256
tests/stress/lib.mjs
Normal file
@@ -0,0 +1,256 @@
|
||||
// Shared helpers for quiz portal stress tests.
|
||||
// Boots a fresh uvicorn server, logs in as admin, creates quiz + session.
|
||||
// Provides a Student class that wraps an authenticated WS + cookie state.
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { setTimeout as sleep } from "node:timers/promises";
|
||||
import WebSocket from "ws";
|
||||
|
||||
const QUIZ_ROOT = "/home/ameer/RD/Projects/Apps/quiz";
|
||||
const PORT_BASE = 8200;
|
||||
|
||||
export function nowMs() { return Date.now(); }
|
||||
export function logLine(scenario, level, msg, extra = {}) {
|
||||
const rec = { ts: new Date().toISOString(), scenario, level, msg, ...extra };
|
||||
process.stdout.write(JSON.stringify(rec) + "\n");
|
||||
}
|
||||
|
||||
export function pickRandom(arr) { return arr[Math.floor(Math.random() * arr.length)]; }
|
||||
export function rand(min, max) { return Math.random() * (max - min) + min; }
|
||||
|
||||
// Boot a fresh server on its own port + DB. Returns { url, stop }.
|
||||
export async function bootServer({ port, secret = "stress-secret-12345678", adminPw = "stresspw" } = {}) {
|
||||
const tmp = mkdtempSync(join(tmpdir(), "quiz-stress-"));
|
||||
const dbPath = join(tmp, "stress.db");
|
||||
const env = {
|
||||
...process.env,
|
||||
QUIZ_DB_PATH: dbPath,
|
||||
QUIZ_SECRET_KEY: secret,
|
||||
QUIZ_ADMIN_PASSWORD: adminPw,
|
||||
QUIZ_HOST: "127.0.0.1",
|
||||
QUIZ_PORT: String(port),
|
||||
QUIZ_PUBLIC_URL: `http://127.0.0.1:${port}`,
|
||||
};
|
||||
const proc = spawn(
|
||||
`${QUIZ_ROOT}/.venv/bin/uvicorn`,
|
||||
["app.main:app", "--host", "127.0.0.1", "--port", String(port), "--log-level", "warning"],
|
||||
{ cwd: QUIZ_ROOT, env, stdio: ["ignore", "pipe", "pipe"] },
|
||||
);
|
||||
// Pipe server stderr to our stderr so panics are visible
|
||||
proc.stderr.on("data", chunk => process.stderr.write(`[server:${port}] ${chunk}`));
|
||||
|
||||
const url = `http://127.0.0.1:${port}`;
|
||||
// Wait for /healthz
|
||||
const deadline = Date.now() + 20_000;
|
||||
while (Date.now() < deadline) {
|
||||
try {
|
||||
const r = await fetch(`${url}/healthz`);
|
||||
if (r.ok) {
|
||||
return {
|
||||
url,
|
||||
adminPw,
|
||||
stop: () => new Promise(res => {
|
||||
proc.once("exit", () => { rmSync(tmp, { recursive: true, force: true }); res(); });
|
||||
proc.kill("SIGTERM");
|
||||
// Hard-kill fallback
|
||||
setTimeout(() => proc.kill("SIGKILL"), 2000);
|
||||
}),
|
||||
};
|
||||
}
|
||||
} catch {}
|
||||
await sleep(150);
|
||||
}
|
||||
proc.kill("SIGKILL");
|
||||
throw new Error(`server on ${port} did not come up`);
|
||||
}
|
||||
|
||||
// Cookie jar helper - parses Set-Cookie headers from fetch response.
|
||||
export class CookieJar {
|
||||
constructor() { this.jar = new Map(); }
|
||||
ingest(response) {
|
||||
const raw = response.headers.getSetCookie?.() || (response.headers.get("set-cookie") ? [response.headers.get("set-cookie")] : []);
|
||||
for (const line of raw) {
|
||||
const [pair] = line.split(";");
|
||||
const eq = pair.indexOf("=");
|
||||
if (eq > 0) this.jar.set(pair.slice(0, eq).trim(), pair.slice(eq + 1).trim());
|
||||
}
|
||||
}
|
||||
header() {
|
||||
return [...this.jar.entries()].map(([k, v]) => `${k}=${v}`).join("; ");
|
||||
}
|
||||
get(name) { return this.jar.get(name); }
|
||||
set(name, value) { this.jar.set(name, value); }
|
||||
clear() { this.jar.clear(); }
|
||||
}
|
||||
|
||||
export async function jsonReq(method, url, { jar, body, headers = {} } = {}) {
|
||||
const opts = { method, headers: { ...headers } };
|
||||
if (jar) opts.headers["Cookie"] = jar.header();
|
||||
if (body !== undefined) {
|
||||
opts.body = JSON.stringify(body);
|
||||
opts.headers["Content-Type"] = "application/json";
|
||||
}
|
||||
const r = await fetch(url, opts);
|
||||
if (jar) jar.ingest(r);
|
||||
let data = null;
|
||||
const txt = await r.text();
|
||||
try { data = txt ? JSON.parse(txt) : null; } catch { data = txt; }
|
||||
return { status: r.status, ok: r.ok, data, headers: r.headers };
|
||||
}
|
||||
|
||||
// Build admin session: login + upload pool + create session. Returns { sid, jar }.
|
||||
export async function setupSession(serverUrl, adminPw, pool) {
|
||||
const jar = new CookieJar();
|
||||
const login = await jsonReq("POST", `${serverUrl}/admin/login`, { jar, body: { password: adminPw } });
|
||||
if (!login.ok) throw new Error(`admin login failed: ${login.status} ${JSON.stringify(login.data)}`);
|
||||
const create = await jsonReq("POST", `${serverUrl}/admin/api/quizzes`, { jar, body: { pool_json: pool } });
|
||||
if (!create.ok) throw new Error(`quiz create failed: ${create.status} ${JSON.stringify(create.data)}`);
|
||||
const sess = await jsonReq("POST", `${serverUrl}/admin/api/sessions`, { jar, body: { quiz_id: create.data.quiz_id } });
|
||||
if (!sess.ok) throw new Error(`session create failed: ${sess.status} ${JSON.stringify(sess.data)}`);
|
||||
return { sid: sess.data.sid, jar };
|
||||
}
|
||||
|
||||
// Student wrapper: join + connect WS + collect messages.
|
||||
export class Student {
|
||||
constructor(serverUrl, sid, studentId, name) {
|
||||
this.serverUrl = serverUrl;
|
||||
this.sid = sid;
|
||||
this.studentId = studentId;
|
||||
this.name = name;
|
||||
this.jar = new CookieJar();
|
||||
this.ws = null;
|
||||
this.messages = [];
|
||||
this.lastMsgByType = {};
|
||||
this.events = new EventTarget();
|
||||
this.closed = false;
|
||||
}
|
||||
async join() {
|
||||
const r = await jsonReq("POST", `${this.serverUrl}/api/session/${this.sid}/join`, {
|
||||
jar: this.jar,
|
||||
body: { student_id: this.studentId, name: this.name },
|
||||
});
|
||||
if (!r.ok) throw new Error(`student join failed: ${r.status} ${JSON.stringify(r.data)}`);
|
||||
return r;
|
||||
}
|
||||
connect() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const wsUrl = this.serverUrl.replace(/^http/, "ws") + `/ws/student/${this.sid}`;
|
||||
const headers = { Cookie: this.jar.header() };
|
||||
this.ws = new WebSocket(wsUrl, { headers });
|
||||
this.ws.on("open", () => resolve());
|
||||
this.ws.on("message", buf => {
|
||||
let msg;
|
||||
try { msg = JSON.parse(buf.toString()); } catch { return; }
|
||||
this.messages.push({ ts: nowMs(), msg });
|
||||
this.lastMsgByType[msg.type] = msg;
|
||||
this.events.dispatchEvent(new CustomEvent("msg", { detail: msg }));
|
||||
});
|
||||
this.ws.on("close", (code, reason) => {
|
||||
this.closed = true;
|
||||
this.events.dispatchEvent(new CustomEvent("close", { detail: { code, reason: reason.toString() } }));
|
||||
});
|
||||
this.ws.on("error", err => reject(err));
|
||||
});
|
||||
}
|
||||
// Wait until a NEW message of the given type arrives (does not use cache).
|
||||
// Use lastMsgByType[type] to inspect cached values without waiting.
|
||||
waitFor(type, { timeoutMs = 5000, useCache = false } = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
if (useCache && this.lastMsgByType[type]) return resolve(this.lastMsgByType[type]);
|
||||
const handler = ev => {
|
||||
if (ev.detail?.type === type) {
|
||||
this.events.removeEventListener("msg", handler);
|
||||
clearTimeout(timer);
|
||||
resolve(ev.detail);
|
||||
}
|
||||
};
|
||||
const timer = setTimeout(() => {
|
||||
this.events.removeEventListener("msg", handler);
|
||||
reject(new Error(`timed out waiting for WS type=${type} after ${timeoutMs}ms`));
|
||||
}, timeoutMs);
|
||||
this.events.addEventListener("msg", handler);
|
||||
});
|
||||
}
|
||||
send(obj) {
|
||||
if (this.ws?.readyState === WebSocket.OPEN) this.ws.send(JSON.stringify(obj));
|
||||
}
|
||||
submit(qIdx, answer) {
|
||||
this.send({ type: "submit", question_idx: qIdx, answer });
|
||||
}
|
||||
disconnect() {
|
||||
if (this.ws && !this.closed) {
|
||||
try { this.ws.terminate(); } catch {}
|
||||
}
|
||||
}
|
||||
async reconnect() {
|
||||
this.closed = false;
|
||||
this.lastMsgByType = {};
|
||||
await this.connect();
|
||||
}
|
||||
}
|
||||
|
||||
// Admin WS wrapper.
|
||||
export class Admin {
|
||||
constructor(serverUrl, sid, jar) {
|
||||
this.serverUrl = serverUrl;
|
||||
this.sid = sid;
|
||||
this.jar = jar;
|
||||
this.ws = null;
|
||||
this.messages = [];
|
||||
this.events = new EventTarget();
|
||||
}
|
||||
connect() {
|
||||
return new Promise((resolve, reject) => {
|
||||
const wsUrl = this.serverUrl.replace(/^http/, "ws") + `/ws/instructor/${this.sid}`;
|
||||
this.ws = new WebSocket(wsUrl, { headers: { Cookie: this.jar.header() } });
|
||||
this.ws.on("open", () => resolve());
|
||||
this.ws.on("message", buf => {
|
||||
let msg;
|
||||
try { msg = JSON.parse(buf.toString()); } catch { return; }
|
||||
this.messages.push({ ts: nowMs(), msg });
|
||||
this.events.dispatchEvent(new CustomEvent("msg", { detail: msg }));
|
||||
});
|
||||
this.ws.on("error", err => reject(err));
|
||||
});
|
||||
}
|
||||
open(qIdx, timeLimit = 60) { this.ws.send(JSON.stringify({ type: "open_question", question_idx: qIdx, time_limit: timeLimit })); }
|
||||
close() { this.ws.send(JSON.stringify({ type: "close_question" })); }
|
||||
next() { this.ws.send(JSON.stringify({ type: "next" })); }
|
||||
end() { this.ws.send(JSON.stringify({ type: "end_session" })); }
|
||||
waitFor(type, { timeoutMs = 5000 } = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const handler = ev => {
|
||||
if (ev.detail?.type === type) {
|
||||
this.events.removeEventListener("msg", handler);
|
||||
clearTimeout(timer);
|
||||
resolve(ev.detail);
|
||||
}
|
||||
};
|
||||
const timer = setTimeout(() => {
|
||||
this.events.removeEventListener("msg", handler);
|
||||
reject(new Error(`admin timed out waiting for type=${type} after ${timeoutMs}ms`));
|
||||
}, timeoutMs);
|
||||
this.events.addEventListener("msg", handler);
|
||||
});
|
||||
}
|
||||
disconnect() { try { this.ws?.terminate(); } catch {} }
|
||||
}
|
||||
|
||||
// A small fixed pool used for stress runs.
|
||||
export const STRESS_POOL = {
|
||||
title: "Stress Pool",
|
||||
score_fn: "linear_decay",
|
||||
time_limit_default: 10,
|
||||
questions: [
|
||||
{ id: "s1", text: "2+2?", options: { A: "3", B: "4", C: "5", D: "6" }, correct: "B", explanation: "" },
|
||||
{ id: "s2", text: "Capital of France?", options: { A: "Berlin", B: "Madrid", C: "Paris", D: "Rome" }, correct: "C", explanation: "" },
|
||||
{ id: "s3", text: "Fastest sort?", options: { A: "Bubble", B: "Quick", C: "Insertion", D: "Selection" }, correct: "B", explanation: "" },
|
||||
{ id: "s4", text: "HTTP code for not found?", options: { A: "200", B: "301", C: "404", D: "500" }, correct: "C", explanation: "" },
|
||||
{ id: "s5", text: "Speed of light (m/s)?", options: { A: "3e8", B: "3e6", C: "1.5e8", D: "9.8" }, correct: "A", explanation: "" },
|
||||
],
|
||||
};
|
||||
|
||||
export { sleep };
|
||||
81
tests/stress/package-lock.json
generated
Normal file
81
tests/stress/package-lock.json
generated
Normal file
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"name": "quiz-stress",
|
||||
"version": "0.1.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "quiz-stress",
|
||||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"playwright": "^1.58.2",
|
||||
"ws": "^8.18.0"
|
||||
}
|
||||
},
|
||||
"node_modules/fsevents": {
|
||||
"version": "2.3.2",
|
||||
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
|
||||
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"os": [
|
||||
"darwin"
|
||||
],
|
||||
"engines": {
|
||||
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/playwright": {
|
||||
"version": "1.59.1",
|
||||
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.59.1.tgz",
|
||||
"integrity": "sha512-C8oWjPR3F81yljW9o5OxcWzfh6avkVwDD2VYdwIGqTkl+OGFISgypqzfu7dOe4QNLL2aqcWBmI3PMtLIK233lw==",
|
||||
"license": "Apache-2.0",
|
||||
"dependencies": {
|
||||
"playwright-core": "1.59.1"
|
||||
},
|
||||
"bin": {
|
||||
"playwright": "cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"fsevents": "2.3.2"
|
||||
}
|
||||
},
|
||||
"node_modules/playwright-core": {
|
||||
"version": "1.59.1",
|
||||
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.59.1.tgz",
|
||||
"integrity": "sha512-HBV/RJg81z5BiiZ9yPzIiClYV/QMsDCKUyogwH9p3MCP6IYjUFu/MActgYAvK0oWyV9NlwM3GLBjADyWgydVyg==",
|
||||
"license": "Apache-2.0",
|
||||
"bin": {
|
||||
"playwright-core": "cli.js"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
}
|
||||
},
|
||||
"node_modules/ws": {
|
||||
"version": "8.20.0",
|
||||
"resolved": "https://registry.npmjs.org/ws/-/ws-8.20.0.tgz",
|
||||
"integrity": "sha512-sAt8BhgNbzCtgGbt2OxmpuryO63ZoDk/sqaB/znQm94T4fCEsy/yV+7CdC1kJhOU9lboAEU7R3kquuycDoibVA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=10.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"bufferutil": "^4.0.1",
|
||||
"utf-8-validate": ">=5.0.2"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"bufferutil": {
|
||||
"optional": true
|
||||
},
|
||||
"utf-8-validate": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
14
tests/stress/package.json
Normal file
14
tests/stress/package.json
Normal file
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"name": "quiz-stress",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"api": "node api_stress.mjs",
|
||||
"ui": "node ui_stress.mjs"
|
||||
},
|
||||
"dependencies": {
|
||||
"ws": "^8.18.0",
|
||||
"playwright": "^1.58.2"
|
||||
}
|
||||
}
|
||||
66
tests/stress/run_loop.sh
Executable file
66
tests/stress/run_loop.sh
Executable file
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env bash
|
||||
# Loop runner for the stress harness.
|
||||
# Runs api_stress.mjs each cycle with a fresh random seed, and runs ui_stress.mjs
|
||||
# every $UI_EVERY cycles (default 5). Logs JSON lines to runs/<timestamp>.jsonl.
|
||||
# Run this in tmux: tmux new -d -s quiz_stress 'bash run_loop.sh'
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
mkdir -p runs
|
||||
|
||||
UI_EVERY=${UI_EVERY:-5}
|
||||
SLEEP_BETWEEN=${SLEEP_BETWEEN:-3}
|
||||
LOG="runs/run-$(date -u +%Y%m%dT%H%M%SZ).jsonl"
|
||||
SUM="runs/summary.jsonl"
|
||||
|
||||
echo "{\"event\":\"loop_start\",\"ts\":\"$(date -u +%FT%TZ)\",\"log\":\"$LOG\",\"ui_every\":$UI_EVERY}" | tee -a "$SUM"
|
||||
|
||||
cycle=0
|
||||
total_pass=0
|
||||
total_fail=0
|
||||
total_warn=0
|
||||
|
||||
trap 'echo "{\"event\":\"loop_stop\",\"ts\":\"$(date -u +%FT%TZ)\",\"cycles\":'$cycle',\"total_pass\":'$total_pass',\"total_fail\":'$total_fail',\"total_warn\":'$total_warn'}" | tee -a "$SUM"; exit 0' INT TERM
|
||||
|
||||
while true; do
|
||||
cycle=$((cycle + 1))
|
||||
seed=$(( (RANDOM * 32768 + RANDOM) % 1000000 ))
|
||||
port=$((8200 + (cycle % 50)))
|
||||
|
||||
printf '\n----- cycle %d (seed=%d port=%d) api -----\n' "$cycle" "$seed" "$port" | tee -a "$LOG"
|
||||
out=$(timeout 120 node api_stress.mjs "$seed" "$port" 2>&1)
|
||||
echo "$out" | tee -a "$LOG" >/dev/null
|
||||
summary=$(echo "$out" | grep '"runner"' | grep '"summary"' | tail -1)
|
||||
if [ -n "$summary" ]; then
|
||||
p=$(echo "$summary" | sed -n 's/.*"pass":\([0-9]*\).*/\1/p')
|
||||
f=$(echo "$summary" | sed -n 's/.*"fail":\([0-9]*\).*/\1/p')
|
||||
w=$(echo "$summary" | sed -n 's/.*"warn":\([0-9]*\).*/\1/p')
|
||||
total_pass=$((total_pass + ${p:-0}))
|
||||
total_fail=$((total_fail + ${f:-0}))
|
||||
total_warn=$((total_warn + ${w:-0}))
|
||||
echo "{\"event\":\"cycle\",\"ts\":\"$(date -u +%FT%TZ)\",\"cycle\":$cycle,\"kind\":\"api\",\"seed\":$seed,\"port\":$port,\"pass\":${p:-0},\"fail\":${f:-0},\"warn\":${w:-0},\"running_pass\":$total_pass,\"running_fail\":$total_fail,\"running_warn\":$total_warn}" | tee -a "$SUM"
|
||||
else
|
||||
echo "{\"event\":\"cycle\",\"ts\":\"$(date -u +%FT%TZ)\",\"cycle\":$cycle,\"kind\":\"api\",\"seed\":$seed,\"port\":$port,\"status\":\"NO_SUMMARY\"}" | tee -a "$SUM"
|
||||
fi
|
||||
|
||||
if [ $((cycle % UI_EVERY)) -eq 0 ]; then
|
||||
printf '\n----- cycle %d (seed=%d port=%d) ui -----\n' "$cycle" "$seed" "$((port + 100))" | tee -a "$LOG"
|
||||
out=$(timeout 180 node ui_stress.mjs "$seed" "$((port + 100))" 2>&1)
|
||||
echo "$out" | tee -a "$LOG" >/dev/null
|
||||
summary=$(echo "$out" | grep '"runner"' | grep '"summary"' | tail -1)
|
||||
if [ -n "$summary" ]; then
|
||||
p=$(echo "$summary" | sed -n 's/.*"pass":\([0-9]*\).*/\1/p')
|
||||
f=$(echo "$summary" | sed -n 's/.*"fail":\([0-9]*\).*/\1/p')
|
||||
w=$(echo "$summary" | sed -n 's/.*"warn":\([0-9]*\).*/\1/p')
|
||||
total_pass=$((total_pass + ${p:-0}))
|
||||
total_fail=$((total_fail + ${f:-0}))
|
||||
total_warn=$((total_warn + ${w:-0}))
|
||||
echo "{\"event\":\"cycle\",\"ts\":\"$(date -u +%FT%TZ)\",\"cycle\":$cycle,\"kind\":\"ui\",\"seed\":$seed,\"port\":$((port + 100)),\"pass\":${p:-0},\"fail\":${f:-0},\"warn\":${w:-0},\"running_pass\":$total_pass,\"running_fail\":$total_fail,\"running_warn\":$total_warn}" | tee -a "$SUM"
|
||||
else
|
||||
echo "{\"event\":\"cycle\",\"ts\":\"$(date -u +%FT%TZ)\",\"cycle\":$cycle,\"kind\":\"ui\",\"seed\":$seed,\"port\":$((port + 100)),\"status\":\"NO_SUMMARY\"}" | tee -a "$SUM"
|
||||
fi
|
||||
fi
|
||||
|
||||
sleep "$SLEEP_BETWEEN"
|
||||
done
|
||||
181
tests/stress/ui_stress.mjs
Normal file
181
tests/stress/ui_stress.mjs
Normal file
@@ -0,0 +1,181 @@
|
||||
// UI-side stress: real Chromium browser contexts driving the SPA.
|
||||
// Tests scenarios that only matter at the JS layer:
|
||||
// - happy path through the student SPA UI
|
||||
// - sleep/wake (browser context closed mid-quiz, instructor advances, browser reopens)
|
||||
// - cookie tampering via document.cookie
|
||||
// - simultaneous browsers with same student_id
|
||||
// Boots its own server. Slower than api_stress but exercises real DOM rendering.
|
||||
|
||||
import { bootServer, setupSession, STRESS_POOL, sleep, logLine, jsonReq, CookieJar, Admin } from "./lib.mjs";
|
||||
import { chromium } from "playwright";
|
||||
|
||||
const SEED = parseInt(process.argv[2] || Date.now(), 10);
|
||||
const PORT = parseInt(process.argv[3] || (8300 + (SEED % 100)), 10);
|
||||
const HEADLESS = process.env.HEADLESS !== "0";
|
||||
|
||||
let pass = 0, fail = 0, warn = 0;
|
||||
const failures = [];
|
||||
|
||||
function expect(cond, scenario, msg, extra = {}) {
|
||||
if (cond) { pass++; logLine(scenario, "pass", msg, extra); }
|
||||
else { fail++; failures.push({ scenario, msg, extra }); logLine(scenario, "fail", msg, extra); }
|
||||
}
|
||||
function note(scenario, msg, extra = {}) { warn++; logLine(scenario, "note", msg, extra); }
|
||||
|
||||
async function joinAsStudent(page, baseUrl, sid, sid_id, name) {
|
||||
await page.goto(`${baseUrl}/?sid=${sid}`);
|
||||
await page.waitForSelector('input[name="student_id"], input[id*=student]', { timeout: 5000 });
|
||||
// Codex SPA uses input[name=student_id]
|
||||
const idInput = await page.$('input[name="student_id"]');
|
||||
const nameInput = await page.$('input[name="name"]');
|
||||
if (!idInput || !nameInput) throw new Error("join form fields not found");
|
||||
await idInput.fill(sid_id);
|
||||
await nameInput.fill(name);
|
||||
await page.click('button:has-text("Join")');
|
||||
await page.waitForSelector('text=Waiting for instructor', { timeout: 5000 });
|
||||
}
|
||||
|
||||
async function adminOpenQuestion(server, jar, sid, qIdx, timeLimit = 10) {
|
||||
// Open via admin instructor WS
|
||||
const admin = new Admin(server.url, sid, jar);
|
||||
await admin.connect();
|
||||
const w = admin.waitFor("question_open", { timeoutMs: 5000 });
|
||||
admin.open(qIdx, timeLimit);
|
||||
await w;
|
||||
return admin;
|
||||
}
|
||||
|
||||
// Scenario 1: happy path through the SPA
|
||||
async function uiHappy(server, browser) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const ctx = await browser.newContext();
|
||||
const page = await ctx.newPage();
|
||||
await joinAsStudent(page, server.url, sid, "U1", "UIStudent");
|
||||
const admin = await adminOpenQuestion(server, jar, sid, 0, 10);
|
||||
await page.waitForSelector('text=2+2?', { timeout: 5000 });
|
||||
expect(true, "ui_happy", "Q1 text rendered in browser");
|
||||
await page.click('button:has-text("B")');
|
||||
await page.waitForSelector('text=Submitted in', { timeout: 5000 });
|
||||
expect(true, "ui_happy", "submitted view shown");
|
||||
const closedW = admin.waitFor("question_closed", { timeoutMs: 5000 });
|
||||
admin.close();
|
||||
await closedW;
|
||||
await page.waitForSelector('text=Reveal', { timeout: 5000 });
|
||||
expect(true, "ui_happy", "reveal view shown");
|
||||
admin.disconnect();
|
||||
await ctx.close();
|
||||
}
|
||||
|
||||
// Scenario 2: sleep/wake via real browser context close-and-reopen, with persisted cookie.
|
||||
async function uiSleepWake(server, browser) {
|
||||
const { sid, jar } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const ctx1 = await browser.newContext();
|
||||
const page1 = await ctx1.newPage();
|
||||
await joinAsStudent(page1, server.url, sid, "U2", "Sleeper");
|
||||
// Capture the cookie so we can restore it in a fresh context (simulating phone-wake on same device)
|
||||
const cookies = await ctx1.cookies();
|
||||
const adminA = await adminOpenQuestion(server, jar, sid, 0, 10);
|
||||
await page1.waitForSelector('text=2+2?', { timeout: 5000 });
|
||||
await page1.click('button:has-text("B")');
|
||||
await page1.waitForSelector('text=Submitted in', { timeout: 5000 });
|
||||
// "Phone goes to sleep" — close the context entirely
|
||||
await ctx1.close();
|
||||
// Instructor closes, advances, opens Q2 (skip 1)
|
||||
let cw = adminA.waitFor("question_closed", { timeoutMs: 5000 });
|
||||
adminA.close();
|
||||
await cw;
|
||||
adminA.next();
|
||||
await sleep(150);
|
||||
let ow = adminA.waitFor("question_open", { timeoutMs: 5000 });
|
||||
adminA.open(2, 10);
|
||||
await ow;
|
||||
// "Phone wakes" — fresh context with same persisted cookie
|
||||
const ctx2 = await browser.newContext();
|
||||
await ctx2.addCookies(cookies);
|
||||
const page2 = await ctx2.newPage();
|
||||
await page2.goto(`${server.url}/?sid=${sid}`);
|
||||
// Should see Q3 (idx 2) text "Fastest sort?"
|
||||
try {
|
||||
await page2.waitForSelector('text=Fastest sort?', { timeout: 5000 });
|
||||
expect(true, "ui_sleep_wake", "browser shows the LATEST question after wake");
|
||||
} catch (e) {
|
||||
expect(false, "ui_sleep_wake", "browser did NOT show latest question after wake", { err: e.message });
|
||||
}
|
||||
// Try to submit on the new question
|
||||
try {
|
||||
await page2.click('button:has-text("B")');
|
||||
await page2.waitForSelector('text=Submitted in', { timeout: 5000 });
|
||||
expect(true, "ui_sleep_wake", "post-wake submit acked in UI");
|
||||
} catch (e) {
|
||||
expect(false, "ui_sleep_wake", "post-wake submit failed in UI", { err: e.message });
|
||||
}
|
||||
adminA.disconnect();
|
||||
await ctx2.close();
|
||||
}
|
||||
|
||||
// Scenario 3: cookie tampering via document.cookie (browser cookie is HttpOnly so this should be a no-op)
|
||||
async function uiCookieTamper(server, browser) {
|
||||
const { sid } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const ctx = await browser.newContext();
|
||||
const page = await ctx.newPage();
|
||||
await joinAsStudent(page, server.url, sid, "U3", "Tamper");
|
||||
// Confirm document.cookie does NOT see qz_student (HttpOnly)
|
||||
const visible = await page.evaluate(() => document.cookie);
|
||||
expect(!visible.includes("qz_student"), "ui_cookie_tamper", "qz_student is not visible to JS (HttpOnly verified)", { document_cookie: visible });
|
||||
// Try to overwrite (browsers will silently ignore HttpOnly overwrite from JS)
|
||||
await page.evaluate(() => { document.cookie = "qz_student=GARBAGE; path=/"; });
|
||||
await page.reload();
|
||||
// Should still be in lobby (cookie wasn't actually changed)
|
||||
try {
|
||||
await page.waitForSelector('text=Waiting for instructor', { timeout: 4000 });
|
||||
expect(true, "ui_cookie_tamper", "tamper attempt did not log student out");
|
||||
} catch (e) {
|
||||
note("ui_cookie_tamper", `tamper may have succeeded (lobby not re-rendered): ${e.message}`);
|
||||
}
|
||||
await ctx.close();
|
||||
}
|
||||
|
||||
// Scenario 4: two browser contexts with same student_id race (different cookies → 2 participants)
|
||||
async function uiDupId(server, browser) {
|
||||
const { sid } = await setupSession(server.url, server.adminPw, STRESS_POOL);
|
||||
const ctxA = await browser.newContext();
|
||||
const ctxB = await browser.newContext();
|
||||
const pA = await ctxA.newPage();
|
||||
const pB = await ctxB.newPage();
|
||||
await joinAsStudent(pA, server.url, sid, "DUPUI", "FirstBrowser");
|
||||
await joinAsStudent(pB, server.url, sid, "DUPUI", "SecondBrowser");
|
||||
expect(true, "ui_dup_id", "two browsers with same student_id both reach lobby");
|
||||
await ctxA.close(); await ctxB.close();
|
||||
}
|
||||
|
||||
const SCENARIOS = [
|
||||
["ui_happy", uiHappy],
|
||||
["ui_sleep_wake", uiSleepWake],
|
||||
["ui_cookie_tamper", uiCookieTamper],
|
||||
["ui_dup_id", uiDupId],
|
||||
];
|
||||
|
||||
logLine("runner", "info", "starting ui stress", { seed: SEED, port: PORT, headless: HEADLESS });
|
||||
|
||||
const browser = await chromium.launch({ headless: HEADLESS });
|
||||
let server = null;
|
||||
try {
|
||||
server = await bootServer({ port: PORT });
|
||||
for (const [name, fn] of SCENARIOS) {
|
||||
logLine(name, "start", `seed=${SEED}`);
|
||||
try {
|
||||
await fn(server, browser);
|
||||
logLine(name, "ok", "scenario completed");
|
||||
} catch (e) {
|
||||
fail++;
|
||||
failures.push({ scenario: name, msg: "uncaught", extra: { err: e.message } });
|
||||
logLine(name, "fail", "uncaught exception", { err: e.message });
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
if (server) await server.stop();
|
||||
await browser.close();
|
||||
}
|
||||
|
||||
logLine("runner", "summary", "done", { pass, fail, warn, failures });
|
||||
process.exit(fail > 0 ? 1 : 0);
|
||||
Reference in New Issue
Block a user