Purpose of this document. The CS3IP Project Brief is a snapshot of the current technical architecture and is frozen. This diary is different. It is a chronological log of how that architecture got here, what was decided, what research or external input changed it, and why, updated after each milestone. This diary itself has no frozen state and never will. It is intended to double as raw material for the CS3IP report's process/methodology narrative. Tickets (project FYP) are cross-linked inline in the sub-pages below where a log entry maps to a specific ticket, and FYP-21 collects the project's key reference links (Brief, Diary, GitHub, board) in one place.
| Student | Waleed Ahmed, 240048282 | Supervisor | Ziyang Wang |
|---|---|---|---|
| Diary started | 25/09/2026 | Last log entry | 05/10/2026 |
<aside> 🗂️
The 54 dated log entries live in the six sub-pages below, split by theme and kept in their original chronological order within each. This page keeps the read-first material: current status and open items.
</aside>
| Area | Status |
|---|---|
| Project Definition Form | Sent to Ziyang for greenlight, unchanged. Working design realigned to match it. |
| Evaluation scope/rigor question | Resolved 27/09/2026. Ziyang replied and confirmed the more rigorous evaluation design is worth keeping (FYP-12, Done). I am proceeding with a hierarchical-versus-flat team comparison on identical tasks, scored by an LLM judge against a fixed rubric, alongside a small human evaluation study reusing the same tasks and configurations. See the 27/09/2026 log entry for the full account. |
| TheAgentCompany infrastructure scope | Resolved 25/09/2026. Cherry-pick the 69 Software Engineering task definitions and their weighted partial-credit scoring formula, rather than standing up the full GitLab/Plane/RocketChat/ownCloud stack. |
| LLM model selection per tier | Resolved 25/09/2026. Reframed as a deliberate config parameter (RoleConfig.model), not a document fact. |
get_channel_history(n) retrieval method |
Resolved 25/09/2026. Naive last-N, not recency/importance/relevance-weighted retrieval. Implementation-level, not written into the brief. |
| MAST failure-tagging mechanism | Resolved 25/09/2026. Hybrid: rule-based where the orchestrator signal already makes the code unambiguous, LLM-judge for subtler categories. Implementation-level, not written into the brief. |
| Implementation, Layer 1 | Core Pydantic models and ticket FSM. Commit 59f9ee4 (FYP-14). |
| Implementation, Layer 2 | Role/settings config, loader, and pyproject.toml. Commit e4a57c4 (FYP-15). |
| Implementation, Layer 3 | Orchestrator loop: SQLite event log, Agent/AgentContext/AgentResponse interface, step/run/resume control flow with retry-cap bookkeeping and intervention-mode pausing. First layer with a passing test suite. Commit e240cef (FYP-16). |
| Implementation, Layer 4 | Sandbox controller: SandboxController interface (run_tests/run_command/read_file/write_file/git_diff/git_commit), Docker-backed DockerAttemptSandbox (one container per ticket-attempt), FakeSandboxController for tests, host-side GitWorkspace. awaiting_test changed to a deterministic system step rather than agent self-report. Commit 6494242 (FYP-17). |
| Implementation, Layer 5 | Dashboard: FastAPI backend with REST endpoints for tickets, a ticket's thread, and the unfiltered feed, plus a websocket that polls the event log and pushes new rows live. Static chat feed and kanban frontend, no build step. Commit 07e56b9 (FYP-18). |
| Implementation, Layer 6 (Evaluation harness, FYP-19) | Complete. All 5 layers done: task/checkpoint models and the TheAgentCompany scoring formula (Layer A, commit a649062); FAIL_TO_PASS/PASS_TO_PASS hidden-test isolation (Layer B, commit 18997c6, bug fix ac8fd92); the hybrid MAST failure tagger (Layer C, commit 828b37c); the single-agent baseline runner and score comparison (Layer D, commit d6e0132); and the pipeline that ties it together plus two faithfully ported real TheAgentCompany tasks and a list/show CLI (Layer E, commit 69d8a4a). 79/79 tests passing (2 self-skip, pending Docker access/FYP-22). Real end-to-end runs still need a real LLM-backed Agent (deferred) and Docker access, but the harness itself is ready for both. |
| Project Brief | Finalized 25/09/2026 as the foundational architecture reference. No open design questions remain in it. |
| Project poster (FYP-45 to FYP-48) | Draft built, measured for symmetry and exported as a PDF on 05/10/2026. A space is held open for the human evaluation study. Not for printing or submission yet: it is waiting on the first supervisor meeting on 08/10/2026, 3:30pm to 4pm, for the poster's requirements and for ethics approval (FYP-49 to FYP-52). The current text and files are on the Poster page. |
newgrp docker to apply to already-running shells, not a further fix. See FYP-22 for the critical git_commit() bug this uncovered once real runs became possible.sde-* (47 tasks), found while researching real tasks to port for Layer E. The earlier note about no clean prefix existing was itself an oversight in that first research pass, not a fact about the repo. This still doesn't exactly match the paper's stated 69-task count, so full reconciliation would still need the paper's appendix. That is not needed for the 2 tasks already ported (both drawn directly from the confirmed sde-* set), only if more get ported later.