Vortex — Roofline Lab

\documentclass[conference]{IEEEtran}
\IEEEoverridecommandlockouts
\usepackage{cite}
\usepackage{amsmath,amssymb,amsfonts}
\usepackage{algorithmic}
\usepackage{graphicx}
\usepackage{textcomp}
\usepackage{xcolor}
\usepackage{hyperref}
\usepackage{booktabs}
\usepackage{array}
\usepackage{tabularx}
\newcolumntype{Y}{>{\raggedright\arraybackslash}X}
\hypersetup{hidelinks}
\begin{document}

\title{VORTEX Roofline Lab: An Evidence-Aware Architecture, Prototype Evaluation, and Integration Roadmap}
\author{\IEEEauthorblockN{Aaron Singh}
\IEEEauthorblockA{\textit{Department of Electrical and Computer Engineering} \\
\textit{San Francisco State University}}}
\maketitle

\begin{abstract}
This paper presents VORTEX Roofline Lab, a project in a twenty-project AI infrastructure portfolio. The current repository is an active prototype with implemented behavior, automated correctness tests, and explicit boundaries around unverified hardware or production claims. We describe the system model, current implementation, goals, and evaluation method, then propose five integrations that advance the project toward reproducible system-level validation. A local audit on August 14, 2026 executed 8 project tests successfully. No accelerator, deployment, or performance conclusion is inferred unless a corresponding committed benchmark or profiler artifact exists.
\end{abstract}

\begin{IEEEkeywords}
VORTEX, AI infrastructure, reproducibility, prototype validation, systems evaluation, hardware--software co-design
\end{IEEEkeywords}

\section{Introduction}
VORTEX Roofline Lab (P01) addresses a bounded problem within the portfolio's track-01-gpu-kernels track. Its declared status is \emph{Active Prototype} \cite{metadata}. The project validates its central mechanism before adding device-specific acceleration, production orchestration, or real-world hardware. This ordering matters because optimization without a trusted reference can make incorrect behavior appear successful.

The project goals are to establish deterministic behavior, encode correctness as tests, create machine-readable evidence, identify limiting resources through measurement, and integrate results with adjacent portfolio systems without losing provenance. The repository contains one paper named exactly after its singular folder: \texttt{P01-vortex-roofline-lab.tex}.

\section{Technical Context}
GPU kernel development requires a correct numerical reference before optimization. Reference equivalence, workload shape, warm-up policy, and profiler evidence are therefore first-class design concerns. The portfolio benchmark standard requires environment, workload, method, metric, artifact, reproduction, and limitation fields; unknown values remain explicitly unmeasured \cite{benchmark}.

\section{System Model and Architecture}
The prototype is organized around the following domain model:
\begin{equation}
I=F/B,\quad P_{attainable}=\min(P_{peak},BW\,I)
\end{equation}
The equation is a design and test abstraction rather than a claimed empirical law. It supports invariants and expected-value checks while later implementations replace synthetic inputs with representative workloads or devices.

The software architecture contains an input/configuration layer, a deterministic core, validation and evidence output, and a local visualization. The principal inspected source artifacts are \texttt{python/compare\_python\_numpy.py, python/run\_baseline.py}. Unsupported real-world conditions are surfaced as limitations rather than silently simulated.

\section{Detailed Script Operation and Rationale}
The baseline script creates deterministic vectors, computes element-wise addition with a readable Python reference, checks every output, measures repeated execution, and appends environment-aware JSONL evidence. The companion comparison script runs the same workload with Python lists and NumPy so implementation overhead can be studied before an XPU port.

The execution path is:
\begin{enumerate}
    \item Validate size, repetitions, warm-ups, and deterministic seed.
    \item Generate two repeatable input vectors and compute the Python reference output.
    \item Check output correctness before accepting any timing sample.
    \item Measure repeated runs and summarize latency and throughput.
    \item Attach Git/system metadata and append a JSONL record for review.
\end{enumerate}

\begin{table*}[t]
\caption{Implementation artifacts and why they exist}
\label{tab:p01-implementation}
\centering
\small
\begin{tabularx}{\textwidth}{p{0.24\textwidth}YY}
\toprule
\textbf{Artifact} & \textbf{Observed responsibility} & \textbf{Engineering rationale} \\
\midrule
python/compare\_python\_numpy.py & make\_inputs, measure, summarize, run\_comparison, build\_record, main & Implements the inspectable, unit-tested project core. \\
python/run\_baseline.py & vector\_add\_reference, run\_vector\_add, validate\_settings, output\_is\_correct, git\_value, git\_worktree\_is\_dirty, system\_memory\_gb, build\_record, main & Implements the inspectable, unit-tested project core. \\
scripts/reproduce.sh & Fixed test and demonstration entry point & Gives another developer one command for local reproduction. \\
PROJECT.yaml and ANALYSIS.md & Status, completed work, planned work, and claim boundaries & Separates declared intent from evidence-backed implementation. \\
streamlit\_app.py & Local evidence and status visualization & Makes outputs inspectable without upgrading simulation into a hardware claim. \\
\bottomrule
\end{tabularx}
\end{table*}

\section{Implemented Prototype}
The metadata and source audit found these completed features \cite{analysis}:
\begin{itemize}
    \item Beginner-readable Python CPU vector-add reference
    \item Deterministic synthetic input generation
    \item Correctness validation and unit tests
    \item JSONL benchmark evidence writer with Git metadata
    \item Optional Python-list versus NumPy comparison
\end{itemize}

On August 14, 2026, \texttt{python3 -m unittest discover -s tests -p 'test\_*.py'} completed successfully with 8 tests. The inspected test artifacts are \texttt{tests/unit/test\_python\_numpy\_comparison.py, tests/unit/test\_vector\_add\_reference.py}. This is evidence of local correctness for encoded cases, not production scale or hardware performance.

\section{Testing Methodology and Observed Results}
Testing uses Python's standard \texttt{unittest} discovery and exercises the public behavior of the reference implementation. The audit reran the suite from the project folder with \texttt{python3 -m unittest discover -s tests -p 'test\_*.py'}. All 8 discovered tests passed. The result establishes correctness only for the encoded local cases; it does not establish accelerator correctness, real-device behavior, production reliability, or benchmark completion.

\begin{table*}[t]
\caption{Audited test matrix}
\label{tab:p01-tests}
\centering
\scriptsize
\begin{tabularx}{\textwidth}{p{0.37\textwidth}Yp{0.21\textwidth}}
\toprule
\textbf{Test artifact and case} & \textbf{Behavior being checked} & \textbf{Observed result} \\
\midrule
tests/unit/test\_python\_numpy\_comparison.py:test\_inputs\_repeat\_with\_same\_seed & Inputs repeat with same seed. & Pass (local, 2026-08-14) \\
tests/unit/test\_python\_numpy\_comparison.py:test\_comparison\_passes\_correctness & Comparison passes correctness. & Pass (local, 2026-08-14) \\
tests/unit/test\_python\_numpy\_comparison.py:test\_summary\_contains\_throughput & Summary contains throughput. & Pass (local, 2026-08-14) \\
tests/unit/test\_vector\_add\_reference.py:test\_vector\_add\_reference & Vector add reference. & Pass (local, 2026-08-14) \\
tests/unit/test\_vector\_add\_reference.py:test\_vector\_lengths\_must\_match & Vector lengths must match. & Pass (local, 2026-08-14) \\
tests/unit/test\_vector\_add\_reference.py:test\_generated\_data\_is\_repeatable & Generated data is repeatable. & Pass (local, 2026-08-14) \\
tests/unit/test\_vector\_add\_reference.py:test\_output\_correctness\_check\_detects\_bad\_value & Output correctness check detects bad value. & Pass (local, 2026-08-14) \\
tests/unit/test\_vector\_add\_reference.py:test\_invalid\_benchmark\_settings & Invalid benchmark settings. & Pass (local, 2026-08-14) \\
\bottomrule
\end{tabularx}
\end{table*}

No numerical performance result is promoted by this test run. Where scripts emit JSON or JSONL, those outputs remain raw or simulation-specific until a reviewed summary includes hardware, software, workload, warm-up, repetition, correctness threshold, Git revision, and limitations.

\begin{table*}[t]
\caption{Declared status versus audited evidence}
\label{tab:p01-audit}
\centering
\small
\begin{tabularx}{\textwidth}{p{0.20\textwidth}Yp{0.25\textwidth}}
\toprule
\textbf{Audit field} & \textbf{Finding} & \textbf{Evidence source} \\
\midrule
Declared status & Active Prototype & PROJECT.yaml \\
Evidence-backed status & Active local prototype; 8 tests passed & Source plus local unittest run \\
Accepted measured results & None recorded in measured\_results & PROJECT.yaml \\
Mismatch / claim boundary & No XPU speedup or roofline result has been demonstrated yet & PROJECT.yaml and ANALYSIS.md \\
Next proof required & Intel XPU implementation; Profiler capture and roofline analysis & Planned features \\
\bottomrule
\end{tabularx}
\end{table*}

\section{Claim Boundaries and Risks}
The project records these unverified or excluded claims:
\begin{itemize}
    \item No XPU speedup or roofline result has been demonstrated yet
\end{itemize}
The main risk is confusing synthetic or modeled behavior with deployed-system behavior. Other risks include incomplete workloads, platform-dependent timing, missing failure injection, and interfaces not yet exercised across device boundaries. Performance claims require a reviewed record meeting the portfolio standard.

\section{Evaluation Plan}
Evaluation proceeds through correctness tests, deterministic reproduction with Git and environment metadata, repeated benchmarks reporting latency/throughput/memory/error metrics, and a named profiler capture tied to exact hardware and source revision. Success requires reference equivalence within a documented tolerance, preservation of safety and resource invariants, clear failures, and evidence reproducible from a clean environment.

\section{Goals, Milestones, and Success Criteria}
The project goals are staged so that correctness precedes performance and integration. A goal is complete only when its proof artifact is committed or otherwise reviewable; prose or a simulated number alone is insufficient.

\begin{table*}[t]
\caption{Project goals and required proof}
\label{tab:p01-goals}
\centering
\small
\begin{tabularx}{\textwidth}{p{0.06\textwidth}YY}
\toprule
\textbf{ID} & \textbf{Goal} & \textbf{Completion evidence} \\
\midrule
G1 & Intel XPU implementation & Passing tests and a reviewed source artifact \\
G2 & Profiler capture and roofline analysis & Machine-readable result with reproduction metadata \\
G3 & Publish a reviewed CPU baseline summary & Reference-equivalence or domain-correctness report \\
G4 & Implement reference-equivalent XPU execution & Named profiler, deployment, or integration artifact \\
\bottomrule
\end{tabularx}
\end{table*}

\section{Future Work and Integrations}
The five project-specific next steps are:
\begin{enumerate}
    \item Intel XPU implementation
    \item Profiler capture and roofline analysis
    \item Implement and validate equivalent SYCL and PyTorch XPU kernels.
    \item Publish a reviewed roofline report linked to a named profiler capture.
    \item Feed normalized benchmark summaries into P19 and P20.
\end{enumerate}
The early items complete declared evidence; the later items connect downstream portfolio consumers. Each integration should add tests and a reviewable artifact such as JSONL evidence, a report, profiler capture, deployment manifest, trace, or labeled data set.

\section{Conclusion}
VORTEX Roofline Lab is an evidence-aware active prototype: its implemented behavior and tests are real, while unbuilt hardware, deployment, and performance goals remain labeled. Completing the five integrations in dependency order will advance it from a learning artifact toward a credible portfolio component.

\begin{thebibliography}{00}
\bibitem{metadata} Aaron Singh, ``VORTEX Roofline Lab PROJECT.yaml,'' local portfolio repository, updated 2026-08-13.
\bibitem{analysis} Aaron Singh, ``VORTEX Roofline Lab: README, ANALYSIS, source, and test artifacts,'' local portfolio repository, accessed Aug. 14, 2026.
\bibitem{benchmark} Aaron Singh, ``AI Infrastructure Portfolio Benchmark Standard,'' local portfolio repository, accessed Aug. 14, 2026.
\end{thebibliography}
\end{document}