Objective:
The goal of this competition is to evaluate the reasoning capabilities of modern large language models (LLMs) in computer architecture and systems. Participants will submit challenging, reasoning-based questions, and the best questions will be tested against top AI reasoning models. The objective is to identify questions where AI fails, revealing the limits of LLM reasoning in computer architecture and systems.
Submission Guidelines:
-
We will only evaluate 30 questions per participant. If you submit more than 30, we will randomly select 30 for evaluation, and the rest will not be considered. Choose your best questions carefully to maximize your chance.
-
Note: We will use a text embedding mechanism to detect substantially similar questions across all submissions. In cases where similar questions are identified—whether from the same participant or from different participants—only the earliest submission will be considered, and duplicates may be filtered out by the judges.
-
Note: Submissions derived from exams, quizzes, or other copyrighted materials are acceptable only if you have obtained all necessary approvals and strictly adhere to your institution’s licensing policies. We reserve the right to disqualify winners if it is later discovered that any submitted questions violate these policies.
-
-
To ensure quality and accountability, you must submit your affiliation (e.g., academic institutions, research labs, or industry organizations) for your submission to be accepted.
-
Questions must meet the following criteria:
-
Clear and Concise: Formulate a clear, concise, and complex reasoning-based question related to computer architecture and systems.
-
Computer Architecture Specific: The question must be deeply rooted in architecture concepts (e.g., pipelining, branch prediction, cache behavior, or your own research).
-
Reasoning-Based: The question must require logical reasoning, step-by-step deduction, or multi-concept synthesis, rather than factual recall.
-
Non-Trivial: The question should not be directly answerable from standard textbooks or documentation without reasoning.
-
Accompanied by a Rationale: The participant must provide a clear correct answer with an explanation of why it is correct in the rationale textbox. Give a detailed, step-by-step explanation of why the answer is correct. Include citations to relevant references (e.g. textbooks, research papers, blogs, etc.)
-
AI Generated Questions: AI generated questions are not encouraged and can be removed at the judges’ discretion.
-
Why Participate?
-
Test the limits of AI reasoning in computer architecture.
-
Engage in a thought-provoking challenge of designing tough reasoning problems.
-
Earn recognition as an expert in designing hard architecture questions.
-
Top contributors will be acknowledged at MLArchSys 2025 and may have their questions featured in a future research paper or benchmark.
-
🏆 Winners of the competition will receive a monetary award at the workshop.
-