From Question to Evidence
The Student Affairs Data Lake creates the foundation for connecting evidence across programs, systems, and contexts. A Program Impact Project puts that foundation to work by connecting a consequential question to multiple sources of evidence to understand outcomes, inform decisions, improve practice, and communicate value. The framework strengthens consistency without forcing every project into the same method.
PROJECT STANDARD: Begin with the decision and intended use. Select the evidence and method only after the question, population, theory, and comparison strategy are clear.
What decision are we trying to inform?
Define the question, why it matters, who will use the result, and what action could follow.
Consider: Decision or intended use, executive sponsor, program partner, target decision date.
Document: The question, intended use, and decision the evidence will inform.
How might the program contribute to the outcome?
Clarify the participant experience, intended outcomes, pathway of contribution, timing, and plausible alternatives.
Consider: What participants experience, what the program is expected to change, and what else could influence the outcome.
Document: Intended outcomes and the theory of contribution.
What evidence can help answer the question?
Identify the program, institutional, survey, qualitative, financial, and contextual evidence available.
Consider: What each source can tell you, what it cannot, and how sources can strengthen or challenge one another.
Document: Key evidence sources, owners, access needs, and limitations.
Who are we studying, and compared with whom?
Define the population, program exposure, time period, comparison strategy, and inclusion and exclusion rules.
Consider: Eligibility, cohorts, time at risk, exposure definitions, and known differences between groups.
Document: Population, period, exposure, and comparison strategy.
Can we trust the evidence for this question?
Confirm definitions, completeness, level of detail, linkage, time logic, reconciliation, and fitness for the intended use.
Consider: Whether records represent students, activities, or events; missingness; duplicate records; join behavior; and changes in definitions over time.
Document: Validation tests, exceptions, limitations, and fitness for use.
What approach best answers the question?
Select a method proportionate to the question and available evidence, and make assumptions explicit.
Consider: Quantitative and qualitative approaches, covariates or matching variables, sample-size considerations, missing-data strategies, and robustness checks.
Document: Method, assumptions, and analytical decisions.
What can we reasonably conclude?
Distinguish what the evidence shows from what it may suggest, what remains uncertain, and what context may help explain the findings.
Consider: Results, comparisons, adjusted findings, contextual or qualitative evidence, limitations, and the boundary between association and causation.
Interpretation ladder:
Describe → Compare → Adjust → Explain cautiously → Act
Document: Findings, limitations, contextual explanation, and appropriate language for dissemination.
What should happen because of what we learned?
Identify decisions, improvements, investments, communications, or follow-up evidence.
Consider: Who owns the response, what will change, when it will happen, and what evidence will show whether the response worked.
Document: Decision, action, owner, timing, and implementation evidence.
What happens next?
Document action, monitor implementation, refresh evidence, and identify the next question.
Consider: What changed, what was learned, whether the intervention or improvement had the intended effect, and what evidence is needed next.
Document: Follow-up outcomes, refresh plans, and the next evidence need.
Closing the Loop: A project is not complete when the report is delivered. It is complete when the decision, action, or learning is documented and the next evidence need is clear.