Data Scientist Interview Questions and Answers

I would start by writing the hypothesis as a sentence with a direction and a magnitude: the checklist raises week-one retention by at least [X] percentage points for new signups. The primary metric is week-one retained accounts, with guardrails on support contacts and time-to-first-value, because a checklist that annoys users could lift the headline metric while damaging the experience. I would randomize at the account level the moment a user starts onboarding, then check sample ratio mismatch before looking at any outcome. Power analysis decides the sample size and the runtime, and I would pre-register the readout plan so nobody fishes for a winning segment after the fact.

I look for novelty first in the time series, because a treatment effect that spikes in the first days and then decays is usually excitement rather than durable value. I would split the readout by cohort, comparing users exposed for the first time against users whose behavior has had time to settle, and I would compare the later window against the early one. If the effect survives only in the first days, I treat the steady-state estimate as the number to decide on and I say so explicitly. When the budget allows, I extend the holdback and re-read, because an honest slower answer beats a fast wrong one.

I would say that a p-value answers a narrow question: if the change truly did nothing, how surprising would the data we saw be? A small p-value means the result is unlikely under pure chance, not that the effect is large or certain, so I avoid saying the test proves it works. Then I bring it back to the decision. Our estimate is a lift of [X]% with a confidence interval that includes zero, so the data is consistent with no effect and also with a meaningful gain. If shipping is cheap and reversible, I might still recommend a limited rollout while we keep measuring.

I start from the decision the model feeds and the cost of each kind of mistake, because accuracy is meaningless until you know whether a false positive or a false negative hurts more. For a churn outreach list, precision at the top of the ranking matters more than global AUC, so I would set the bar on that slice. I check calibration, because a score of [X] should mean roughly [X] in a hundred if the team is going to set thresholds on it. Then I look at the worst-performing segment and compare the model against the current rule, including the option of changing nothing.

When randomization is off the table, I first write down the causal question and draw the graph, so I can see which variables confound the treatment and the outcome. Then I pick a design that fits the setting: difference-in-differences if I have a credible control group and pre-period, propensity-score matching if selection happens on observables, or an instrumental variable if something shifts treatment without touching the outcome. I test the assumptions rather than asserting them, for example checking pre-trends both visually and statistically. Finally I run a sensitivity analysis showing how large an unmeasured confounder would have to be to erase the effect, and I report the estimate with that caveat attached.

I let the audience decide. If a clinician or a regulator has to justify each decision, a logistic regression with clean coefficients may be worth more than a gradient-boosted model that is a few points better on AUC, because an unexplainable score simply will not be used. I always build the simple baseline first, so I can say exactly what the extra complexity buys. If the complex model earns its place, I use SHAP values or partial dependence to show which features drive a prediction, and I am honest that these are approximations of the model rather than the model's own reasoning.

I once shipped a churn model with target leakage: a feature that recorded a retention call was populated after the outcome I was predicting, so offline AUC looked excellent while the live ranking was useless. I found it when the outreach team reported that the top-ranked accounts had already been contacted. I owned it in the review, rebuilt the feature set with a strict as-of join, and re-ran the evaluation on a time-based split instead of a random one. The lasting change is that I now trace every feature's timestamp against the prediction moment and keep a holdout period the model never sees.

I start with the minimum detectable effect that would genuinely change our decision, because powering for a trivial lift wastes weeks of traffic. Then I estimate the baseline rate and variance from historical data and compute the sample size per arm at conventional significance and power levels. Dividing by daily eligible traffic gives the runtime, and I round up to whole weeks so the test covers full seasonal cycles. I write that runtime into the design document and commit to not peeking at the primary metric before it finishes, because early stopping on a noisy signal inflates the false positive rate. If traffic is too low, I say so and propose a switchback design instead.

With only a handful of treated regions, a standard difference-in-differences is fragile, so I would look at a synthetic control: I build a weighted combination of untreated regions that tracks the treated region's pre-campaign trajectory closely. The credibility comes from the pre-period fit and from placebo tests, where I pretend each donor region was treated and see how often an effect that large appears by chance. If the fit is poor, I fall back to a matched-market comparison and say plainly that the estimate is weaker. I would also check whether the campaign was targeted at regions already trending upward, since that would break the design.

Lead with experiments, causal analyses and forecasts; keep reporting work to a single line, because that framing reads as a data analyst rather than a data scientist.

Replace them with a business metric under a real constraint, such as an imbalanced outcome, a time-based split or a decision the model changed.

Explain how the model's output entered a workflow, for example that the scores set the outreach queue order, so the impact claim is traceable.