Both combine many models into one. Beyond that, they don't have much in common — one trains in parallel to cancel out variance, the other trains in sequence to grind down bias.
"Ensemble learning" is often taught as a single bucket — combine several models and the result beats any one of them. That framing hides the fact that bagging and boosting are solving two different problems with two different architectures. Bagging (bootstrap aggregating) trains many models in parallel, each on its own random resample of the training data, then averages or votes their predictions together — this cancels out variance. Boosting trains models in sequence, where each new model is deliberately built to fix the errors the previous models left behind — this grinds down bias. Random Forest is bagging; AdaBoost, Gradient Boosting, and XGBoost are boosting. Picking between them isn't a coin flip — it depends on which kind of error your model actually has.
Bagging draws several bootstrap samples from the training set — random samples of the same size, drawn with replacement, so each one leaves out roughly a third of the rows and duplicates others. A separate model is trained on each sample, entirely independently of the others — no model ever sees another model's data, errors, or output. Because every model overfits to the noise of its ownsample in a slightly different way, averaging their predictions (or taking a majority vote) causes that sample-specific noise to cancel out, while the real signal — which every sample shares — survives. That's exactly why Random Forest works well even though its individual trees are grown deep and are individually overfit-prone: bagging's whole job is to clean up after them.
Boosting does the opposite on every axis. It trains one model, looks at which examples it got wrong (or scored with high error), then trains the nextmodel with those specific examples up-weighted so it's forced to pay attention to exactly what the first model missed. This repeats for many rounds, each new weak learner incrementally patching the mistakes of the ensemble built so far. The final prediction is a weighted sum of every model in the sequence, not just the last one. Because each round is explicitly correcting error rather than just resampling, boosting is fundamentally a bias-reduction process — a sequence of individually weak, simple learners incrementally builds up a strong overall fit. AdaBoost, Gradient Boosting, and XGBoost all follow this pattern. The tradeoff: because boosting keeps chasing whatever the ensemble got wrong, it will chase noisy or mislabeled examples too, which makes it more prone to overfitting than bagging if the data is noisy.
Recall that a model's total expected error splits into bias and variance (see the Bias-Variance Tradeoff explainer). Bagging and boosting each start from a different assumption about which term is the problem:
Averaging n independent estimators divides variance roughly by n— that's a mathematical fact about averages, not a training trick — which is why bagging deliberately keeps its base learners flexible (Random Forest trees are grown deep, individually overfit-prone, because bagging's job is to clean up exactly that). Boosting instead has no averaging step to lean on; it reduces error by having each new model fit whatever the ensemble-so-far still gets wrong, which is a bias-reduction mechanism — but it also means boosting has no built-in defense against noisy labels: if a training example is simply mislabeled, boosting will keep up-weighting it every round, trying harder and harder to fit noise it can never actually learn, which is the classic route to a boosted model overfitting.
False. "Combines multiple models" is the only thing they have in common — everything that matters is different. Bagging trains models in parallel, with zero interaction between them, and targets variance. Boosting trains models in strict sequence, where each model exists specifically because of the previous ones' mistakes, and targets bias. Those are opposite architectures solving opposite problems, not two flavors of the same idea. It's also why the choice between Random Forest and XGBoost isn't arbitrary: if your model is underfitting (high bias — it's too simple to capture the pattern even on training data), bagging a bunch of copies of that same simple model together won't help, because averaging cancels variance, not bias — you need boosting. If your model is overfitting (high variance — it's unstable and swings wildly between training samples), boosting a sequence of increasingly specialized correctors will often make that instability worse, not better — you need bagging. Reaching for the wrong ensemble family for the error you actually have doesn't just fail to help; it can actively make the real problem worse.
Explains the two mainstream ensemble-learning architectures and why they are not interchangeable: bagging (bootstrap aggregating) trains many models in parallel on independent bootstrap samples and averages or votes their predictions to reduce variance, while boosting trains models sequentially, each one built to correct the previous models' errors, to reduce bias. Illustrated with side-by-side architecture diagrams for both, plus why the choice between algorithms like Random Forest (bagging) and XGBoost (boosting) depends on which error source actually dominates.
"Ensemble learning" is usually introduced as one idea — combine several models, get a better one — which makes bagging and boosting look like two interchangeable flavors of the same trick. They are not. Bagging's models never interact with each other and are all trained at once; boosting's models are trained one at a time and each one is defined by the errors of the ones before it. Treating them as equivalent means missing which specific error source (bias or variance) each one is built to fix, which is the entire reason to choose one over the other for a given problem.
Bagging (bootstrap aggregating) draws several bootstrap samples from the training set — random samples of the same size drawn with replacement, so each sample typically includes about 63% of the unique original rows, some duplicated, and leaves the rest out. A separate model is trained on each sample independently and in parallel; none of the models communicate during training. Because each model overfits to the particular noise of its own sample in a different way, averaging their predictions (regression) or taking a majority vote (classification) causes that sample-specific noise to average out, while the true underlying signal — present in every sample — reinforces itself. This is why Random Forest, the best-known bagging algorithm, deliberately grows its individual decision trees deep and largely unpruned: those trees are supposed to be low-bias, high-variance, individually overfit-prone learners, because bagging's entire mechanism is designed to cancel exactly that variance back out. Random Forest additionally randomizes which features each tree is allowed to split on, which further decorrelates the trees and makes the variance-cancelling effect of averaging stronger.
Boosting builds an ensemble one model at a time. The first model is trained on the data with equal weight given to every example. Its errors are then identified, and the training weights are adjusted so that misclassified or high-error examples matter more; the second model is trained on this reweighted data, which forces it to focus specifically on what the first model got wrong. This repeats for many rounds — AdaBoost reweights misclassified examples directly, while Gradient Boosting and XGBoost instead fit each new model to the residual errors (or negative gradient of the loss) left by the ensemble so far, which is mathematically a form of gradient descent performed in function space. The final prediction is a weighted combination of every model trained along the way, not just the last one. Because each round is a strong, deliberate correction rather than an independent resample, boosting builds a strong overall fit from individually weak, simple learners (shallow trees, often just a few levels deep, sometimes single-split "stumps") — it is fundamentally a bias-reduction process. The same sequential-correction mechanism that reduces bias also has a downside: because boosting keeps up-weighting whatever the ensemble still gets wrong, mislabeled or genuinely noisy training examples get chased harder each round, which is why boosting is more prone to overfitting on noisy data than bagging, and why techniques like a learning-rate shrinkage factor and early stopping are standard practice in gradient boosting implementations.
Because bagging targets variance and boosting targets bias, the right choice depends on what is actually wrong with the base model. A single decision tree, deep and unpruned, is a classic low-bias/high-variance learner — it can represent complex patterns but is unstable across different training samples — which is exactly the profile bagging (Random Forest) is designed to fix. A single shallow decision stump is a classic high-bias/low-variance learner — stable but too simple to capture much — which is exactly the profile boosting (AdaBoost, Gradient Boosting, XGBoost) is designed to fix. Bagging a collection of already-simple, high-bias learners together does little good, because averaging cancels variance, not bias, and there was never much variance to cancel. Boosting a collection of already-flexible, high-variance learners can actively make things worse, because the sequential correction process has no averaging step to rein in instability, and can amplify it round after round. This is why Random Forest and XGBoost are not interchangeable defaults — the choice reflects a diagnosis of which error source dominates the problem at hand.
Yes to both. Random Forest is the best-known bagging algorithm: it trains many decision trees in parallel on bootstrap samples (plus random feature subsets per split) and averages or votes their outputs. XGBoost, along with AdaBoost and standard Gradient Boosting, is a boosting algorithm: it trains decision trees sequentially, each one correcting the errors of the ensemble built so far, combined into a final weighted sum.
Boosting generally is, on noisy data. Bagging's averaging step actively suppresses overfitting by cancelling out each model's sample-specific noise. Boosting has no such safety net — its sequential error-correction mechanism will keep up-weighting and re-targeting any example it gets wrong, including mislabeled or inherently noisy examples, which can drive it to overfit that noise. This is why boosting implementations lean heavily on regularization tools like learning-rate shrinkage, tree-depth limits, and early stopping.
Each can have a secondary effect on the other term, but it is not their primary mechanism. Bagging's averaging step can very slightly reduce bias if the base models are diverse enough, but its overwhelming effect is variance reduction. Boosting can somewhat reduce variance if regularized carefully (shrinkage, subsampling), but its overwhelming effect, and the reason it exists, is bias reduction through sequential error-correction.
No — both are general-purpose meta-algorithms that can wrap almost any base learner (linear models, neural networks, etc.). Decision trees are just an unusually good fit for both: shallow trees are naturally high-bias, low-variance learners that suit boosting, and deep trees are naturally low-bias, high-variance learners that suit bagging, which is why tree-based implementations of both dominate in practice.
Yes — for example, training several boosted ensembles independently on different bootstrap samples and averaging their outputs, or using random feature/row subsampling inside a boosting round (as XGBoost and LightGBM both support) to add some of bagging's variance-reduction behavior on top of boosting's bias-reduction. These hybrids exist precisely because the two mechanisms target different error sources and can, to a degree, be stacked.
Try our AI and Data Science Studio
More calculators, simulators, and guides for this discipline.