Tandem mass spectrometry has become the method of choice for high-throughput, quantitative analysis in proteomics. Peptide spectrum matching algorithms score the concordance between the experimental and the theoretical spectra of candidate peptides by evaluating the number (or proportion) of theoretically possible fragment ions observed in the experimental spectra without any discrimination. However, the assumption that each theoretical fragment is just as likely to be observed is inaccurate. On the contrary, MS2 spectra often have few dominant fragments. Using millions of MS/MS spectra we show that there is high reproducibility across different fragmentation spectra given the precursor peptide and charge state, implying that there is a pattern to fragmentation. To capture this pattern we propose a novel prediction algorithm based on hidden Markov models with an efficient training process. We investigated the performance of our interpolated-HMM model, trained on millions of MS2 spectra, and found that our model picks up meaningful patterns in peptide fragmentation. Second, looking at the variability of the prediction performance by varying the train/test data split, we observed that our model performs well independent of the specific peptides that are present in the training data. Furthermore, we propose that the real value of this model is as a preprocessing step in the peptide identification process. The model can discern fragment ions that are unlikely to be intense for a given candidate peptide rather than using the actual predicted intensities. As such, probabilistic measures of concordance between experimental and theoretical spectra will leverage better statistics.