The autophagy-lysosomal pathway is impaired in many neurodegenerative diseases characterized by protein aggregation, but the link between aggregation and lysosomal dysfunction remains poorly understood. Here, we combine cryo-electron tomography, proteomics, and cell biology studies to investigate the effects of protein aggregates in primary neurons. We use artificial amyloid-like β-sheet proteins (β proteins) to focus on the gain-of-function aspect of aggregation. These proteins form fibrillar aggregates and cause neurotoxicity. We show that late stages of autophagy are impaired by the aggregates, resulting in lysosomal alterations reminiscent of lysosomal storage disorders. Mechanistically, β proteins interact with and sequester AP-3 μ1, a subunit of the AP-3 adaptor complex involved in protein trafficking to lysosomal organelles. This leads to destabilization of the AP-3 complex, missorting of AP-3 cargo, and lysosomal defects. Restoring AP-3μ1 expression ameliorates neurotoxicity caused by β proteins. Altogether, our results highlight the link between protein aggregation, lysosomal impairments, and neurotoxicity.