Neutralizing antibodies (nAbs) elicited against the receptor-binding site (RBS) of the spike protein of wild-type SARS-CoV-2 are generally less effective against recent variants of concern. RBS residues E484, K417 and N501 are mutated in variants first described in South Africa (B.1.351) and Brazil (P.1). We analyzed their effects on ACE2 binding and K417N and E484K mutations on nAbs isolated from COVID-19 patients. Binding and neutralization of the two most frequently elicited antibody families (IGHV3-53/3-66 and IGHV1-2), which can both bind the RBS in alternate binding modes, are abrogated by K417N, E484K, or both. These effects can be structurally explained by their extensive interactions with RBS nAbs. However, nAbs to the more conserved, cross-neutralizing CR3022 and S309 sites were largely unaffected. The results have implications for next-generation vaccines and antibody therapies.