Background: Human milk from coronavirus disease 2019 (COVID-19)-recovered women may be useful as oral antibody therapy to prevent severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection and provide long-term immunity to neonates and young children. As convalescent plasma is already used as antibody therapy, this study aimed to compare the binding capacity of antibodies specific to the receptor-binding domain (RBD) of SARS-CoV-2 between human milk and serum from COVID-19-recovered women. Materials and Methods: The areas under the curve (AUCs) for IgA, IgM, and IgG specific to the SARS-CoV-2 RBD in human milk and serum samples were measured using enzyme-linked immunosorbent assay. Milk samples were collected from 12 COVID-19-recovered women, while serum samples were from 10 COVID-19-recovered women. The antibody concentrations were also determined. Results: Our study reveals that SARS-CoV-2 RBD-specific antibody titers differed between human milk and serum samples from COVID-19-recovered women. When the AUCs were not divided by the antibody concentration, SARS-CoV-2 RBD-specific IgA, IgM, and IgG levels were higher in the serum sample group than the human milk group (p < 0.001). However, the titers of SARS-CoV-2 RBD-specific IgM (AUC/μg of IgM) and IgG (AUC/μg of IgG) were higher in human milk samples than serum samples (p < 0.05). The titer of SARS-CoV-2 RBD-specific IgA (AUC/mg of IgA) was higher in the serum sample group than the human milk group (p < 0.01). Conclusions: Human milk antibodies specific to the RBD of SARS-CoV-2 must be purified to obtain comparable binding capacity observed with SARS-CoV-2 RBD-specific serum antibodies.