Why is it rude to say "hold on"?

Um yes, my mom yells at me for saying this when i'm busy and it extremely aggrivates me. How is this rude? It is like a synonym for "please wait."