Dude, its not like there's a tiny little baby hanging out in the sperm. Two sex cells fuse into one cell and then that divides into 4 then 8 then 16 then 32.
If it was the "tiny little mans" theory then there would be any girls born and no one would inherit anything from their mother