It's the "g" sound that most people don't like. That, and the fact that they're not used to feminine names beginning with an "o."
I'm a Russophile, so
Olga is fine by me. I love the nn
Olya.
By the way, are you pronouncing it the Russian way, "ULL-guh," or the English way, "OHL-guh?" Just curious.