Q:

Is drinking wrong if you just have one or two

Answer

No one or two is just fine, especially during an occasion, even suggested to pregnant woman 1 or 2/mo' glass of red wine or beer.