The records or primary keys you pass into the associate function are all the records you want to associate, not additional ones.
In other words, to unassociate records do something like
$user->associateGroups( array() );
i.e. associating no group with a given user. This also means that if you want to only associate an additional record, you'll first have to retrieve all currently associated records and also pass them to the associate function.
I asked the same question some months ago and got no answer sorrowly. I stumbled upon the "solution" only by chance. It should be pointed out in the documention more clearly.