A very commonplace experience is to have security guards run out of a building you are photographing and insist that you cannot because the management do not allow it, because the building is copyright, because it is private property, because you need permission, that they will have you arrested, or you will be sued. They have no authority nor legal basis for any of this so long as you are photographing from the public road or path. They have no power to demand you delete pictures, and no power to demand your name and address.

In most cases, if explanation doesn't do the trick, calling the police is a good idea.