For months members of the public have been using GeoSpy, a tool trained on millions of images that can find the location a photo was taken based on soil, architecture, and more. It's GeoGuesser at scale.
the article mentions that the tool takes into consideration architecture, distance between buildings and so on.
i think youd have to mess up the backgrounds somehow
Use unidentifiable backgrounds or remove them in post. I'd suggest small patches of sand, dirt or grass, walls, etc. Avoid anything that can be found on a map like mountains (and other large scale topography), buildings, power lines, etc.