On your "potayto potahto" point, there's a book called Metaphors We Live By that has as its thesis the claim that we act and think in terms of metaphors. At one point, the authors take up the question of whether we have any experience at all that isn't framed metaphorically, and their answer is no. Even our most basic-seeming physical experiences, we understand metaphorically. It's metaphors all the way down, as it were. It would seem you have felt Lackoff and Johnson's thesis in your bones, as it were.
If the starting domain is internal (which has to be, since experience is internal) you simply wouldn't be able to form a concept of something without making (intuitively) a first metaphor of it to the internal experience. This quickly becomes an exponential function, though, since all sorts of intermediate metaphors are internal-internal. For example: you see a first, single object, so you then make a link to its identified qualities to the mental image of it, but at that time you already have factored in all sub-components as their own internal-internal metaphor: one, first, external, object, set of qualities etc. It is highly unlikely that information for internalized objects exists solely in one place instead of this scheme where images (internal metaphors) are spread to other points of storage.
Related: it's one of the reasons (not the only one) why I think math is not cosmic but a property of some beings' internal organization - and clearly of humans too.
Related2: if we assume that any single concept of anything is creating nested internal-internal metaphors (as well as the internal-external mapping) then it's to be expected that higher level connections between things would naturally use higher level metaphors.
Related3: you could ask yourself if the mapping of the external object, to the internal notion of it, actually ever reaches the external space; the answer should be no, which implies another group of approximations which still remain within the internal space but have to form distances between internal positions.