Ok, let me start over.
Construction of the problem:
Let there be a circle with radius R=2 with the center of that circle called O. Construct a square around that circle such that both touch (not intersect) in four points and the circle is completely contained within the square. Now construct a circle such that it touches the square twice, the circle once and is again completely contained in the square.
Preliminary considerations:
Notice that this problem has the same symmetry as a square, especially it is symmetric under mirroring on the diagonals. Thus the center of the second circle most lie on that diagonal. Call that center C (equivalent to what I called N before, but you mean something else, thus I rename it here).
Other points that I will reference is the point at which both circles touch (A as in your notation) as well as a corner of the square (B). There are four points each of A,B,C, due to symmetry. In the following I will choose one B, and the A and C that lie on the line OB this ensures that all points are on the same half-diagonal of the square.
Solution:
By construction the square has the side length OB=2R, thus the half diagonal has the length R*\sqrt(2).
OTOH from the construction it follows that OB = OA + AC + AB. OA connects the center of the big circle with a point of the circumference and thus has length R. AC similarliy has the length r.
Now we have to consider AB. to do this we consider the touching points of the small circle with the square. Since in both cases a line touches a circle the lines will be tangential and thus the radii pointing towards these points will be perpendicular on those lines. Thus it follows that the object made out of A,B and these two touching points is a square with side r and AB is the diagonal. This implies AB = \sqrt(2)*r.
Thus R\sqrt(2) = R + r*(1+\sqrt(2)), and r = R *(\sqrt(2)-1)/(\sqrt(2)+1).