Calculate new position of rotated rectangle

HINT: The length of the perimeter remains constant:

O1O2 + O2O3 + O3O4 + O4O1 = O1N2 + N2N3 + N3N4 + N4O1

"The length of the perimeter remains constant"? Huh? Point N3 can be dragged anywhere, so the length of the perimeter can change significantly. In my example images, the perimeter remained reasonably constant, but that's only happenstance.

Well, in that case:

O1N2 = N3N4
O1N4 = N2N3

otherwise it won't remain rectangular

True that opposite sides are constrained to be equal; but to be rectangular, must also constrain angles to be 90 degress (otherwise you may end up with a parallelogram).

Translate the top rectangle linearly so that O1 -> (0,0). Then after some changes, translate the new rectangle back to O1's original position.

With that linear translation understanding, we can assume that O1 is at (0,0). Now you can treat O2, O3, and O4 as vectors in the plane.

Define unit vectors, u1 and u2:
u1 = O2/|O2|, where |O2| is the length of O2
u2 = O4/|O4|

Notice that O2 is the projection of O3 onto the line defined by b*u1 where b is some constant.
Notice that N2 is the projection of N3 onto the line defined by  c*u1 where c is some constant.

Since u1 and u2 form an orthonormal basis in the plane, then the projection is related by
|N2| = N3 .* u1   where .* is the dot product between N3 and u1
        http://en.wikipedia.org/wiki/Dot_product
After calculating |N2|, then N2 = |N2| u1

Similarly, compute N4.

Translate 4 points linearly back to original O1 and you have the coordinates.

There is a simple solution for your question.  Notice that  vectors

O1 N2 =   a O1 O2
O1 N4 =   b O1 O3

Your problem is solved, if you can find  "a" and "b". The only constraint  is that

O1 N2   - O1 N4 = O1 N3

It is equivalent to

a O1 O2 - b O1 N4 = O1 N3

This give you a system of two equations of  "a" and "b" corresponding to x and y coordinates.

a [O2(x) - O1(x)] - b [ N4(x) - O1(x)] = N3(x) - O1(x)
a [O2(y) - O1(y)] - b [ N4(y) - O1(y)] = N3(y) - O1(y)

You can solve the system of equations using Cramer's rule.
http://en.wikipedia.org/wiki/System_of_linear_equations

Hope it helps!

In previous post, I wrote:
If you can move N3 to the left of O4, for example, then the dot product of N3 and u1 can be negative number.

So, to be more general allowing for any N3, then to get the coordinates of N2, you simply say:
      N2 = (N3 .* u1) u1
Now, the ± signs take care of themselves.
(Again, as mentioned earlier, we first translated the system where O1 is the origin).