The area of the square is so large that it completely encloses rectangle R. So, let's just create a rectangle with one endpoint on the line y = -3x and placed horizontally.
From a point (x, -3x), create a rectangle to the right and up. The distance from this point to the right end of rectangle R is: 8 - x; the distance from this point up to the top of rectangle R is: 12 - -3x).
The total area is: (8 - x)(12 - -3x) = (8 - x)(12 + 3x) = 96 + 12x - 3x².
The first derivative is: 12 - 6x
Setting this equal to zero: 12 - 6x = 0 ---> 12 = 6x ---> 2 = x
If x = 2, y = -6
The horizontal length is 8 - 2 = 6, the vertical length is 12 - -6 = 18 ---> Area = 108