The calculator doesn’t understand your implicit multiplication, so you need to enter a*b=11, not ab =11. If you do this the calculator produces the correct two (complex) results.
In your case, the calculator simply ignored the ab=11 part and assigned an arbitrary variable, r1, to b and then solved for a as a = 6 - b → 6 - r1.