You're not quite clear with your algebra, specifically, what's an equation and what's a question.

(x-y)^2 => x^2 - 2xy + y^2

Good, rephrase the question. However, I don't know what "=>" means here; it should be "=".

So, the new question is "What's the value of x^2 - 2xy + y^2?"

And, x^2 + y^2 = 29 => 29 - 2xy

This seems confused, is it an equation, or a question? I'd recommend that you think (and write):

x^2 + y^2 = 29, so the question get's rephrased as "What's the value of 29 - 2xy ?"

At this point, it should be clear that statement 1 is sufficient, as it gives you the missing unknowns in the question as values.

if x = 5 then 29- 2(5)y = 29 - 10y

-10y = 29

This is your real mistake. Where did this equation come from? I guess that you've assumed that 29 - xy = 0, but that's stated nowhere in the problem. You've made an equation out of the question. Be sure that your scratch work helps you avoid this kind of confusion (which is the big trap of problems like this one).