Ok two things, first thing I will address is how to take this integral through the Green's theorem path (I just do it with Stokes' theorem since that's just how I think of it, same thing) And the other thing I will address later on is the conservative field comment I made. \[\iint 6x-8y dA = \iint \nabla \times F dA \]
So in order to make that substitution, I'm really saying that this is the curl of some vector field
\[6x-8y=\nabla \times F\]
More explicitly, let's just calculate the terms starting with \(\vec F = \langle F_x, F_y, 0 \rangle \)
\[\nabla \times F = \langle 0, 0, \frac{\partial F_y }{\partial x} - \frac{\partial F_x}{\partial y} \rangle \]
So now we equate this component with our curl (Yeah, cross products are awkward for 2D since you have to put in extra 3D garbage in with it, oh well, tensors are better :P)
\[ \frac{\partial F_y }{\partial x} - \frac{\partial F_x}{\partial y} = 6x-8y\] This next step is completely arbitrary to break it up into what's "obvious" although I'm pretty sure it doesn't matter how we break it up, not sure:
\[ \frac{\partial F_y }{\partial x} = 6x\]
\[ \frac{\partial F_x}{\partial y} = 8y\]
So finding one possible choice of F is not hard:
\[\vec F = \langle 4y^2, 3x^2, 0 \rangle \]
Now we are able to take this step further:
\[\iint 6x-8y dA = \iint \nabla \times F dA = \oint F \cdot dS= 0 \]
The path of our line integral is the boundary of a circle with radius 5, \(\vec c(t) = \langle 5 \cos t, 5 \sin t \rangle\) so we could proceed with this integral:
\[\int_0^{2 \pi} \vec F(x(t),y(t)) \cdot \vec c'(t) dt\]
\[\int_0^{2 \pi} 4*5^2 \sin^2(t) * (-5) \sin (t) + 3*5^2 \cos^2(t) *5 \cos(t) dt\]
which is an odd power of a trig function over an entire period so it should be zero, assuming everything I did was right, so this is at least how you could do it.
So let's move on to the second thing, the conservative field comment. I guess in order to be conservative, the line integral for all closed paths must be equal to 0, in other words the integral only depends on the state of the two end points and not the path it takes. So yeah I guess I goofed, it might not be. If I have any more mistakes above or you can clarify anything please do.