One way to factor a trinomial like this is to follow the following steps. This method involves some guessing and trial and error.
1. Set up two sets of parentheses:
( )( )
2. On the left side of each parentheses, write two factors of the first term, 3x^2. The factors of 3x^2 are 3x and x:
(3x )(x )
3. On the right side of the parentheses, write 2 factors of -20. Here's where you need to do some trial and error. The factors of -20 are:
-20, 1
-1, 20
4, -5
-4, 5
2, -10
-2, 10
In addition, each of those guesses can be placed two ways, for example, if you use 4 and -5, you can place them these ways:
(3x + 4)(x - 5) and (3x - 5)(x + 4)
How do you know which version works? Do OI of FOIL and add the two terms. Whichever choice of factors that gives you the middle term is the correct one.
With (3x + 4)(x - 5), OI is -15x + 4x = -11x (Doesn't work)
With (3x - 5)(x + 4), OI is 12x - 5x = 7x (dosen't work.
Let's try 2, -10
(3x + 2)(x - 10) OI is -30x + 2x = 28x (doesn't work)
(3x - 10)(x + 2) OI is 6x - 10x = -4x (Works!)
Therefore, 3x^2 - 4x - 20 = (3x - 10)(x + 2)
There is another method called factoring by parts that invloves less guessing. I can show you that too, if you're interested.