Tony,
Brushed and brushless mode stuff can be a little tricky at times if you're not prepared for it.
The very first thing you need to do is do a one touch setup for your ESC.
Press and hold the button, turn the esc on, wait for the red LED to come on, then pull full throttle.
When the green LED comes on go full brake.
When the green LED blinks return the trigger to neutral.
When you do this this reverts the ESC back to factory default settings, which in the case of the GTB, is for brushless opperation.
If you do not have a motor sensor harness plugged into the ESC you will more than likely see the error code I think you were describing, which is a solid red LED with a blinking green LED next to it. This simply means that the ESC doesn't see a sensor harness plugged in.
If you have a brushless motor plugged in at neutral you should normally see a red, blue, and amber LEDs lit simultaneously.
Next step, go to brushed mode:
Turn the Esc back off.
Press and hold the setup button, and turn the ESC on... wait THROUGH the red LED...you should then see the red AND blue LED come on at the same time.
At this point release the button, the LEDs will blink simultaneously 9 times. On a 10th blink they will stay on for a second or two... durring this 10th "blink" press the setup button again.
You should see a bit of a light show but once it's all said and done you should see a red LED and a blue LED on at neutral.
Hope this helps dude!