I think I mentioned it but I am using a combination of android.speech, J.A.R.V.I.S, CMU Sphinx, and Pocket Sphinx
Basically to fit my needs I need a Speech Recognition engine not a STT engine so I am using Google’s web and android STT to text to help train my CMU Sphinx dictionary until it becomes more mature. Until then users will need to click or touch a button before they start talking. but I am not focusing on that until I actually get an alpha version out there.
EDIT
To answer your question that is exsactly what CMU Sphinx and Pocket Sphinx are for.