L.,
You can do all of this online. I met with my financial advisor first, who is also my CPA, just to make sure I had my ducks in a row as far as what to do first (and of course he handles all of my income tax issues etc for the business along with our family's personal tax needs). We determined that my best course would be to become a LLC, so I'd know how to identify my business for registration purposes.
However, I registered my business both with the federal government (for my tax ID #) and with Indiana (in.gov) through their websites, and also paid online by credit card. Because I am a retail establishment with fairly low income, I pay my sales taxes monthly - Amanda's right: however you're paying, you REALLY want to pay on time. If we're even a day late, we get a form letter saying state internal revenue thinks we owe an average of $15,000 - LOL!