It is possible to do it by hand - with calls to javac (with the right classpaths etc), or compiling from within your IDE, jar to do JAR/WAR/EAR packaging with some manual file copying etc - but - the chances are that you will want a repeatable process, so that when you come to rebuild your application you don't have to go manually through all these stages again. Also, if you have more than one developer working on the project, a repeatable and consistent build process is essential.
Ant is truely an excellent build tool - if you're not familiar with how it works, I'd suggest investing a little time in getting to know it.
Here's a couple of good links...
JavaWorld: Automate your build process using Java and Ant
Apache Ant Resources (links to articles / books)