The agent is running on a windows 2003 server platform within a vmware instance.
The jboss server is running as a windows service.
This combination probably makes the process looks different than what the agent's jboss-as plugin is looking for - so even though your JBossAS server is running, it cannot be found by the auto-discovery code (this is not unexpected - the plugin makes a best attempt to find running JBossAS processes but it won't be able to find them all based on all the possible ways you can run a JBossAS server).
2008-11-14 10:30:25,483 DEBUG [InventoryManager.discovery-1] (org.rhq.core.system.ProcessInfo)- Unexpected error occurred while looking up process info for process with pid [0]. Did the process die? Cause: org.hyperic.sigar.SigarException: Incorrect function.
You can ignore this - pid [0] is a root system process that the native library, Sigar, can't inspect. Its not your JbossAS process so don't worry about this. Its a DEBUG message, just informational.
Is there any possibility to configure the agent manually (not using the autodiscovery) by giving all required stuff for communication with the jboss server ?
Yes! Its called the "manual-discovery" feature. See:
https://docs.jbosson.redhat.com/confluence/display/JON2/a.+Inventory#a.Inventory-Manualdiscovery and
https://docs.jbosson.redhat.com/confluence/display/JON2/a.+Inventory#a.Inventory-Manuallyaddchildresources 
