What do you mean by "instance"? Number of copies of JBoss AS running at one time? Or number of web apps deployed to JBoss AS at one time? The answer to both is: how much memory/heap do you have? And for the latter, there is also the question of how big your app is - I suspect that I could deploy thousands of copies of a simple hello world servlet/jsp.
here instances meaning like default folder inside <jboss_home>/server how many i can create ? my total sever memory(RAM) size is 64 GB.
OK, so you meant server instances. In general, add the amount of heap and permgen you allocate to each server instance and make sure that the total doesn't go over 64GB. For example, if each server instance uses 2GB of max heap and permgen space, then you can run 32 instances.
The other things you'll want to monitor is the CPU and network utilization. Building on the above example, if all 32 server instances are very busy, you'll need at least 16 cores perhaps more. In general, for really busy server instances, you don't want more than 2 per core. And that is assuming typical web apps that access database - if you have a web app that is memory bound, such as creating lots of reports and doing lots of computations, then you might need 2 cores for each server instance. You might even need additonal NICs to handle the network load. On the other hand, if the 32 server instances are not that busy, you can make do with less.