in the rev you are on (i am guessing 427..) we have an Ai rework in the midle that's not finsihed yet..that's why you have a problem with the mobs...
As far for the ram usage..
you can use this
java -server -Dfile.encoding=UTF-8 -Xmx4g -Xmn2g -Xmn1g -XX:PermSize=512m -XX:SurvivorRatio=8 -Xnoclassgc -XX:+AggressiveOpts -cp ./../libs/*;l2jhellas.jar com.l2jhellas.gameserver.GameServer
it will use 4gb ram which is MORE than enough imo....