yes, you will have to move or copy (by ftp probally) all of the contents of mysite.com/joomladirectory to your home directory. that database will not need to be changed. all you should have to edit after the move is the config file. you will have to change the location of the site from the joomladirectory to just /
Now if I move my site to a different server all I have to do pretty much is back up my entire site and its folders along with my database file, correct?