How easy stuff is depends on what software you're using... change port 21 to some 1083 or such, above 1024, don't allow anonymous access, don't give any users execute rights on files. You may also require everyone who connects to use SFTP. Oh, and lock users in their home directories.
Basically, as long as you follow the very basic precautions (lock users in directories, no anons, no execute rights), it's fairly safe. Of course, one could detect that you're running a FTP server, but gaining access to it would be a matter of knowing the username/password. Assuming that the passwords are fairly secure, you're fine - the random script kiddies on the Web won't be spending a week cracking your FTP anyhow.