Your entire investment is electronic when it's on the web. If you pay $30,000 for a spiffy, cool, fantastic web based program, you don't want to lose it just because someone accidentally wiped a hard drive.
Getting a copy of the site on a CD usually doesn't do you a lot of good. Don't forget, web applications aren't static. Much of what you see on a day to day basis comes out of a database. Beyond that, much of what makes your website work depends on other programs that may or may not be available when your web developer disappears off the face of the earth.
Your best bet is to have a live backup that synchronizes a couple times a day to a server you can check up on. It's even better if you can have that backup server sit in your office so that, in an absolute worst case scenario, you can pick up the box, go to a new ISP and be up and running again.
Get the passwords
I don't care if you have no idea what an SSH shell is, or an FTP server, for that matter, make sure you get all the passwords for all the services you're paying for. If something goes horribly wrong with your developer, you can hand the cryptic list to another developer and get out of trouble.
Remember: letting your web developer control everything means your web developer controls YOU as well.