You can get it all on your own PC - oh yes.

For windows 9x-Me (if that's what you use) you can use EasyPHP download from http://www.easyphp.org/ it's very lovely and will install and configure Apache/PHP/MySQL for you in a few clicks - including phpMyAdmin etc...

If you got a NT OS (NT/W2k/XP) then the WAMP package does the same for you get it over at http://www.wampserver.com/en/index.php

You can set it up so it only runs when you start it rather than running as a service if you only want it for testing purposes or as a service if you want to run a webserver on your PC (not recommended).