Hmm... when you installed the Nvidia card, did you install new drivers too?

To check memory, use MemTest...

Else, try reinstalling your AGP Chipset driver, and Ati or Nvidia drivers...

Did you make any tweaks to windows? I remember there being an issue with Ati cards and Lots of memory, and setting the LargeMemory tweak ? Just a thought.