A few days ago, I got some of my neural net code to work with a GPU. The GPU is called "Tesla C2075". It has a peak performance of 515 GFLOPS. It has 448 CUDA cores that run at 1.15 GHz and it has 6 GB of GDDR5 memory.
My code needed about 10 hours to run before. After that, it only needed 10 minutes. That is 60 times faster! The library that worked this miracle for me is called Theano.
Out of curiosity, I've searched for current high-end gaming graphics cards. I found the NVIDIA Titan Z:
The Titan Z has 5760 CUDA cores. It reaches 2 × 4061 GFLOPS and has 12 GB of memory. That technological wonder costs only 2802 Euro.
To put that into perspective: In 2005, you would probably have been in place 68 of the TOP500 supercomputers worldwide! (source).
Isn't that crazy?