I recently started a new job where I'm in the process of researching hardware possibilities for a high performance computational cluster. We currently have a research cluster consisting of 40 dual CPU servers that is about 4 years old. One question that has been brought up recently is if, now that you can get quad-core CPU's, is there a significant trade off between CPU processing and memory access and utilization?
Our current idea is to build the new cluster with 320 CPU's to start and leave room for future growth. Companies like IBM, Dell, etc. are now making servers available that support two quad-core CPU's each, so theoretically we could reach this number with 40 dual quad-core CPU servers. What I'm not so sure about is whether or not packing 8 CPU's into a single server with 32gb of RAM makes sense. The cluster will be running commercial products like Mathematica as well as home-brew MPI-based applications. Our current cluster is bound much more by memory utilization than I/O so we're anticipating seeing similar behavior in the new cluster.
So the really big question is whether the CPU's will become memory-starved if all 8 CPU's on a given node are running memory-intensive applications. I know it's a bit of a "religious" issue but is Intel or AMD considered better in such an environment, or are they pretty much on equal footing? Are more cores per chip always better, or is there a reason why we would want fewer cores per chip, or perhaps (in the quad-core case) single-socket instead of dual-socket systems to prevent RAM bandwidth starvation? Bottom line - is it worth the additional cost (not just for servers but physical space, power, cooling, etc) of splitting CPU's out into more physical servers to improve computational throughput?