General questions regarding multi-core CPU's and memory utilization

Posted on 2007-10-01
Last Modified: 2008-02-07
I recently started a new job where I'm in the process of researching hardware possibilities for a high performance computational cluster.  We currently have a research cluster consisting of 40 dual CPU servers that is about 4 years old.  One question that has been brought up recently is if, now that you can get quad-core CPU's, is there a significant trade off between CPU processing and memory access and utilization?

Our current idea is to build the new cluster with 320 CPU's to start and leave room for future growth.  Companies like IBM, Dell, etc. are now making servers available that support two quad-core CPU's each, so theoretically we could reach this number with 40 dual quad-core CPU servers.  What I'm not so sure about is whether or not packing 8 CPU's into a single server with 32gb of RAM makes sense.  The cluster will be running commercial products like Mathematica as well as home-brew MPI-based applications.  Our current cluster is bound much more by memory utilization than I/O so we're anticipating seeing similar behavior in the new cluster.

So the really big question is whether the CPU's will become memory-starved if all 8 CPU's on a given node are running memory-intensive applications.  I know it's a bit of a "religious" issue but is Intel or AMD considered better in such an environment, or are they pretty much on equal footing?  Are more cores per chip always better, or is there a reason why we would want fewer cores per chip, or perhaps (in the quad-core case) single-socket instead of dual-socket systems to prevent RAM bandwidth starvation?  Bottom line - is it worth the additional cost (not just for servers but physical space, power, cooling, etc) of splitting CPU's out into more physical servers to improve computational throughput?
Question by:brucepennypacker
    LVL 4

    Expert Comment

    Hi, you may do better with AMD multi-core CPUs that support NUMA and have HyperTransport.  Intel have an upcoming equivalent of HyperTransport (of course, Intel says it'll be a HyperTransport killer!) called CSI - but this is touted for 2008/2009...
    I hope that gives you some leads to go down!

    Best of luck!
    LVL 8

    Accepted Solution

    I'm agreeing with the AMD call on the CPUs.  Personally I prefer Intel for a number of different applications, but for what you're doing AMD is better.  The AMD chips have the memory controller on the CPU/die which leads to better access times when going to RAM.

    Now, as for memory, with all these CPUs why would you onlly go with 32GB of RAM?  In this type of environment I'd really hope that you're running a 64-bit OS, in which case you should seriously consider going to at least 64GB on each server.  Windows 2003 Enterprise (R2) can support up to 2TB of RAM - get as much per server as you can afford, but honestly going with 32 could potentially cause a bottleneck if you're running memory intensive applications.

    That being said, I assume your applications are coded for actually utilizing all these available CPUs?  It they haven't been optimized for this type of environment you're really putting the cart before the horse.  One other thing to keep in mind is the size of the L1, L2, and possibly if they have it L3 cache sizes on the die - and what's actually available to each CPU.  Many new processors have shared caches between all the CPU's on each physical die, look for a die that will either give you a larger shared cache or provide multiple caches specific to each CPU core.

    Building an environment like this is a pretty interesting project (which I've had some experience with), but your time spent doing background research and thinking out all layers of the design will be rewarded and easily paid back when you build out an environment that works well and gives blazing performance.

    Are you looking at any particular processors?  If so, which ones?  i'd be happy to look at specs and offer some input.

    Featured Post

    IT, Stop Being Called Into Every Meeting

    Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

    Join & Write a Comment

    Suggested Solutions

    Hyper-convergence systems have taken the IT world by storm and have quickly started to change our point of view of how the data center should and could be architected. In this article, I’ll explain the benefits of employing a hyper-converged system …
    Learn about cloud computing and its benefits for small business owners.
    Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…
    This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now