ViaSat, a global provider of satellite and wireless communications, securely connects businesses, governments, and organizations to the Internet. Learn how ViaSat’s Network Solutions Engineer, drove the transition from a traditional network support to a DevOps-centric model.
Course Of The Month
5 days, 15 hours left to enrollBecome a Premium Member and unlock a new, free course in leading technologies each month.
Solved
Non-uniform PCIe Access - are any OSs "NUPA" aware?
Posted on 2012-12-26
Wondering which PCIe slots to use on dual Intel CPU E5 based servers, and whether any operating system know how to load balance to keep PCIe access local to the CPU.
It's well known that modern OSs are NUMA aware and will try to allocate RAM/cores so that memory access is local rather than across the hypertransport/quickpath busses because modern CPUs have inbuilt memory controllers.
What about PCIe cards though now that the PCIe controllers are built into the CPUs rather than on the southbridge? Say for example I have a HP DL380 gen8 or Dell R720 with two CPUs and I put dual port NICs in slots 1 and 4 and teamed them together for 4*1Gb, would the OS be clever enough to know that slot 1 was on CPU1 and slot 4 was on CPU2 and direct the outbound packets to the local NIC or would half of the traffic go over the HT bus in a random manner? Is there any benefit of spreading NICs and FC HBAs over both CPUs inbuilt PCIe controllers?
I'm going to sleep on it and see if anyone knows for sure.
It's well known that modern OSs are NUMA aware and will try to allocate RAM/cores so that memory access is local rather than across the hypertransport/quickpath busses because modern CPUs have inbuilt memory controllers.
What about PCIe cards though now that the PCIe controllers are built into the CPUs rather than on the southbridge? Say for example I have a HP DL380 gen8 or Dell R720 with two CPUs and I put dual port NICs in slots 1 and 4 and teamed them together for 4*1Gb, would the OS be clever enough to know that slot 1 was on CPU1 and slot 4 was on CPU2 and direct the outbound packets to the local NIC or would half of the traffic go over the HT bus in a random manner? Is there any benefit of spreading NICs and FC HBAs over both CPUs inbuilt PCIe controllers?
I'm going to sleep on it and see if anyone knows for sure.
[X]
Welcome to Experts Exchange
Add your voice to the tech community where 5M+ people just like you are talking about what matters.
- Help others & share knowledge
- Earn cash & points
- Learn & ask questions
3-
2
5 Comments
Active today
Andy, it's an excellent question, and we've been asking ourselves the same question and testing with recent large cluster deployments of Dell R720s, and which PCIe slots, to insert quad Broadcom network cards, to network team with Windows 2012 Hyper-V.
at present, after much testing, an high network I/O, and performance monitoring, it's difficult to ascertain, and draw any conclusions, and we have raised this Service Request to our Technical Specialist at Dell!
at present, after much testing, an high network I/O, and performance monitoring, it's difficult to ascertain, and draw any conclusions, and we have raised this Service Request to our Technical Specialist at Dell!
Active today
Did Dell come back with anything? It's really a question for the likes of VMware, MS and the NIC/HBA manufacturers to address rather than the server makers. I think we have to assume there's nothing that uses locality in load balancing at the moment so we might as well just put the cards on any CPU and get average 50% local traffic.
Not sure if there is a heat/turbo gain to spreading the cards over both CPUs, if the processors and airflow were identical then maybe the additional heat from the PCIe controller part of the chip might come into play and slow the CPU on that die down but that would be delving into the 1-2% speed improvements you can get by picking the fastest of a particular chip part number.
Not sure if there is a heat/turbo gain to spreading the cards over both CPUs, if the processors and airflow were identical then maybe the additional heat from the PCIe controller part of the chip might come into play and slow the CPU on that die down but that would be delving into the 1-2% speed improvements you can get by picking the fastest of a particular chip part number.
Active today
EE is chasing as abandoned so I'll close. Guess it doesn't matter much which CPU the peripherals are on at the moment otherwise the manufacturers would be shouting about slight speed improvements from clever drivers.
Featured Post
Question has a verified solution.
If you are experiencing a similar issue, please ask a related question
Background Information
Recently I have fixed file server permission issues for one of my client. The client has 1800 users and one Windows Server 2008 R2 domain joined file server with 12 TB of data, 250+ shared folders and the folder structure i…
This article outlines why you need to choose a backup solution that protects your entire environment – including your VMware ESXi and Microsoft Hyper-V virtualization hosts – not just your virtual machines.
In this Micro Tutorial viewers will learn how to use Windows Server Backup to create full image of their system. Tutorial shows how to install Windows Server Backup Feature on Windows 2012R2 and how to configure scheduled Bare Metal Recovery backup.…
This video shows you how easy it is to boot from ISO images for virtual machines with the ISO images stored on a local datastore on the ESXi host.
Suggested Courses
Course of the Month5 days, 15 hours left to enroll
707 members asked questions and received personalized solutions in the past 7 days.
Join the community of 500,000 technology professionals and ask your questions.