I have a list with more than 30K items and a lot of issues with it. First there is a problem with InfoPath that I am trying to resolve in a separate question. Here I just need to understand how that 5K limit works.
So I have a list with over 30K of items and new ones are added daily. When I go to the list my default view is last 7 days which is usually less than 1000 items. This view is grouped by person so you need to click a name to expand it. It seems to be working OK. The weird thing is that when I use the column filter to show me results for only one name it shows it collapsed with the correct number of items but when I click the name to expand it shows the error:
This view cannot be displayed because it exceeds the list view threshold (5000 items) enforced by the administrator.
Why??? The original view returns only about 1K items and when I want to filter it even more to about 20 items I get that error? I have 4 columns indexed including person and date modified that I use for filtering here.
Any my sub-question: if I want to keep all items but the ones older than 3 months move automatically to a second list for archiving what functionality of SharePoint I should use for that? What would be the best practice?
The short answer is you need a real database, SharePoint is not disigned for this amount of data. It is a flat data structure so that you can navigate quickly. For large amounts of data these design constraints actually work against you.
I would move the data to a SQL table. You can still use InfoPath and SharePoint, just the heavy lifting and storage and access is handled by the database and is what it was designed to do well.