I have 2 items , itemA and ItemB. I have experts who choose which items is the best. However, I am more interested in quality then quantity. I have a past prediction success rate as percentage of time the particular expert is "correct" 0 -100%.
I want to find out the best method/formula/statistic to compare the two items.
Examples (items, expert success percentage)
ItemA: 0.8,0.75,0.4 - 3 votes average .65
ItemB:0.9,0.77,0.58,0.5,0.8,0.9,0.4,0.5,0.3 - 9 votes average ~.63
Because itemB had more votes, its average was reduced, however it had most successful experts vote for it.
The solution should not require me defining "success", like defining success as .7 and averaging or counting only experts > .7.
This example is simple, it reality the number of experts vary a lot more 120 versus 2000 experts for 2 items.