How do I create a web part to list all calendars in a site?

You should be able to set the filters to show all items of Content Type 'Event' to get the calendar items in a site.

However the formatting to make it intelligible might be a challenge depending what you're after.

If something fancy is needed you'll need to edit item style;

http://www.codeproject.com/Articles/756834/Customizing-the-Content-Query-Web-Part-and-Item-St

I'm not looking to list the events, just a list of the Calendar Apps in the site and hopefully a link to the App when selected.  Maybe an aspx page calling the SharePoint API.

To get something as specific as that, indeed you might need custom code. This could be script calling the client object model rather than something compiled.

If you see the examples here, check out the section on Apply Filters to List Retrieval  - the idea being the list title of "Calendar" could be your filter.