Someone is using my email address to send emails to my contacts!

Sounds like someone is "spoofing" your address, just specifying it as the sender. Nothing you can do to prevent that, however, if you ensure that correct SPF records are set up on your DNS server, it will make you a less attractive target, and this will probably not happen again.

More here:

https://en.wikipedia.org/wiki/Sender_Policy_Framework

If emails are using both your address as the sender, plus people you're familiar with as recipients (as you seemed to indicate in your question), then it seems unlikely to just be spoofing; it says to me that someone's contacts have been compromised, though not necessarily yours.

@Malmensa is correct though that emails can appear to bounce back to you even when they weren't sent from your account.

It may be the case that someone who shares similar contacts with you has been compromised instead. If you have a number of different bounce message involving different people you know, then consider whether the selection of contacts included might also be included in a friend or colleagues contacts. If so, it might be them that's compromised (and your address being spoofed to send scam/spam email to others). If the contents of the bounced emails look like a scam that friends/colleagues might fall for, then it might be worth warning them. Otherwise, it's just a matter of understanding whose contacts have been compromised and taking appropriate action.

What I remember now is that I received a scam email the other day asking money from me for a medical emergency. It seemed to come from an aunt of mine, which I know wasn't at all possible because that particular aunt of mine is wealthy and doesn't need to ask money from anyone.

So just for fun, I replied to the message anyway, and to confuse the sender I wrote it in my mother tongue instead of in English, the way I should be talking to my aunt.

The next day Message Delivery Failures started to flood my inbox. I'm worried about those sent away that succeeded—there should have been hundreds, if not thousands!

And by the way, the email addresses that came "back" to me were unfamiliar to me. But how do I know that this scammer didn't have access to my contacts? He did it to me with my aunt's email address!

If the account has been compromised, I'd expect to see the bogus outgoing mail messages to appear in the Sent Mail folder.  Depending on whether POP is being used (as opposed to something that syncs such as IMAP or Exchange) it may only appear on the Sent Mail folder on the server (visible through the web interface).

If nothing appears in either Sent Mail folder (local or server), then I would agree that someone is spoofing your address.

In any case, it would be a reasonable thing to change your password as suggested above.

Problem solved.