First divide your problem in 2 parts: entire and decimal portion.

You must make sucessive divisions and modulo to the entire portion

double f = 185.80; /* this is your original value */

int entire = (int)floor(f); /* floor will give you the entire portion and (int) will "cast" the value to int */

This is just the starting point. So, to know if value has a $100 component, divide it

int bill;

bill = entire / 100;

if result (bill) is greater than 0, then the value contains one or more $100 bills

now to make next calculus, just rest $100 values from entire portion:

entire -= bill*100;

Well, next is your job, continue dividing, for decimal portion, the behaviour is the inverse.

Go ahead.