sql server 2008 r2 memory leak

we are running software that utilities sql server 2008, this software seems to run horribly
with 8 workstations using windows 7 this software hangs frequently when making calls to the sql server. sometimes windows even states it is unresponsive.
we checked the indexes on each table and some of the more frequently used tables had indexes of 90% fragmentation and up,
obviously we have been staying on top defraging of that but with 80 databases and 25 tables per database it takes some time.
we have also noticed that over time more errors / lag starts occurring,
and in monitoring the server in a couple weeks the ram usage of sqlserver.exe has climbed from a gig to 7gigs.
is it possible that the service installed on the server by our software vendor is causing a memory leak in sqlserver.exe

What other things could be leading to the issues we are having?

please note: the software vendor installs our software andthe sql server instance,