Clear out MSDB file in sql server

This project is using SQL SERVER 2012 and on the server it has around 1500 databases.

In a single SQL Server instance? What's the average database size and how large is the biggest one?

I am noticing that SQL Server is consuming all the assigned memory (which is around 40GB).

That's the expected behavior even a minimum of 8GB should be reserved for the operating system.

Many time Transactions or query are getting timeout by SQL Server these days.

You might need to add more memory or a better solution is to build a 2nd SQL Server box and migrate some of the databases there.

I checked and found that msdb file has grown to 68 GB in size. I am not sure that this is causing all these slow ness.

Is not. How much of those 68GB is data and how much is transaction log?

I tried to run the maintenance plan to clear out msdb but it just did not get ended.

Which maintenance plan are you running? What you should do is to configure the SQL Agent to keep a shorter period of jobs history since having 1500 databases it should produce a lot of lines for each maintenance task, for sure. And not counting with each database specific job from their respective applications.