Encrypt/password protect network share

Here is a MS document on encryption of an SMB share.
SMB security enhancements | Microsoft Docs 

 
On the password request, in a way that technically is already happening if you have the NTFS permissions setup on the folder that is shared.
If the permissions are set then anyone who does not have permissions cannot access the files. (note there are also share permissions that can be set)
For example if you give an AD group named HR full access to a folder but no other groups / users then only an authenticated members of the HR group will have rights to see the files / folders within that share. (note be careful to not remove the system / backup permissions or you can cause other issues such as backup failure)

yeah I get that and yes, thats being done. I'm just trying to take it another step or two if possible

That is all there is natively unless you consider using the Encrypting File System EFS.  
EFS in my experience is ok for 1 or 2 users but is really hard to maintain and administrate for more than a handful of users.
NTFS and Share permissions are usually sufficient for protecting files from access. Bitlocker for files at rest and SMB encryption for network traffic.

I’m not necessarily needing to stick with native, if there’s software that can do that or provide mfa that would be under consideration 

I'm assuming since you are wanting to have each user's access to the encrypted share be done with unique credentials that your attempting to implement some form of AAA. To do this with non-native tools, you're going to need to install 3rd party software which might be defeating the purpose of your original intent (securing them the best way you can).

If securing the files is the goal, you should consider separation, segmentation, and/or DMZ.

what is your timeframe from implementing the new more secure document management system that includes encryption?

I was thinking about moving the file server to dmz and using our duo mfa to secure and requiring them to log into it via rds when needing to access. Does that seem viable?

Sure, make sure you setup a GPO that will not permit the remote users to attach their local storage resources.

Much depends on what extent you need and what you are trying to prevent.

Just trying to prevent a situation where their machine gets compromised and the threat actor has unrestricted access to that network share

Good backups are a must.
You could use VPN that secures All traffic which will limit this by avoiding the situation where the user's System is a bridgehead into your network. The MFA on the RDS is yet another security feature/enforcement mechanism.
note you have to limit what the people connecting to your VPN can access.
You do not want the SecureAll option, but then allow all traffic from the client to travel through your network and out, finalizing a connection if the system is compromised.

Do you have a vendor software that deals with security and auditing of access to these documents? is that what you are considering to transition to?

Backups are fine, I have multiple locations as well as immutable. Since the nature of our business it’s more about customer confidence and dealing with their info. Considered throwing the data on azure and using their mfa as well

you could, make sure though to discuss the security precautions you want to be implemented including limiting IT staff from Azure being able to access your data.
Much relies on you.

Azure might be a better option given your systems are windows 2008.

my systems are 2016/2019