You have been honest and said this is homework, I will not do it but will point you in the right direction (doing hoemwork is against the membership agreement). You have formulated the problem in correctly

Firstly find out the thing you need to minimise ie cost of material which is surface area related

area top and bottom =2wl

area sides =2hl+2wh

So cost C = 0.2*2wl + 0.1*(2hl+2wh)

but the volume must be 512 so the wlh= 512

ie the problem is to minimise C

C = 0.2*2wl + 0.1*(2hl+2wh)

subject to wlh= 512

There are a number of ways of doing this but it looks if you are just looking at the partials (ie no langrange multipliers). The first thing is to eliminate one of the varaiblee from the cost equation using the conatraint, say substitute w=512/lh. Now minimise the cost, ie calculate dC/dl and dC/dh and equate to zero. You can do the rest