It's easier to peel the skin of dried seeds. However, to make it more easier, you can soak it in water for 2 hrs and then peel it.
Back home, apart from the white covering, brown skin is also removed. But it seems it has some nutrition and so I didn't remove it completely.
Please keep in mind that the gravy tends to thicken as it rests. Make sure you use mango which is a bit sour to add flavour to the curry, but make sure it's not too overpowering.