The last time I tested this a few years ago, it went like this:
1. Remove 1% castable resistance for every point of debuff delve value, until castable resistance hits zero.
2. Remove 1% of any remaining resistance (first item, then racial) for every two points of remaining debuff delve value. This can take your effective resistance below zero.
3. Remove 1% of any remaining item resistance for every point of spell resistance piercing. This cannot go below zero.
I don't remember if spell pierce can remove racial resists or not, at step 3. I didn't include it, but that could be incorrect.
Edit: Also note that debuffs affect tier 1 resists only. All tier 2 resists such as AOM, Empty Mind, innate tank resists, and Fury multiply against whatever damage is calculated for tier 1.
-----signature-----
Alb Tristan <Wraiths of Albion>
Asilithiel, Speakssoftly, Kegnor, Kesteral, Badgers, Amarynthia
Hib Galahad <Juggernaut>
Hulainn, Hansil, Cythrael
"Friars are in great need of a donkey. We need something to carry our ale."