First, let me give you some basic information:
Every type of armor has a base armor rating (a number). For example, on leather armor this number is 5. This means that leather armor provides an armor rating of 5 against all attacks, regardless of damage type. It then has Crush 3 and Freeze 3, this means that instead of 5, it only provides an armor rating of 3 against Crush attacks and Freeze attacks. Without understanding anything else about the system, what this should tell you is that people in leather armor are more vulnerable to crushing and freezing damage.
The next thing you have to look at is the type of damage you're dealing and the penetration value of that damage. Every weapon and damaging spell or ability has a penetration value listed. This value is compared against the armor's armor rating against that damage type.
So if we use the above example of leather armor and we use, for instance, a fine war hammer which does piercing or crushing damage, the interaction would look like this (simplified):
I attack (my accuracy against your deflection), I score a regular hit. My war hammer has a penetration value of 9, your armor has an armor rating of 5 against piercing and 3 against crushing, the game will use crushing damage for the attack since it will hit harder. Because my penetration value of 9 is more than double the armor rating of your defense, my attack will do 30% more damage (this is called overpenatration).
If the reverse were true and I were attacking your fine brigadine armor with an armor rating of 10 against my club with a penetration value of 7, then my attack would do -70% damage (this is called no penetration).
It sounds like what is happening is you're using an attack type that the target is extremely resistant to so you're dealing very little, if any, damage to your target when you hit.
You can see the relative stats of your target by hovering over them in combat, you'll see a list of their defenses and armor ratings against certain attacks. Make sure you're targeting their weakest ratings if you wish to drop them quickly and easily.