Ok, I have been testing the weapon proficiency abilities again and found that I was able to turn them off when I tried fighting some old monsters again. It seems the game fixed itself or I missed something? Basically what I did was to click half sword after I see the icon cool down effect wear off, I could toggle it off. Now I can properly turn it off after I retry the Sea Caves from the start of the game.