This is as-intended.
As a paladin, you really wouldn't have a good IC reason to remove the holy symbol of your deity. If there is some ability/bonus amulet that you would like to wear, you can have the amulet consecrated by a cleric so that it functions as a holy symbol and provides the bonuses.