Once you have removed all active tasks relating to an existing category you will then be presented with the Delete Category option button to delete the category all together.
However if there are still tasks linked with the category then you will not be shown that delete category button.