This misunderstanding probably stems from the List Description of Listed Builidngs - they tend to describe the outside of the structure. This is so that they can be identified - it is a general description of the building which is Listed, and not a list of the things which are Listed.
Misunderstanding this can cause a lot of problems - and it is something you hear a great deal.

The whole building is Listed - even the kitchen units! However, no Conservation Officer is going to mind you replacing your Magnet kitchen for a Howdens - so long as nothing important is damaged during the works.

Other buildings associated with - or connected to - your building are also protected as they are considered to be Curtilage Structures. There may be especially important features (windows, doors, embellishments) or highly significant parts of a particular structure which are described in more detail, but the whole building is protected.
This is why it is important to carry out a Heritage Impact Assessment if you wish to consider any major works - even to barns or other ancillary structures.