Three prominent caves (coves) can be found in the red sandstone cliff beside the road leading from Waterfoot to the Red Arch. The most notable is the Dry Cave, which was used as a hedge school during the Penal Law period. The cave nearest the village is know as 'Nanny's Cave' and is where Anne (Nan)) Murray lived, she came to Waterfoot when she was 50 years old and lived here rent-free, until 1847, when she died, aged 100. She supported herself by spinning, knitting and making an illicit still known as ‘Poteen.’ Anne navigated the liquor laws by charging a visitor for a drink of water and in return gave them a glass of poteen for free.

a 'Hedge School' is a name given to a rural practice which began during the Penal Law period in Ireland (1723 to 1782) when Catholic schools were forbidden. The Penal Law targeted education by the main Catholic religious orders and sought to force middle-class and Irish Catholics of the gentry to convert to Anglicanism, if they wanted a good education.  Subsequently, educated men and women from the rural areas began teaching in defiance of the law, these schools functioned in places where they would not be easily discovered.

Although the name ‘Hedge School’ suggests that this took place outdoors most teaching would have taken place inside a building, or in the case of Red Bay, inside the Dry Cave. The establishment of the National School system in the 1830s saw the decline in the practice of Hedge schools, though many continued,  more from rural deprivation and need than religious oppression. The third cove was used by a smithy and is known as the Blacksmith’s Cave, one of the caves was also used by fishermen to store nets and equipment.