I've made pickles from many types of fruit and vegetable, but I've never had the chance to pickle capers - because I've never been able to get my hands on the plant.
The caper bush thrives in Mediterranean countries and the small, unopened flower buds and the larger fruit are pickled, the former for capers, the latter for caper berries. Because the caper bush looks similar to many other types of wild plants, it wouldn't be a good idea to randomly pick unopened flower buds from whatever bush you come across on a hike, in case they are poisonous.
Most of us are familiar with capers that have been salted and pickled and are sold in jars. They're usually drained of their liquid and used in dishes to add a salty, acidic pungency. The tiniest capers (usually labelled nonpareil) are considered the best because the texture is not as coarse as the larger ones.
I like them in salsa verde. In a food processor, coarsely chop a large quantity of parsley with garlic, lemon juice and zest, extra-virgin olive oil, drained capers and a cured anchovy or two. Serve the salsa verde as an accompaniment to grilled fish and meats.
Whole capers are also used in the classic French dish of brains with beurre noisette, because the capers' acidity cuts through the richness of the brown butter and the meat.