I dropped in at Kakulas Sister Grocer on Market street while I was in Fremantle. This is a Fremantle outlet of Kakulas bros in Northbridge. I think this is the best place to buy olives.
The small store is full of European, Middle Eastern grocery, quality cheese, olives and sweets. The unique product/grocer ranges make you feel incredibly creative (and hungry).