Preheat oven to 350F.
Evenly coat the sides and bottom of the square baker with softened butter.
Strain the black olives from their brine, rinsing thoroughly three times to remove any excess brine. Pat the olives dry and cut in half, spreading them evenly on the bottom of the baker.
In a mixing bowl, whisk together the milk and eggs until combined. Whisk in the sugar and flour.
Pour the prepared batter over top of the olives.
Bake uncovered at 350F for 25-30 minutes, until the top is golden brown and the batter is set.
Cool before slicing and serving.