This was a regression from 44691448911f1d29d4d79dbdd5553734761e57c4
The callback should be called, not only be called when the refcount
reaches 0. For example, the C# callback needs to know when the
refcount reaches 1, in order to swap to a weak GC handle.