Depends on what you're using to tumble. Vibratory tumblers with steel shot are going to peen the metal and can create orange peel, especially with soft metal like 24K. The softer the metal, the more pronounced the peening.
Tumbling is fine for removing metal for a more refined finish or for polishing, but done properly it doesn't really work harden anything.
I have used a rotary tumbler with stainless steel media
and I think they have a very gentle action. I don't think it would ruin the inlay.
I actually think they do a better job at "polishing" engraved pieces because it does not remove the fine details like buffing with a wheel would.
That said, I also do not think tumbling would have very much of a work hardening effect.