You dont have to drill with large drills. I use 0,6mm drill for 1mm stones. And If the I prefer not to drill through the metal in case of jewelry. So it will stay stronger. What you can do to clean up the back is. With a slightly larger ball burr that the drill lightly make a facet on the hole. Dont go in to the metal. Just enough to remove the metal burrs on the back. You can even do it by hand. Just push and twist. Then the back will look clean.
Dont drill to large then you will have hardly any metal for the stone's seat.
For (micro)pave work I dont even use setting burrs. I only use ball burrs to make the seats. Drill small hole then use gradually larger ball burrs to get to the stone size.
To remove the metal between the holes you can use a cylinder burr or a tapered burr in a small diameter.
You could also use a small graver.
As for the burrs making a weird pattern in the hole. Try to push harder and see if that fixes it. Also check if all the facets on the burr are intact.
