you have to increase the theme input bitmaps.
Here are the steps i would use:
Step 1: search button and button frame + maximixed on skinstudio and export it to a png file.
please read this here and als click on blue forwrd button site to read where to get the buttons : http://www.stardock.com/products/skinstudio/help/changing_titlebars_and_borders.htm
Step 2: use a good upscaling alogrithm program:
a) ImageResizer.exe image2d hawkynt google projects image2d
2xbrh or 3xbrlv4 from pixeljoint forums.
both do work better than nearest neighbur/bicubic or Lancsoz, ImageResizer.exe does offer a lot of algorithms (EPX3,Hqx,xbr are the ones that might interest you) , the other is a small cmd app.
Step 3: Downscale with nearest neighbour or another bicubic algorithm ( often called none could be done with imageresizer too, to the size you want to use and export to png)
Step 4: reimport it in skinstudio
Step 5: increase the caption heigth (depending on the theme more difficult, use a simple image eitor and the programs above)
step 6: import them in skinstudio too
step 7: Enjoy
The Problem is that you have to increase the real image size , and that is not always a simple task.I did write you the link for the best upscalers i know, but sometimes you might to use adaptive blur on the images after that. Perhaps you may send me the theme you want to in a pm and your wanted caption heigth ( only if its a free theme )since it is indeed a task that can take up to 15 minutes (normally 2 for all simple buttons) ( or an hour if you don`t know how ).
i hope i could help you. The dpi thing can only be solved in assuming that either existing skins do need to be upscaled with good algorithms or that the in applciation dpi size has to be changed ( both are not a simple task for stardock, so don`t expect soon updates from them. Try windows own cusomized dpi app scaling or change the theme/skin itself.).
The same problem exists in the linux world too. They did now use a html/css and svg based skinning method on both popular platforms gtk+ and qt.
Decreasing monitor resolutions isn`t a solution here b.t.w