Hi Jim, thanks and welcome
I see that they are contained in PNG files with the characters in sort of a grid. I have seen this kind of technique used in other things as well.
Check out the manual included with the REXPaint download (manual.txt). It covers all of REXPaint's supported features, including a section of particular relevance in this case called "Custom and Extended Fonts"
Also, how do I get hold of the font/character set that you used in your demo pictures with the dungeon graphics?
The original tileset by Kenney is available on their site (linked in the
original v1.50 announcement, assuming I'm correctly guessing which one you're referring to), though the reformatted version for REXPaint that I used for testing was only released on Patreon, where I also releases a few others. You could use a raster art program to take their original and make your own version using the parts you need.