The simplest solution is to see if it's possible to combine the modes so that they're no longer modes at all. Originally the purpose was to more easily separate map look mode from everything else so it was easier to implement (done during the 7DRL...). Since then there are also a few input behaviors that work differently between mouse and keyboard mode. Merging them and having the cursor auto-hide on keyboard and unhide itself on mouse input would be ideal. I'll have to investigate.
I would also add that we need a way to scroll through a large inventory without using the mouse and scroll w heel but I feel like that might already be possible and I just don't know what button it is.
'[' and ']' to scroll, plus 't', 'm', and 'i' to sort by different criteria, and press again to reverse-sort. (So by using just 't' you can quickly see all items in a size 20 inventory.)
I should also add Ctrl-[/] to allow for paging up/down in an inventory via keyboard, rather than one at a time.