Ah okay, now I get it, thanks for the example. Yeah that's not possible because the inventory can only show up to 10 items at once, i.e. the digits 1~0, and therefore many of your items won't even have a number associated with them. The swapping feature is also at its most useful when the inventory is beyond 10 items, e.g. many combat-oriented Cogminds are carrying 15-25 items by the mid game, and therefore many of them won't be listed without scrolling. (If it helps, notice that the items are not only reduced to a single category, but also alphabetized as listed for the swap.)
Also, you can already do keyboard-based swapping directly using only visible "labels" (reference numbers/letters), via the original method we've had since the 7DRL: Hold Ctrl-Shift and press the letter you want to swap out, followed by the number you want to swap into its place. That does exactly what you're looking for. So for example:
If I look at the screen and decide, 'ok I want to swap I and 2' then I can press / and then 2I and it will work, but I can't press I2 and have it work.
In the above scenario, hold Ctrl-Shift and press 'i' followed by '2' and it will swap those two. No modal '/' necessary.
Beginners will mostly be using the mouse, in which case drag-drop is the easiest way to swap parts, while keyboard players have more than one option available depending on what's most convenient at the time.