External hardware used to be far more important than it is now. In terms of Palm devices and devices powered by the Palm OS, there were only three external hardware devices that I was aware of.
Swivel Systems SG20
The SG20 module was the first external module I discovered some years ago. I don't know how long it was made for or how many were sold but it was a very niche product to say the least. The SG20 was an innovative product for its time. It was a GM module housed in palm modem casing (or very similar) that clipped on to the bottom of a Palm III series PDA, or a TRG Pro. It would also fit a Palm V or Vx with a special adapter.
The sound quality was very good indeed using a 3.5 stereo jack socket. But the SG20 didn't stop there, it also had MIDI out capability
The Tsunamidi for Palm V
This tone module offers a full General MIDI sound set and wavetable synthesis. This means you'll have 128 different instrument sounds plus drums clipped onto your Palm V or Palm Vx handheld!
The Tsunamidi is about the same size as the Palm V Modem. It takes two AA batteries and doesn't use any additional power from the handheld. It has a built-in speaker and headphone jack and a port for an AC power adapter. There is also a special cable (included) that gives you full sized MIDI-in and MIDI-out ports to connect to any electronic music instruments or equipment.
I owned one of these for a while, and it was a nice device to have, but in many ways very similar to the SG20 device.
This module only works in the Handspring visor PDA series. It is almost impossible to find these days. However, the Yamaha chip used in the module is the same as the one used in several of Sony's Clie devices.
The module had a 3.5 mm headphone jack and a tiny speaker too. The applications that worked with it were miniMusic's suite of software.
This was a MIDI adapter for Palm PDAs prior to Palm OS 5. It converts any Serial HotSync Cradle or Cable into a MIDI-out interface that you can connect to any electronic music instrument (requires full RS2332 serial signal, some handhelds require special cables to meet this specification). There are some low cost handhelds that include ONLY a USB port instead of a full HotSync connector, like the Palm Zire, preventing anyone from making a serial cable and preventing a MIDI connection of any kind.
These are still available from miniMusic
Other external hardware options
Of course you could connect other MIDI hardware. With the exception of most OS5 Palms serial out was supported. People have used the Yamaha series of portable MIDI studios such as the QY100 or the MU15.
As for Windows Mobile / Pocket PC devices, there was ages ago a MIDI adapter for some of the iPaq series called MIDI Port, but I haven't seen them for a long time.