I'm pretty sure software rendering is done entirely on the CPU, while OpenGL rendering is done on the GPU (thus increasing performance for toasters bad and old computers with "good" GPUs and such).
Besides that, software can render colourmaps correctly (changing the hue and saturation of stuff), can render ripples on top of water (and other) FOFs (if the FOF is set up to have ripples), can have specialized palette changes (OpenGL is hard-coded to just tint the screen, I think), and can render "portal" lines, but is limited to just 256 colours, thus dark halls and such may "flicker" and/or get a gray or otherwise weird colour and such. Also the ground textures may flicker around horizontally, as if the ground is shaking a little, when moving around in complex-shaped sectors.
OpenGL renders colormaps by just tinting the insides (so a red object in a blue colourmap will appear black in OpenGL, while blue in software's hue-changing-ness).
Sadly, OpenGL can not render ripples on the surface of FOFs (but the screen will still wave softly when the camera is submerged in water), but it's not really a major thing, so that's okay.
As said above in the software section, palette changes, such as when using the Armaggeddon Shield's blast, being teleported, etcetera, have different "sub"-palettes which can be customized by custom palettes, while OpenGL just tints the screen with probably-hard-coded colours (which as a result also affects MD2s and such).
Speaking of MD2s, yeah, they're OpenGL-exclusive. Personally I prefer the sprited look, however, but I understand that other people may prefer MD2s.
As for portal lines, they're not used anywhere(?) in vanilla SRB2, and their use for/in custom maps is limited, so the lack of support for them is not too bad.
Also, OpenGL is limited to 16777216 (256x256x256) colours instead of 256, thus making dark halls, translucent objects, etcetera appear much smoother. Whether that's a good or bad thing is subjective. I personally prefer OpenGL for just that reason.
And the flickering ground textures when moving? If it's not completely gone, it's definitely far less noticable than in software.
Additionally, OpenGL supports the "shadow" console variable, which software doesn't support.
Edit: Oh, and you can use texture filtering in OpenGL to smooth out textures or add mip-maps to make far textures look less grainy, but this will leave ugly edges around translucent objects (such as basically all sprited objects), so I don't really recommend using that.
Edit 2: Also, another huge thing is that looking up/down in software is much different from OpenGL.
One last thing... When playing in software mode in non-green resolutions (multiples of 320x200), software will render in the aspect ratio the resolution would have, while OpenGL renders everything (the world, the HUD, everything) in locked 8:5 aspect ratio (just like the green resolutions), and just "stretches" it out over the chosen resolution. But generally, this shouldn't matter a lot at all for you (unless you custom-compile a version of the game to have ultra-wide (or ultra-tall) resolutions). At least I'm pretty sure that is the case, judging by a few extreme tests with 1920x1080 resolution (16:9 (or 8:4.5) aspect ratio).
If there are any points about this which I'm wrong on or missed, please correct me.
Edit: So TL;DR: Stick with OpenGL for quality, or software for stylism.