Software QA Testing Project on srb2

[Captain Obvious]

So, for our Software Quality Assurance and Testing class, we were assigned to find some open source code and run tests on it. Congratulations! I've decided to have our group use srb2! After showing my brethren how awesome this game is (all I've shown have said it's fantastic), we intend to do all we can to break it and document it. Both static and dynamic analysis will be used.

Suggestions?

 

[Rob]

I wouldn't object to stress tests on the netcode and see what you guys come up with for results on that lovely front.

Just be aware that SRB2 is built on hacks. Not completely, but enough of them that you're guaranteed to run into some interesting issues. I am looking forward to what you come up with in your documentation though.

 

[Monster Iestyn]

This sounds interesting, how will your group document your findings exactly?

Should also be no surprise to you that some parts of our code might be somewhat neglected:
* the "sdl12" subfolder was for the old SDL 1.2-specific interface code, which we don't use anymore. Not to be confused with the "sdl" folder which is for the SDL2 code used by srb2win.exe in the latest public release!
*"win32ce" subfolder is for the Windows CE port, I have no idea if that even works anymore or not
* "android" and "nds" subfolders as far as I'm aware are just experimental ports to what-they-say-on-the-tin that haven't (didn't? never will be?) really gotten off the ground at all

 

[fickleheart]

Are you talking full-on unit testing? If so, good luck. The entire codebase is a mess.

That said, if whatever you're doing involves changes to the source (including documentation, refactoring, or whatever), I encourage you guys to shoot us a PR with any changes you make. And if you guys need any help navigating the minefields hidden through the source, feel free to hit up our IRC channel (#srb2fun on irc.esper.net) and someone should be able to assist you.

Honestly, if you guys want specific requests for things to test/document, I'd love to see the software renderer dissected more. (It's somewhat contained within the r_*.c files, for a place to look.) It's probably one of the better organized and documented (I use those terms lightly) parts of the codebase due to not being messed with by too many people, and there's some pretty cool stuff going on back there if you've never dissected the Doom renderer before.

 

[Captain Obvious]

So, you'll be surprised to hear.... it's actually very very low. We might not be doing it right. But both of the static analyzers we've used have shown low errors per LOC. sloccount says the current master branch has 400,040 lines of code (in multiple languages). Cppcheck static analysis said we had:

errors: 175
warnings: 306
stylistic warnings: 1413
portability warnings: 51
performance issues: 3
total info messages:13,581

 

[Captain Obvious]

Quick update: we used two other tools on the code. Flawfinder and Cppdepend. They gave more expected results. I don't have the data at the moment, but the codebase still had markedly lower errors than what any of the devs would probably think.

We haven't done actual "unit tests" because C isn't an object-oriented language, but if there's some way to modulize pieces of srb2, then it would be interesting to break things down.

However, if you broke it down by source code file, about 80% of the errors are in just a couple files. I'll get back to you on which ones. We didn't run the tests on my system.