[Adium-devl] A Unit-Testing Dissertation

Fri Jan 19 05:52:29 UTC 2007

On Jan 19, 2007, at 06:13, Evan Schoenberg wrote:

> Thanks for a great post, Brian.  While I think the burden on our  
> relatively limited development resources would be too great to  
> require unit tests on newly submitted code, verifiable correctness  
> is a wonderful thing to have and we should definitely begin having  
> a methodology of testing, with encouragement toward thorough unit  
> testing of both old and new code.

Hi,

Since Chris assigned me to handle the management of the module  
rewrite/review, I've come up with a plan for that, which already  
includes your suggestion. I'm too lazy to word the whole thing into  
an article right now (it's 6:45am here), so here's the relevant part  
of my chat log (hal2k being myself):

Jan 18 20:28:38 <hal2k>	The_Tick: I'd like to take a wiki-based  
approach to it. Meaning, I'd write up a wiki page listing all  
necessary rewrites/reviews (ordered by importance), and every dev can  
choose to select a specific module he'll work on next. that way, I  
won't command anybody to do too much work in time they don't have,  
and they don't have to do parts they don't like.
Jan 18 20:29:36 <The_Tick>	and it'll be documented to boot
Jan 18 20:29:42 *	The_Tick likes
Jan 18 20:29:44 <hal2k>	The_Tick: that's part 2 of my plan
Jan 18 20:29:55 <hal2k>	actually, part of part 2 :)
Jan 18 20:30:24 <hal2k>	The_Tick: on the top of that wiki page, a few  
important steps would be explained
Jan 18 20:30:39 <The_Tick>	or the steps could be in a separate page
Jan 18 20:30:42 <The_Tick>	linked at the top
Jan 18 20:30:43 <The_Tick>	either way
Jan 18 20:30:45 <hal2k>	The_Tick: the first step in any module is to  
write up the API (the header file basically)
Jan 18 20:30:59 <hal2k>	The_Tick: then, it should be documented using  
headerdoc syntax
Jan 18 20:31:40 <hal2k>	The_Tick: 3, test cases should be written up  
(using ocunit hopefully) -- ideally, this should be done by somebody  
else than the developer working on that module
Jan 18 20:31:48 <hal2k>	The_Tick: 4, the module should be implemented
Jan 18 20:32:00 <The_Tick>	hrmm
Jan 18 20:32:02 <Catfish_Man>	Big Design Up Front, eh?
Jan 18 20:32:11 <The_Tick>	sounds like this coincides with what brian  
said, somewhat
Jan 18 20:32:19 <The_Tick>	Catfish_Man: have an experience with this  
method?
Jan 18 20:32:23 <Catfish_Man>	yes
Jan 18 20:32:26 <hal2k>	Catfish_Man: that's the ordered approach,  
instead of chaos :)
Jan 18 20:32:33 <Catfish_Man>	hal2k: it ignores iteration
Jan 18 20:32:41 <Catfish_Man>	http://en.wikipedia.org/wiki/ 
Big_Design_Up_Front
Jan 18 20:33:02 <hal2k>	Catfish_Man: the issue is that (hopefully)  
multiple modules will be written at the same time
Jan 18 20:33:17 <hal2k>	Catfish_Man: you can't do that if the whole  
API is in flux in the meantime
Jan 18 20:33:21 <Catfish_Man>	true
Jan 18 20:33:45 <Catfish_Man>	but it's also hard to specify an API  
properly before the problem has been explored
Jan 18 20:33:48 <hal2k>	Catfish_Man: changes to the API are possible  
(including test cases), but they should be rare and have a reason
Jan 18 20:33:51 <Catfish_Man>	so I think we're going to end up witha   
compromise
Jan 18 20:34:03 <hal2k>	Catfish_Man: exploring the problem should  
happen before you start coding
Jan 18 20:34:11 <Catfish_Man>	not always possible
Jan 18 20:34:52 <hal2k>	adium isn't rocket science
Jan 18 20:35:31 <Catfish_Man>	"... and that BDUF assumes that  
designers have a "crystal ball" and are able to foresee problem areas  
without extensive prototyping and at least some investment into  
implementation."
Jan 18 20:35:58 <Catfish_Man>	I agree that we should attempt to  
specify the API, but attempting to set it in stone will not work
Jan 18 20:36:16 <hal2k>	Catfish_Man: I didn't consider stones part of  
that wiki page :)
Jan 18 20:36:23 <hal2k>	it's a wiki after all
Jan 18 20:37:07 <Catfish_Man>	sure. I'm just saying that the bar for  
API changes should be set lower than it might otherwise seem. I've  
had to change the wkmv2 spec quite a few times based on  
implementation experience
Jan 18 20:37:52 <hal2k>	Catfish_Man: ok
Jan 18 20:41:18 <hal2k>	Catfish_Man: actually, my idea behind writing  
up the API first was to get some test cases
Jan 18 20:41:42 <Catfish_Man>	ya. That can definitely be useful
Jan 18 20:42:05 <Catfish_Man>	once I figure out this *(&%$#(%*& js  
issue, I'm planning on making an automated tester for the wkmv
Jan 18 20:42:11 <The_Tick>	wouldn't test cases while being written/ 
after writtne be just as useful?
Jan 18 20:42:35 <Catfish_Man>	The_Tick: in general before/during is  
considered to be somewhat more useful than during/after
Jan 18 20:43:02 <hal2k>	yes, because the blackbox test cases  
shouldn't be affected by the implementation
Jan 18 20:43:07 <Catfish_Man>	where during includes later iterations  
(so it's always being added to)
Jan 18 20:43:24 <hal2k>	you can guarantee that they're not affected  
if the implementation doesn't exist yet
Jan 18 20:44:09 <Catfish_Man>	hal2k: some modules will not be  
amenable to this sort of testing, unfortunately
Jan 18 20:44:23 <The_Tick>	ah, makes sense
Jan 18 20:44:31 <hal2k>	Catfish_Man: true, but which ones will be?
Jan 18 20:44:47 <Catfish_Man>	hal2k: the ones that rely on external  
state, in general
Jan 18 20:45:28 <hal2k>	Catfish_Man: well, the usual approach is to  
use some stub objects that simulate the external state for that module
Jan 18 20:45:33 <Catfish_Man>	indeed
Jan 18 20:46:02 <Catfish_Man>	which sometimes works and sometimes  
doesn't. Simulating an AIM server for example would probably require  
more code than Adium ;)
Jan 18 20:46:27 <Catfish_Man>	oh, unless it just sent back pre- 
recorded responses
Jan 18 20:46:33 <Catfish_Man>	that'd still be nontrivial, but not as  
nontrivial
Jan 18 20:47:10 <hal2k>	well, you can actually connect to some AIM  
account for testing, if necessary
Jan 18 20:47:16 <hal2k>	would make the test much slower, though
Jan 18 20:47:20 <Catfish_Man>	mmm... that will lead to spurious test  
failures
Jan 18 20:47:24 <Catfish_Man>	as they mess with their server
Jan 18 20:48:18 <hal2k>	true
Jan 18 20:48:36 <The_Tick>	an irc server
Jan 18 20:48:42 <The_Tick>	I wonder how hard that would be
Jan 18 20:48:53 <The_Tick>	start an irc server that listens only on 127
Jan 18 20:49:16 <Catfish_Man>	hal2k: running a little jabber server  
with a bunch of bots on it could be handy though
Jan 18 20:49:19 <The_Tick>	could integrate itonamaton since it has  
plugin abilities to watch for expected output
Jan 18 20:49:24 <hal2k>	Catfish_Man: yep

Regards,
Andreas Monitzer