I would love some input on this from you fellow developers out there, as this is something I continue to try and improve on… WARNING, CODE BELOW THE BREAK!
(Note, I just realized the inline images here are overlapping like hell. Sorry, I’ve changed them to also be links you can click to view completely)
I have to admit that I often struggle with finding the right balance of unit testing, test driven development, what code to unit-test, how much to test it, etc.. A huge portion of code goes (formally/automated-)untested because it’s in the UI and there just aren’t great ways for testing UIs. Due to the unit testing I have setup for the core Chef codebase, most of the bugs that get reported to me have their roots in the UI layer. That says a lot for how helpful unit testing has been for me, but isn’t very comforting when I go to release new versions since I don’t have a way to verify the UI other than manually trudging through everything and testing before each release.
On Chef I think I am balancing this better than I have in the past but I still run up against things that I just am not sure how to tackle with unit testing. I’ll explain.
Due to preference, I have all of the product code completely independent of any unit testing framework and code (NUnit references, test case fixture, etc). I have a single assembly (Chef.Test.dll) which references the Chef api and is where all of my test cases live. Last night I was working on some new features in Chef and found myself needing to unit test a class that I’m going to want to be Private. With my testing setup, the only way I can unit test this code would be to make it Public. So here’s what I compromised with:
1) Expose the collection I need to get to with a Public method, making use of the Conditional attribute to conditionally compile it.
2) Define the NUNIT compiler directive, but only in DEBUG builds since I don’t want this method to exist in Release builds. This is done in the Project Settings, Build tab.
3) In my test case, I can now access the collection I didn’t want to expose.
Before I went to write this up, it was considerably more ugly. I had the following setup to get access to the collection:
I couldn’t stand looking at that ugly syntax so I went with the Conditional attribute, with a cost of making the test case slightly more ugly by forcing the use of a Ref parameter. It’s still far better than what I started with.
This was the first time I’ve used Conditional, so I got to learn a few things about it:
- It cannot be applied to properties, so I had to make GetTokenDictionary instead of a property called TokenDictionary
- It has to be a Void method, so I couldn’t just return the collection
- It cannot use Out parameters, so I had to use Ref (I was getting worried that Ref wasn’t going to be possible, making the whole thing useless to me here)
I could have just used DEBUG, but as those of you who have worked with me in the past know, I tend to like things to be as explicit as possible. I want to have to say “USE THIS ONLY WITH NUNIT, FOOL!”. That’s just how I am.
You may be wondering what the big deal is with exposing that Method/collection as a Public method. Part of it is just OO purism I suppose – it doesn’t need to be exposed, so why should I be forced to?! The other part is due to the obfuscator I run the assemblies through after they get compiled. How I have it configured, some things wouldn’t get mangled if they were Public, and simply exposing that collection as Public would have forced the class TimeUnit to ALSO be public. Which I also didn’t want.
My question is, how have you guys tackled something like this before? I’m open to ideas, I’d love to learn a better way.
The Johnson Blog 
Comments
7 responses to “Unit Testing”
Man, wordpress sure mangled that to hell. Sigh.
LikeLike
What about this little gem?
System.Runtime.CompilerServices.InternalsVisibleToAttribute
You could allow your NUNIT dlls access to non-public types using this attribute. Your example probably wouldn’t work though (i.e., making a private field public to your unit tests dll. I think the attribute only applies to entire types).
You could hide NUNIT code in a partial class with #if NUNIT around it, too, if that does anything for ya. Or hide code specific to exposing things to your NUNIT dlls.
Otherwise, I like the ConditionalAttribute idea.
LikeLike
Hmmm, I didn’t know about InternalsVisibleToAttribute. I don’t think I like that though, it feels way too much like a Friend Class (which I guess is what I’m ultimately trying to do) but isn’t compiled out. I’ll have to remember that one.
Using a Partial Class is something I didn’t think about. I wonder if on your Partial Class declaration if you can use the Conditional attribute to completely not include the partial portion…. I need to try that.
LikeLike
Seems like the perfect place for using some techniques common in Mock-ing for unit tests. Can you not:
1. make the dictionary protected
2. extend the class and trigger the use of the derived class for your test (this second part can be super easy or more tricky, but hacks in this area can generally be avoided too with “contexts” to determine what type of object to create in some situations.
3. in your extended class, have a public accessor for said dictionary?
(you may even be able to promote it to public in your derived class, don’t remember enough C# to weigh in)
-David
LikeLike
I like that David, I’ll keep that in mind for future similar cases. This particular instance is made quite a bit more tricky with the fact that the TimeUnit is also a private class and not exposed to other assemblies, but something could still be done as you suggest.
Very cool, thanks.
LikeLike
I don’t think this will help you much, but it’s perspective might make you feel better.
When you test software for a real-time embedded-system like avionics for an airplane or torpedo, they take safety quite serious and it is a big issue, which proportionally relates to repeatable testing and documentation of such. And all our code was signed off by both the developer and the lead engineer responsible for it, audited by both internal QA and government GAO weenies.
The problem we would run into is using commercially provided compilers was compiler bugs and usage of code libraries in our RTES. There were two lines of thought.
1) Put the vendor compiler company on the hook contractually for authenticating the testing and integrity of their compilers and libraries. Most would only do this for many truckloads of money, in large denominations; or they just laughed at us.
2) Pick a compiler version and stick with it for all your testing, even if new version are available. Upgrading meant re-running all our tests.
Only use the most basic mature compiler libraries i.e. math. Then write much more efficient dedicated code to replace calls to the commercial libraries. If you need a queue, write an efficient custom queue with minimal functions/methods/procedures. The equivalent library generic queue instantiation might be orders of magnitude larger and slower, and contain numerous unused methods which would never be used in our application, but still had to be tested if present. We could often cut the memory and or speed of applications by 25% upwards of 80+%. Plus we now had complete control of the source code. We only had to test what we used, not the entire vendor library. Then test it; and test it; and test it; until you want a new job.
And this is why aerospace projects cost so much (one of the reasons at least). But that’s better than the airplanes falling out of the sky.
Did you know in French, an Airbus avionics s/w bug is pronounced “pilot error”!
In an Arian rocket, the French is “special feature”. In English it translates to “floating point overflow” that should have been caught in testing.
LikeLike
I have heard some stories before about the need for rolling your own for pretty much everything. Having dealt with a few commercial 3rd party packages/libraries, it’s easy to understand this approach.
Did/do any of the 3rd party companies take up the offer to authenticate/test their software up to Defense levels? It seems like that sort of thing could be a heck of a selling point for a software company to have (just guessing, it would cost a forture obviously)
LikeLike