Software Complexity

Recently we had an R&D analyst from Microsoft stop by and talk with us. It was an interesting mix in the room 2 .Net individuals an architect and a lead developer, one CFM architect (me), and 3 Java folks one architect, one lead, and one Sr. developer. Scott, the fellow from Microsoft, was visiting there to talk about how we can change the software complexity landscape. Everything was open, compiler checks to static analysis tools. Before I give me thoughts on the discussion, which I will never be able to articulate the massive amount of geek power in that tiny room, allow me to explain what Software Complexity is, in Scott's eyes, because I came into the conversation with a differing idea. Software complexity would probably be better referred to as dependency hell, and then some. We're not just talking about dependency management between classes but the tendency of the lines to blur between layers or frameworks and code starts to bubble to the wrong place, this then leads to dependencies that should not exist or should exists in a different manner. The picture was painted with a prototype tool that could show dependencies, including highlighting cyclical dependencies of dlls. The example was Visual Studio's codebase. Scot explained that when the engineers of Visual Studio took a first look at this tool they immediately said those dependencies (pointing to the core and another package). They also noticed that while they were treating the Visual Studio core as a core, it really was much less a core and much more a wrapper/framework around other cores in the system. I started off the conversation with a non tool based approach in that a large problem in our industry is ITs inability to explain to management, or measure for that matter, the cost of complex poorly designed software. You can have a perfectly built application but then when it comes crunch time instead of evaluating the changes/additions we bang something out where we know it will work. This then gets built upon and banged again and 1 year later we have a bad design. I think it is an industry wide issue where we struggle to quantify with hard numbers the cost. We're developers not management though so we want a tool that will fix this or help us fix/manage software complexity. The discussion that followed took us all over the map from compiler checks that would enforce a macro packaging concept to annotations and class loaders that would choke if the view ever tried to instantiate part of the model. We had some really good ideas and some really bad ideas too; one of the best was my co-worker's idea, the java architect. Have a tool that could extract methods and entirely ignore the package structure and repackage them based on dependencies. It is this repackaging that becomes powerful, in essence creates a procedural application then based purely on method dependency repackage the methods and view the differences. The repackaging may not be better but it would allow you to see how heavily a method is used outside its original class/package. This could give you great insight into large complex systems and help with anything from refactoring to finding better edge cases for regression testing. An off shoot from this idea was also to view method dependency and make sure it always flowed upstream and never regressed back down. Let’s face it as our industry runs further down the OO rat hole these types of problems will become more prevalent, especially as our small systems grow into large systems. There are tools out there to help manage dependencies like maven but these tools are not attacking the fact that framework A and framework B should never have been dependents in the first place. What are your thoughts, sky's the limit, what can we do manage software complexity? Is it a real problem?