Ensuring conceptual integrity in Python project with multiple programmers

Question

One objection that I have often heard raised against Python that it is difficult to synchronize a team of many programmers on large Python project. Note: that synchronization is possible in such a project does not necessarily entail that it is practical, cheap or easy. There's still n*(n-1) communication channels between programmers, so communication cost grows more or less with square of number of programmers, on average.

One thing I liked in Java are interfaces. Sadly, both PEP 245 and even modified Guido's version on interfaces ( http://www.artima.com/forums/flat.jsp?forum=106&thread=87182 ) have not been implemented. I was thinking that duck-typing style limited interfaces (no types, just method names) might be very useful, emphatically not for sake of pychecker, faster code, or forcing (dumb) programmers to follow (smart) architect's design, but as means of easy synchronization between 2+ programmers.

So, apart from typical but not working very well suggestion of "more documentation" (nobody wants to write it really), what are your means of synchronization high-level design and conceptual integrity on such large, multi-person projects?

(in part I ask because large projects I have participated in never used Python)

_{I heard objections mentioned above in person: At least 3 times when talking to startups (2 Scala startups, so they have slant towards static typing) and at least once within corp when considering various programming languages for projects. On the web: can't remember this now, I have vague memory of reading it on some OO blogs and forums. Static typing guys (Java, Scala) seem to have had big on this point, even though typically static typing and interfaces are usually meant for other things (speed, IDE autocompletion).}

_{Note I know TDD defense on this but the problem here is that it is defensive since typically it's one person that writes unit tests. Of course, one could envision 1+ programmers sitting down together and writing unit tests before coding as sort of indirect spec + test but have you seen it done? I haven't. I also feel that unit tests, even written by groups, are not quite (limited) design or spec. They are low level after all. There's a reason (good maybe?) that Java has interfaces in addition to and not instead of JUnit.}

Answer 1

I like GlenH7's answer, and I believe there is truth there.

However, my perspective is that the answer to your problems is Code Ownership.

If five large teams are re-using the same model classes, and want to pile on unnecessary crap into them, then the only way out is to say "Team X owns this code, if you want to change it talk to them". Or, "John owns this code, if you want to change it talk to him."

You still need communication, but the nature of it changes. The assumption is that your request for change will be shot down unless there is a very good reason to put it in. Also, this creates a decision making body with final say, rather than co-owned code being modified by a 'design committee' (which frequently leads to either chaos or analysis paralysis).

Consider this also: there is nothing magical about Java interfaces. They give some design hints from the (hopefully smart) architect, but they don't protect against someone who needs a quick hack from going in there and changing your interfaces anyway. The only thing that really protects from that is code control, code ownership, and some degree of personal responsibility.

So if you're facing "interface" madness, designate responsible people for it and allocate code ownership. Make sure these people do their jobs.

Answer 2

I think you're focusing too much on the language that you are using and not the real challenge that's facing you and your team. Remember that there aren't any silver bullets.

Ultimately, your problem is governance. As governance is really the only way to ensure conceptual integrity for any project.

The need for governance follows either a linear or exponential path correlated to the size of the team. When you add in geographic and cultural divides, you're simply piling on to the need for governance.

There are some simple ways to get started.

If no one wants to write a whole bunch of documentation, then make sure code is commented liberally and correctly. Avoid littering with your comments, but make sure others understand the contracts that each section of code is fulfilling.

If you don't want others accessing particular parts of code and you can't encapsulate it, then you should be putting comments around those areas to indicate what can and can't be used. Even if the language doesn't directly support it, you can still mock up interfaces and closed objects within the code. FWIW, I've written OO style code in C. It can be done.

One of the ways you enforce that encapsulation is code review. The tribal elders from each team needs to review all new code before it's allowed into the build. If you have an architect, then they need to be in constant communication with the tribal elders to make sure they understand the vision and that everyone is in agreement with how the project will be built.

Management has to back the architect and the elders at the first skirmish. Someone is going to break the rules and insist it goes in because you're behind the deadline. Don't do it. Politely slap the violator's wrist; force them to fix the code; then allow the commit. Management has to back missing the deadline to guarantee the integrity of the project. Yes, you missed a deadline but you have to focus on the long term goal.

As mentioned in the comments; everyone should be writing unit tests for their code. Assigning that task to a single person is setting the project up for failure. Use those tests to enforce the encapsulation and rules needed to stabilize your project. If new code doesn't have a unit test then don't allow the requirement to be marked complete.

Ensuring the integrity of a project is everyone's responsibility with each tier taking on differing aspects based upon their own job function.