The death of the generalist software developer

This is my first major followup to my post on why a career in computer programming sucks. It probably won't be of much interest to this blog's regular readers who come her to read my articles on economics, politics, evolutionary psychology, and social class.

In my previous post, I explained that a major reason why a career in computer programming sucks is because of the declining value of the computer programmer's knowledge capital. A new software development platform comes out, becomes popular, and the programmers who are too late in jumping onto the bandwagon become obsolete because no one wants to hire someone without two or more years of real world work experience with the new platform.

A common protest to the above point, repeated in the many comments to the original post as well as in various posts at other blogs, is that computer programmers are valued for their generalist knowledge, and the knowledge of a particular language or development tool is only minor.

The concept of the generalist software developer is quickly becoming obsolete. There are two separate specialization trends in operation.

1. TECHNOLOGY SPECIALIZATION

The first trend is technology based, and that's the increasing complexity of the software development tools. I've witnessed this evolution firsthand. Visual Basic has long been a standard tool used to develop corporate client/server applications. Visual Basic 3.0 was a very simple tool, without much in the way of big libraries of functions. This limited what you can do with it, but it made it a lot easier for a new programmer to learn how to use it. Visual Basic became more complicated with newer iterations. Microsoft made it object oriented, and added the ability to program your own COM DLLs. And then Microsoft finally threw it out in favor of the .NET platform.

The interesting thing about .NET is that there are various language flavors to it. There's Visual Basic.NET, C#.NET and even J#.NET. This tips us off that the actual language is only a minor part of the whole development tool. Much more complicated is the huge class libarary, and understanding the idiosyncracies of how everything works. I have no doubt that a programmer who knows how to write a sorting algorithm in Ada or Pascal, but has never used Visual Basic.NET, could nevertheless also figure out how to writing a sorting algorithm in Visual Basic.NET. But here's the kicker: the very fact that the programmer is writing a sorting alorithm at all indicates he doesn't understand the tool. The class library contains various collection objects that will already sort stuff for you. Fancy coding expertise, like the ability to write a Quicksort, is not necessary in the modern software development environment. Knowing how to set everything up so you can call Object.Sort() is what's required.

Because the new technology is so much more complicated than the old technology, it's now impossible for one person to be an expert in all the different tools out there, and even in a single project this is often impossible. In a single project, you will often find database programmers who just do back end work in Oracle or SQL Server, middleware programmers who would do the C#.NET type programming, and then web designers who do the HTML, CSS, and JavaScript programming. It gets even more complicated if the project involves getting two systems, programmed in completely different technologies, to talk to each other. And these types of integration projects are, unfortunately, all too common.

I know there are some programmers out there who think they can easily learn a new tool. Some of them can do it easily, others can do it with a lot more difficulty. A hiring manager, who's goal is to minimize risk, will not hire the programmer who has to learn a new tool, he's going to hire the programmer who already knows it. Furthermore, I think that people who can quickly become experts with a complicated development tool are extremely rare. I've been using .NET for five years and I'm still discovering new stuff. (Although I have to admit that I no longer live and breathe this stuff. There are dedicated computer programmers who spend their weekends learning new programming stuff. They can surely reach my level of expertise in less than five years.)

2. BUSINESS SPECIALIZATION

The second trend, and the more interesting one in my opinion, is that the computer programmer's role in the organization is shrinking. The software developer who does everything--talks to users, develops the "architecture," codes it, tests it, and then deploys it, is pretty rare. In fact, it would probably seen as an example of an out of control software development process. It should be noted that such a process is not truly out of control, it's just being controlled by the software developers instead of the managers.

In the modern paradigm there is a software development process but no software developers. Instead, there are business analysts, project managers, maybe even a product manager, technical writers, quality assurance (QA) "engineers," and finally the computer programmers from whom no creativity or general expertise is required or even desired.

In the modern paradigm, the product manager and the business analysts will be the ones who talk to the users and figure out what needs to be programmed. The business analysts will write up detailed specifications, which are handed to the technical manager who, in turn, assigns it to a programmer and tells him "go program this." At this point in time, the project managers come in and oversee the project, constantly asking the programmers to estimate how long stuff will take. (Smart programmers eventually learn that they should always over-estimate how long stuff will take, although for some strange reason, many programmers who are smart enough to figure out how reprogram Vista in assembly language are too stupid to figure out that they need to aggressively over-estimate everything.)

The complete code is then handed off to the QA department who will test it and find bugs (and there will always be bugs because, with so much control taken away from the progammer, there is no longer any pride of ownership) and toss it back to the programmer to fix. Eventually, the application will be deployed by network "engineers."

In all phases of the controlled software development process, a huge amount of paperwork is generated. The programmer will be asked to contribute to this papework, with time estimates, unit test cases, interface documentation, etc. No one ever reads any of this paperwork, and this makes the creation of it all that much more depressing.

Specialization has the effect of lowering salaries. It's a lot harder to become an expert at everything than it is to become an expert at just one particular task. Futhermore, there's a glut of people who want to do the cool jobs like business analyst or project manager. (Being a project manager actually sucks, but the "manager" title fools people.) But maybe there's a reason there's a glut of such generalist people. Even as the computer languages change, the project managers will still be making the same "gant charts" and the business analysts will still be doing the same analysis. They have more career stability than the computer programmer who becomes obsolete when a new technology comes out.

Specilization allows the use of cheap foreign computer programmers, because we eliminate the problem of foreigners with poor English skills being unable to do much besides write code. Now, writing code is all that the programmer has to do.

This business specialization prevents the computer programmer from "moving up" into other roles. A company looking to hire a business analyst or project manager with five years of experience will not be interested in a computer programmer with zero years of formal experience doing these functions full time. Furthemore, these roles generally pay less money than computer programming. Not many computer programmers will want to give up their computer programmer salaries to make $30,000/year as entry level business analysts or junior project managers.

Comments

[JewishAtheist]

You really have a problem confusing the specific with the general.

All liberals are not the rich Manhattan liberals you happen to be acquainted with and all programming jobs aren't like the ones in your company.

[bbartlog]

Specialization has the effect of lowering salaries

This isn't generally true. Of course if the development process has been changed so that the work can be broken down in such a way that lower-skilled people can do easier, more discrete tasks, then skill requirements and salaries may drop. But the overall picture you paint is kind of self-contradictory; if there really is a constant need to learn new technologies, and it's difficult to do so, then salaries should rise.

[guest]

the computer programmer who becomes obsolete when a new technology comes out.

Except that reasonably good programmers do not become obsolete when a new language comes out. Why is Half Sigma unable to understand this?

[mondo]

There really shouldn't be a constant need to learn new technologies. Some egotistical, super high-IQ PhD computer scientist at Microsoft, Sun, Bell Labs decides to invent a new language that'll make histhe future better, and marketing departments create the necessary spinevangelism to get customers to utilize their latest snake oil.

Anything corporatized loses its soul. I'm sure there are plenty of parallel examples in BIGLAW, finance, medicine of overspecialization and pigeonholing.

I don't think there's really any reason to get into a programming discipline unless you have some crazy idea and are willing to hack it together using whatever's on hand - Perl, Java, VB, or whatnot. Look how Microsoft, Yahoo! and Google started. Most projects I've successfully completed (stuff that still performs after years of use even) involved learning some language on the fly and liberal use of band-aid hacks and square pegs in round holes.

I work in a mixed engineering and architectural environment. That programmers don't have state-level professional licensing standards and more commonly-accepted professional organizations and standardization practices is the real drag on career satisfaction and effectiveness.

A company looking to hire a business analyst or project manager with five years of experience will not be interested in a computer programmer with zero years of formal experience doing these functions full time.

I dont know whats the matter with your company, or maybe the problem is that you are a straight contractor that no one is interested in investing in.

What you do it go to management and say, "hey I am interested in helping out with project management for the Widget X project. " You get put under the overall PM to help out with some specific area and roll up your tasks to his mpp. PM's like this because they might not have understood all the dependencies in the first place, so you bring technical knowledge. Delivery managers like it because it saves them support time when someone who knows their area identifies task early.

I see people do this *all* the time. It really is quite easy to segue into project management. And it pays better than you think.

[Thomas Hansen]

Regarding your statement (in your previous blog) about "who cares if you know Cobol?"...
Cobol developers can basically decide their OWN salary simply because it's so "un-hip" to be in the Cobol world that nobody wants to do it and there's virtually BILLIONS of TONS of Cobol code that needs to be maintained!
Check up your facts... ;)

.t

[Former Programmer]

HS is right again. These are precisely the reasons I got out of software development, even though I couldn't articulate them at the time. As an organization grows, there is a gradual evolution towards specialization. Pretty soon you're left doing a "code monkey" job unless you want to go non-technical.

[Loki on the run]

People who can handle complexity are continually in demand.

I don't see employers being able to hire people from off the street who can understand how to understand and avoid deadlock in complicated systems.

Moreover, I am into my sixth decade now, and I get enquiries about joining this or that startup at least every month.

[Brandon Berg]

HS:
I've gathered that you currently work in the IT department of a non-software company. Have you ever actually worked in the software industry?

Because while I can believe your posts on the topic accurately describe how things work in the IT departments of non-software companies, the things you've said aren't consistent with my experience at a major software firm.

Also, it's kind of silly to mention quicksort as an example of "fancy coding expertise." We covered quicksort in my introductory high school programming class. And again in my introductory college programming class which even non-CS majors had to take (this was Georgia Tech). And any idiot can copy it out of a book, even if he has no idea how it works.

"Fancy coding expertise" isn't about having well-known algorithms committed to memory. It's about writing code that's reliable and maintainable even when it grows to hundreds of thousands, or even millions, of lines (or being able to straighten out the mess of someone who can't). It's about finding good solutions to hard, non-standard problems. It's about writing and debugging complex multithreaded code, which is still hard in any language, and which is going to become more important with the trend towards multicore processors.

If you've never done it before, you should try getting a job at a real software company, just to see what it's like.

By the way, how did you get into programming? It's a bit unusual for someone with your educational background.

[The Jetman]

There's a lot of truth to what you say. It was great, like back in the '80s, when there were fewer leading-edge tools (less powerful, more labor-intensive) and greater opportunity/person. There's still a lot of opportunity, but as you say profiency across a broad range of tools/systems can be extremely tough. I have many great tools, incl the .NET/VS suite, that I can't take time to get really good with, bec I'm too busy paying the rent, with more mundane things.

I've been out of the corporate world for many years now, but I'm still an avid reader of SD Times, eWeek, Infoworld, and various websites of maj trade pubs and such. If generalists are so valuable, why is there a constant, annual debate about bumping H-1B visa limits ? Why are maj corporations trying to completely divest internal development via outsourcing ?

I know w/o any doubt that HR people/headhunters have a very hard time assessing real-world tech proficiency, specialist or generalist. When I was doing the corporate thing, I was playing on both sides of the hiring "street", simultaneously as a supervisor hiring people and as a screener for local headhunters. They didn't understand what really made a good candidate then and I seriously doubt things have improved in the intervening years. My point is that the people on frontlines indirectly have a big impact on issues like this.

There are many forces at play here, beyond the the points you've made, but your mini-essay has made for interesting reading. Later....

[Karl]

It'd be nice if all places were like you list. Small-medium businesses don't have separate managers and developers and qa teams and testers or even paperwork trails. It still takes a generalist to run these kind of shops with a dozen or so developers, and fifty or sixty staff.

[Alex]

> In the modern paradigm...

Well, depends who you are talking to. My impression is that you simply work in a less than stellar corporate IT department run by PHBs, and mistake your local situation for the going trend in the bigger world.

In my world, the "modern" paradigm you are talking about is considered obsolete, and being replaced by methodologies that *de*emphasize formal process and specialization.

By the way, interpretation of trends non-withstanding, anyone suggesting that with the right kind of process a bunch of code monkeys working off detailed specifications can (or ever will) predictably produce quality software at reasonable price should really be laughed at.

[Meet A Developer]

hire web developer / designer for 10 USD per hour http://www.meetadeveloper.com/hire-a-programmer-hire-a-dedicated-developer-hire-a-web-developer-india.html

[SFG]

A piece of spam that's ironically relevant to the discussion! Wow!

Interesting that there is such a diversified response to the article. Being one with an MIS degree and having worked in both Sales and as the Director of Technology in a couple of organizations I can honestly say that virtually all of the Analyst (either Systems or Business) want coding experience with extensive experience. They do not want to hire people who might be able to learn it, they want to hire the people who know it... and they leave the position open until they find what they want. The IT world has become complex and varied and it is hard to get a new position, other than entry-level, as a technology generalist and harder still to find one when you have too-deep a specialization in one technology. While I wish I saw something different I have to agree with many of the statements being made.

[Len Reinhart]

Tools like Visual Studio .NET make it possible for a generalist to pull together a fairly complex application in a reasonable amount of time. There is no need for a marching army of coders. What is needed is the experience of the generalist to avoid the many pitfalls that will lead to disaster in a very complex process. It is the experience of the generalist that allows application of a tool like Visual Studio without learning a bunch of detail about pieces of it that are not being applied to the problem. I know what a regular expression handler does, I don't need to study all the detail of how to use all it's features. I just need to find out how to make it do what I need it to do. My experience tells me that it will a good place to look for a solution a specific problem (as an example).

[Bob]

Dev shops are mobs. There is no rule there is only chaos.

In the GOOD dev shops, the mob is a bunch of solid engineers who wont let anyone spoil their fun at creating great solutions.

In bad shops, the mob is commonly management or process losers who get off on taking developers (who are generally smarter and better paid than themselves) to task to compensate for their lack of ability to create anything of real value to anyone.

So if you want a good programming job, find that good mob and avoid the evil mobs.

Not that some bad shops are a mob of bad programmers, but usually these get superceded very quickly by a process mob or a mgmt mob because of their failure at creating good solutions. So it ends up being the same in the end. A bad mob of programmers will lose their ability to make even simple choices very quickly.

[Bob]

Dev shops are mobs. There is no rule there is only chaos.

In the GOOD dev shops, the mob is a bunch of solid engineers who wont let anyone spoil their fun at creating great solutions.

In bad shops, the mob is commonly management or process losers who get off on taking developers (who are generally smarter and better paid than themselves) to task to compensate for their lack of ability to create anything of real value to anyone.

So if you want a good programming job, find that good mob and avoid the evil mobs.

Not that some bad shops are a mob of bad programmers, but usually these get superceded very quickly by a process mob or a mgmt mob because of their failure at creating good solutions. So it ends up being the same in the end. A bad mob of programmers will lose their ability to make even simple choices very quickly.

[T]

HS - this article and your "why a career" post echo my sentiments exactly about the current IT industry (in fact, we must be working for the same employer). I see a lot of critics posting comments, and I think it comes down to who you consider your peers.

If you work at a "good dev shop", your friends work there, you have no desire to be a real decision maker, and you keep the blinders on, I can see how it might be possible to stay content as a developer.

However, many of my peers are on the business side, MBA holders, executives, etc...to hear how IT is viewed by the real decision makers of a company...well, it's almost impossible to consider IT contributions as more than an after-thought.

Critics - don't get me wrong - I'm a passionate techy that contributes to dev projects during my free time, but imho the truth is - unless your job is in a top-line-generating part of a business (sales, marketing, billable-hrs, M&A), you'll always be viewed as cost to be minimized. I don't measure myself by other people's opinions, but it's hard to see the prestige or opportunity-for-reward in being a commoditized cost.

As much as I love tech, I can't see a future in it that leads further than middle-management in a suburb. I'm going to have to come to terms with that, or change directions quickly.

[Elvis Presley]

HS is a fucking moron. These two articles are the biggest pieces of crap I've read in a blog, period. We all saw during the dot com boom the rush of programmers coming into the market, then the implosion, and most of them went home. That's fine. But saying that programming as a career sucks is totally missing the point. You should do what you love doing, and for a lot of people that's programming. So, being a programmer should be what you are driven to be, and the rest be damned.

One thing that was especially annoying was the generalist argument. Being able to cut and paste Object.Sort() after googling for the syntax is fine. But let's say you've actually coded a quicksort algorithm in college, then you probably know when to use quicksort, what the O-notation behavior is for quicksort on random and nearly ordered lists, when to use it, and when not to. You probably know about insertion sort and skip lists and binary trees, and when to use them, and what the trade-off's are. See, this kind of generalist knowledge is all part of what it means to be a programmer, not just some hack.

I would love to see the types of programs HS produces. Fucking moron.

[anon]

Just leave aside the flame-warriors man and keep on posting very good articles like this one.

[Tech Lead]

Everything in these 2 articles is pretty much spot on.

The only bright spot for programmers are positions like Technical Lead and Application Architecture. Technical Lead jobs can give you higher pay, at the cost of being a little less hands on. But once you have this position nailed down, there is still nowhere to go. You're not a Manager, even though you may manage the technical tasks and day to day issues of 3-7 developers. You're not a Project Manager, even though you basically do their job for them. So its a bright spot, but not much of one.

The architecture route I have yet to travel down, but I feel pulled towards it even though I've never really seen anyone in architecture do anything. But hey, if they want to pay me more and give me more prestige, while doing less, who am I to complain.

[Vlad Patryshev]

I think you are pretty much ignorant regarding both the programmer's profession and what's going on in the industry in general.

I've seen too many generations of slackers that lose it and declare programming dead. It did not happen so far, but those generations had to look for something to do.

[Alex]

Good writing.

There is such a wide range of programming skills that I think it brings a top C++ programmer down to a common denominator and the Flash programmer up to a common denominator.

I created my own Will using a software package so that now qualifies me to be a trial lawyer, I guess would be the example.

Distinguish between programmning and computer science. A career in computer science is very cool and very fun and well paid but you really need to complete a PHD.

Of course a programming career is better then most.

Picking grapes in the central valley in the middle of summer sucks.

[Alex]

I also believe computer programming is actually a trade and should be distinguished from computer science.

Why do you Devry teaches programming next door to the plumbing class?

In fact my father was a bricklayer and I believe tradesmen (plumbers, bricklayers) can make the same as programmers and stay healthier due to the physical nature of the work.

Now a computer scientist career involves working on interesting project similar to say a medical researcher.

For you young folks if you love programming then become a plumber and if you love computer science then get your PHd.

[guest]

Very funny and pretty close to the truth from my experience.

I love the fact that I went to Devry and can run rings around the PHDs.

Do what you love and the money will follow.

I have worked for many years in large companies that, in the 80s and 90s, would employ generalists to write their own bespoke systems. In the 80s it was IBM COBOL that ruled, then in the 90s VB grabbed the crown. However the biggest change has, in my opinion, been the incredible number of IT jobs that have gone eastwards. This is globalisation in action and it will only get worse for IT people in the West. The figures are quite shocking. Big corporations will say it is the cost effective thing to do but from an employment point of view our governments (UK and USA are the worst) have allowed corporations to simply ship jobs eastward. There are no employment laws governing this, we are wide open.

Sorry to depress you all - I told my kids not to get involved in IT!