Software Developer, Python/Django
336 stories
·
1 follower

Pyramid vs. obelisk

1 Share

I may not be able to save the world, but I think I’ve come up with the fundamental idea that will be necessary in order to restore (and, afterward, preserve) the integrity of, at the very least, the software industry.

Software may or may not be “eating the world”, but we’re learning that the class of people who can rigorously solve problems using precise thought is one that ought to be held in high demand. Such people need not necessarily be writing software programs per se, but we’re in a world that cannot grow based on ego and bluster (the traditional traits used to select business leaders) alone. That approach has been played out, and we need competence to get any further. There are a variety of literacies that a leader needs in order to do her job well, and that set of needed capacities is too large for me to tackle right now, so let’s zero in on one important job, which is software engineering. As a society, we need it. We need it, done well. Unfortunately, as a society, we seem to be getting worse at it.

The problem is that most of the people who employ programmers have seen fit, over the past two decades, to diminish this kind of work as much as possible. There have been so many efforts to replace professional programmers with interchangeable, unqualified peons that it is, frankly, sickening. This is, of course, the real purpose of the “Agile Scrum” fad that has infested our industry, as well as the fetishism for open-plan offices. Dan Lyons’s new book, Disrupted, describes the ageist culture that emerges when we allow the industry to drive software, rather than the other way around. We’ve seen competent engineers driven out in favor of cheap, young, and (most importantly) obedient commodity-grade programmers hired to inflate headcount numbers and appease investors. It needs to stop, and now, before this industry gets so thoroughly infested with incompetence and anti-intellectualism that a generation’s worth of work and experience will need to be discarded. So how do we stop it?

Although it shouldn’t be this way, software engineering is pyramidal in shape. For every engineer who gets to work on a real project, there seem to be fifteen grunts only trusted to churn through Jira tickets. We need to reform our professional structure to be more like an obelisk, instead.

Architecturally, a pyramid is the most basic, inefficient way of achieving prominence (or, less pedantically, height). It’s imposing, it’s gigantic, and it historically required massive amounts of slave labor as well as natural resources. Ancient Egyptians built pyramids because they didn’t have the architectural know-how (namely, arches) to build much else at scale. In economics, one invokes the pyramidal shape when describing arrangements that deliver massive yield to the few (at the top of the pyramid) while exploiting the larger lower classes (at the bottom). Hence, the term “pyramid scheme”. One could argue that pyramids (of both kinds) are entropic. A pile of sand or stones will assume an approximately conical shape, and it’s likely that this inspired the (architecturally unsophisticated) pyramids of Ancient Egypt. Likewise, an ill-organized or disorganized set of people will devolve into an oligarchic arrangement, with many supporting the few.

Professional pyramids mean that positional scarcity will exist at every level in the profession (the term “profession”, here, used loosely) as the number of positions declines as an exponential function of experience. It’ll be as hard to stay in the profession as to get in. Except for sociopaths who enjoy social competition for its own sake, most people find this undesirable. Doctors and attorneys wouldn’t be happy if they had to continue fighting for fewer spots with each year of age, even 30 years after graduating. Not only would it be bad for individuals to have that degree of positional competition, but it would also be a waste for society. To push out 50-year-old doctors simply because they “failed” to arrogate managerial positions would be to discard hard-won expertise, and it would generate a culture of inexperience.

Left to their own devices, management hierarchies become pyramidal. If a company of 10,000 people decides on a 12:1 reporting ratio (as is typical, in software) then it’s going to have 9,167 grunts, 764 first-level managers, 64 directors (managers of managers), and only 5 people at the executive level as traditionally defined. Why can’t there be 100 directors instead of 64? There could be, but few companies would want to pay for them if given the choice. That pyramid seems painful, and it would be, if that’s how companies worked. At the top, they often don’t, and I’ve written before about the superior collective intelligence of MBAs as opposed to software programmers. People with elite MBAs aren’t stupid. They realize that they’re going to get screwed under the traditional pyramidal regime. Not only does the pyramid system make it unlikely for people to advance, but they also leave the people in high positions unprotected, because there are too few of them. CEOs promote loyalists not because the companies need so many executives, but because the CEOs are smart enough to realize that sharing rank and prominence is essential for their own survival.

One could argue that business executives, as a group, “conspire” against their companies by working together in order to convince their organizations that they’re more needed than they really are, and that more of them are needed than really are, and that they themselves as a class (executives) can recognize talent in a way that no one else (not shareholders, not workers) can. By doing this, they’ve managed to create massive demand for themselves and their skill set (which is perceived to be more rare than it actually is). On this topic, I’m hesitant to use the word “conspire” because I don’t consider it morally wrong. Corporate life has become a culture of resource extraction, and the executives have simply been the best players at that game. Ultimately, there are very few people who would hesitate, if offered a magical device that would imbue in their employers an extreme and superstitious overvaluation of their work, to use it for personal benefit. Executives have simply managed to do exactly that as a group, rather than as individuals, the latter being far more difficult.

Corporate executives, based on manufactured demand, as well as doctors and attorneys based on actual demand, have taken their leverage and replaced the pyramidal structure with a professional obelisk shape. An obelisk is upright, rising far higher than a pyramid of comparable mass could, until the pinnacle. With a professional obelisk, there’s plenty of room to grow until one encounters positional scarcity. Doctors don’t need to beat down other doctors and climb management hierarchies just to advance; they just need to gain more skills and deliver more value. Likewise, MBA-toting corporate executives might face stiff competition if they want to be Fortune-500 CEOs (because there are only 500 such spots) but they deal with minimal competition if they just want 7-figure incomes and fancy titles by age 50.

An obelisk structure shows that a value is placed on expertise: as long as people continue to improve themselves, there’ll be room for them at a level higher than where they are. Progress will be recognized. Why does the medical profession intentionally create an obelisk-shaped, guild structure? It doesn’t want “Agile” 17-year-olds performing surgery and driving out competent surgeons who didn’t pursue management jobs. It also doesn’t want people taking obscene risks due to an up-or-out culture that persists into one’s 40s and 50s. Finally, it doesn’t want a culture where the median practitioner has less than a decade of experience.

The pyramid is the entropic formation. Without a deliberate professional structure, it will emerge. Why so? An unprotected “profession” (note the oxymoron) will expand until it absorbs the most unqualified people, who draw barely more compensation than they’d get anywhere else, but who are able enough to convince people (unqualified buyers) to buy their services that they can stay afloat. The unqualified and minimally invested (the “quacks”) will, by sheer numbers, outvote the genuine practitioners (who’ve often dedicated their lives to becoming good at something) and dominate the culture– until their incompetence starts hurting people. In medicine and law, this outcome is considered so unacceptable that it’s not controversial for these industries to have professional structures that function as labor cartels. We need to start thinking the same way about software. We need to start driving out the charlatan brogrammers for whom the infantilizing open-plan startup culture was created.

So how do we get an obelisk-shaped, protected profession for software engineers?

Let’s focus on something that hasn’t worked, and look at why it has failed. Many companies have created “engineering ladders” that purport to measure professional development for individual contributors and allow people to grow without becoming managers. There might be a manager-equivalent title (Staff Engineer) and a Director-equivalent one (Principal Engineer, or Architect) and a VP-equivalent one (Fellow). What goes wrong? To start, the disparity in difficulty makes the whole thing extremely self-congratulatory on the part of the managers.

To become a Director- or VP-equivalent engineer, you have to compete against some of the smartest people in the company. It’s legitimately difficult. There are smart, qualified people who don’t make it. If anything, these dual-track systems serve the interests of management by allowing the executives to overstate their value and capability relative to engineers; if one has to be a genuinely great engineer to become Director-equivalent, the thinking is, then engineers will hold Directors and VPs on the much-easier-to-climb managerial tracks in higher regard.

Ultimately, these systems don’t work. Rather, they serve to document and justify an existing pyramidal scarcity. The organization will only allow a small number of engineers to climb into positions with the credibility and compensation afforded to senior managers, because engineers lack the collective organizational skill to thwart the penny-pinching game being played against them. Thus, a pyramid forms, and little is solved, because positional scarcity persists at each level.

I’ve been around for long enough to know that trusting employers is not a strategy. In the long term, it always fails. Don’t get me wrong: there are good companies out there, and there are managerial strategies (e.g. open allocation) that can improve the odds of creating one. Some individual employers probably can be trusted, but this doesn’t scale, because most organizations become pathological and self-serving at some point. So, if we can’t trust employers, then we can’t put stock in employer-assigned titles as a way to insure the credibility of the individual programmer. That means that we need an employer-independent system for doing so. We need a source of credibility for the individual that no employer can touch.

Doctors and attorneys have employer-independent credibility, being vouched-for by their professional associations (in the U.S., the AMA and the ABA). That’s necessary because the core of a profession is that one’s ethical obligations supersede managerial authority. That is, a doctor can’t use the superior orders defense to justify killing a patient. Of course, that only works if the professional has some sort of employer-independent credibility– in which case, losing a job is a transient financial annoyance rather than a possible career killer. If the individual lives entirely on his reputation as made by prior managers, his lack of leverage becomes a lack of moral agency. That is, of course, the problem that most programmers face. We lack moral or even creative autonomy, because we lack employer-independent credibility; once we leave school, our value is derived from the titles and recognitions given to us by profit-seeking corporations.

Some programmers hoped that the open-source world would fill this gap. It can’t. That’s not its job. Evaluating a programmer’s Github profile is a subjective task, and the only objective way of measuring one’s open-source contributions is based on how many people use the software. Here’s the issue, though. Few engineers have the time and resources available not only to build usable open-source projects, but to market their work to the global programming community enough that it actually used, and then to support it once it is. The positional scarcity and power-law distribution of open-source prominence emerge not because the open-source community is full of mean people (it’s not) but because (a) there’s no barrier to entry when it comes to just putting code out there, while (b) most of this code will never be used, because people have an entirely understandable (and, in fact, prudent) aversion to using unproven software.

How do we create an employer-independent credibility for software engineers? As far as I can tell, there are two solutions. The first is one that I hate: to require formal, institutional education. I hate it because it’s hostile to career-switchers, astronomically expensive, and can generate a culture of anti-intellectual credentialism. This leaves the second, which is to create an exam-based system, similar to what the actuarial sciences use. As programmers pass more exams, their professional credibility can grow in an employer-independent way. This would not only give us job security, but the improvement in our general leverage would enable us to negotiate for better working conditions and launch better projects, to the benefit of society as a whole.

Are there problems with exams? Of course there are. For one thing, I happen to believe that “bad test taker” is a real thing. (I was always a good test taker, but I’ve met intelligent people who aren’t.) We’d have to adapt our system to legitimate learning disabilities and to allow project-based alternatives for great programmers who struggle with traditional exams. We’d need to decide what to charge for the exams (as well as the presumably more expensive project evaluation) and how to offer financial aid. We’d have to decide what would be on the exams, and how many there would be. (I, personally, doubt that I’d be qualified to design such a curriculum.) What is an Associate-level programmer, and what’s a Fellow-level engineer, and what do we expect each to know? What are the continuing education requirements going to look like? These aren’t easy problems to solve. On the contrary, they’re quite difficult and organizationally complex.

Here’s the thing, about that. Designing a curriculum and exam track for software engineers is a massive amount of work, much of it ugly. I doubt I’m up to it, personally, because I’d rather have it built by experts in their respective fields of computer science. However, the core idea is here is one that will actually work. Blindly trusting employers hasn’t worked, and fetishizing startups has been a hilarious failure. Over the past twenty years, the programming profession has lost ground. We’ve been infantilized with open-plan offices and “Agile Scrum” practices that senior engineers don’t need. We’ve been left without any form of employer-independent credibility, which is what we’ll need if we want to become a true profession. We’re in a state of squalid failure. Our professional shape is a pyramid rather than an obelisk, making the job painfully political at every level. I’d rather see an imperfect solution (such as a professional curriculum and an employer-independent, exam-based advancement system) that fixes this than no solution at all. This problem won’t be solved by one person and it certainly won’t be solved within a year, but I’m confident that I know, at least, where to start.


Read the whole story
omouse
579 days ago
reply
Toronto, Canada
Share this story
Delete

Our software must get better

1 Share

“That’s good enough, let’s move on”

Lots of things could be better (cars, buildings, candy, etc.) but we understand that the cost of pushing through to the next level is prohibitive. It might be because, as in the case of candy, the mass market just won’t pay for premium ingredients, or, in the case of buildings, the cost of retrofitting the billions of buildings in the world is just too big to fix the stuff that’s already out there.

The building doesn’t fall down, it sort of works, better than good enough, let’s move on.

But software, software is different. Consider:

1. one piece of software can be used by a billion people, no extra cost per person. Unlike candy or anything physical, it doesn’t cost more per user (not a penny more) to have more people use great software instead of settling for good software.

And 

2. fixing software today fixes it for everyone, in the world, going forward (and for connected computers, going backward as well).

Imagine what would happen if this were true for buildings… if the efficiency and style and ambience of every building in the world could be fixed, all at once, in exchange for one investment.

Alas, software tends to be mediocre. There are a few reasons for this:

A. Lock in means that once someone has a success (and the cash flow that comes with it) there’s not much incentive to invest a lot in fixing it (fixing it looks a lot like breaking it, at least at first). Which is why Paypal has had such a miserable user interface for so long. (Do the folks at Paypal know how bad it is? Don't they care?)

B. As software gets more successful, the instinct is to hire more people to work on it (which increases complications and errors dramatically) and to be ever more conservative as well (don’t mess with what’s working).

C. Perhaps the biggest problem: In many markets, especially online, software is free. And free software built by corporations turns us from the user into the product. If you're not paying for it, after all, you must be the bait for the person who is. Which means companies spend time figuring out how to extract value once we're locked in and can't easily switch.

I’ve been developing software on and off since 1984, and empathize with the people who have to make these decisions. But software is too important to be mediocre. 

Compare Roon to iTunes (which has had countless iterations, but never seems to get better, it merely helps Apple sell more of something).

The Roon user experience is fabulous. The only reason they could launch it in the face of a free competitor is that enough people care about music. Which means that software as a service in this area has a shot for a revenue stream that can justify the investment, but even with a demonstrably better product, competing with free, with software installed by default, this is really hard work.

Or compare the heavily promoted (but awful) stamps.com to the elegant but little known alternative, Endicia. It works on the Mac, does tracking, it actually works. Better software, worth it.

Or consider the Address Book built into your Mac, a piece of software that only is used because it's free and hardwired in. It's difficult to import or export data, and it's truly slow. No one, not one person, is happy about how this software helps them work. Without a reasonable business model, though, competing with free is incredibly difficult.

When you can, insist on paying for your software. Our instinct to take the free stuff is often a bad long-term choice—it takes a committed team to keep free software worth the trust we put into it.

Marketing and the economics of an industry don't always lead to the best solution. Sometimes, we need to insist on things getting better.

       
Read the whole story
omouse
579 days ago
reply
Toronto, Canada
Share this story
Delete

The Fallacies Of Distributed Coding

1 Share
If you only ever write code which runs on one machine, and only ever use apps which have no networked features, then computers are deterministic things. It used to be a given, for all programmers, that computers were fundamentally deterministic, and thanks to the internet, that just isn't true any more. But it's not just the rise of the internet, which its implicit mandate that all software must become networked software, which has killed the idea that programming is inherently deterministic. Because everybody's code became a distributed system in a second way.

If you write Ruby, your code is only secure if RubyGems.org is secure. If you write Node.js, your code is only secure if npmjs.com is secure. And for the vast majority of new projects today, your code is only secure if git and GitHub are secure.

Today "your" code is a web of libraries and frameworks. All of them change on their own schedules. They have different authors, different philosophies, different background assumptions. And all the fallacies of distributed computing prove equally false when you're building applications out of extremely modular components.
  1. The network is reliable. This is obviously a fallacy with actual networks of computers, but "social coding," as GitHub calls it, requires a social network, with people co-operating with each other and getting stuff done. This network mostly exists, but is prone to random outages.
  2. Latency is zero. The analogy here is with the latency between the time you submit a patch and the moment it gets accepted or rejected. If you've ever worked against a custom, in-house fork of a BDD library whose name.should(remain :unmentioned), because version 1.11 had a bug, which version 1.12 fixed, but version 1.12 simultaneously introduced a new bug, and your patches to fix that new bug were on hold until version 1.13, then you've seen this latency in action, and paid the price.
  3. Bandwidth is infinite.
  4. The network is secure. Say you're a law enforcement agency with a paradoxical but consistent history of criminality and espionage against your own citizens. Say you try to get a backdoor installed on a popular open source package through legal means. Say you fail. What's to stop you from obtaining leverage over a well-respected open source programmer by discovering their extramarital affairs? I've already given you simpler examples of the network being insecure, a few paragraphs above. I'm hoping this more speculative one is purely hypothetical, but you never know.
  5. Topology doesn't change.
  6. There is one administrator.
  7. Transport cost is zero. Receiving new code updates, and integrating them, requires developer time.
  8. The network is homogeneous.
Open source has scaled in ways which its advocates did not foresee. I was a minor open source fan in the late 1990s, when the term first took hold. I used Apache and CPAN. I even tried to publish some Perl code, but I was a newbie, unsure of my own code, and the barriers to entry were much higher at the time. Publishing open source in the late 1990s was a sign of an expert. Today, all you have to do is click a button.

The effect of this was to transform what it meant to write code. It used to be about structuring logic. Today it's about building an abstract distributed system of loosely affiliated libraries, frameworks, and/or modules in order to create a concrete distributed system out of computers sending messages to each other. The concrete distributed system is the easy part, and people get it wrong all the time. The abstract distributed system is an unforeseen consequence of the incredible proliferation of open source, combined with the fact that scaling is fundamentally transformative.
Read the whole story
omouse
584 days ago
reply
Toronto, Canada
Share this story
Delete

FSF: A preliminary analysis of High Priority Projects feedback

1 Share

The Free Software Foundation (FSF) has posted an initial analysis of the public feedback submitted in response to its call for suggestions about what software projects deserve to be on the high-priority projects list. Several of the existing projects on the list are likely to be removed, such as a replacement for Google Earth and an automatic-transcription application. Multiple potential additions to the list are also described, such as a free-software "Siri"-like personal assistant and a free-software implementation of advanced PDF features. Further input is welcome; the page notes that the FSF "will strive to recommend projects that are actionable. Such projects document ways for members of the free software community to get involved and make the project succeed, with any kind of concrete contributions, from money donation, to code patches, advocacy, etc."

Read the whole story
omouse
598 days ago
reply
Toronto, Canada
Share this story
Delete

CitusDB open-sourced

1 Comment
Citus Data has announced that its CitusDB distributed database has been released, under an open-source license (AGPLv3), as a PostgreSQL extension. "First, Citus 5.0 now fully uses the PostgreSQL extension APIs. In other words, Citus becomes the first distributed database in the world that doesn't fork the underlying database. This means Citus users can immediately benefit from new features in PostgreSQL, such as semi-structured data types (json, jsonb), UPSERT, or when 9.6 arrives no more full table vacuums. Also, users can keep working with their existing Postgres drivers and tools."
Read the whole story
omouse
598 days ago
reply
Looks neat, scaling postgres always felt like a heavy task
Toronto, Canada
Share this story
Delete

MIT Media Lab defaults to free & open source software (NetworkWorld)

1 Share
NetworkWorld reports that software developed at MIT Media Lab will be open source by default. "This effort does away with developers having to get such licenses approved first by an internal committee, which [Lab Director Joi Ito] says "always allowed our developers to open-source their work" anyway."
Read the whole story
omouse
598 days ago
reply
Toronto, Canada
Share this story
Delete
Next Page of Stories