Have you ever tried hiring an engineer who is great at low level software and has a great hand at general algorithms, data structures and application development? If you have, then you know how hard they are to come by. The vastness of each of those fields generally preclude in-depth education or self-learning with such a breadth. But a few of us understand that it is actually quite critical to building good software for a complex system.
I started thinking about this as a systemic problem in software development today when the new robotics organization I’ve been scooted into does not believe in hiring engineers who span breadth of knowledge in algorithms or using good data structures. Instead the focus is on hiring embedded engineers who know how to twiddle bits, read data sheets and set those correct bits in the right registers. As one manager proudly put it, I have not traversed a graph in my entire career of software building.
In the same conversation a principal engineer poured out how much he hated hiring people with broad algorithmic or design knowledge. “The candidate has to solve some problem, I don’t care what the problem is.” This shocked me. It also got me thinking how much I loved my old team where all engineers were grilled on algorithms, constrained resource optimization, data structs, efficiency and general problem solving abilities. We had an excellent band of engineers who I trusted to think broadly, design well and write good code. With my new team where engineers do not look beyond their tiny module, part of a larger SoC I find it hard to put my full trust in them. This is not to say they aren’t smart; they sure are. But their thinking is so narrow, I can almost see where they’ll stumble or what aspect of design will be ignored.
How useful is just knowing how to manipulate bits and set registers? OK, this is useful - but do you really need an engineer to do this once the silicon vendor publishes a well written example or a well written technical reference manual (TRM)? Wink, wink generative machine learning, here’s an industry that could easily be wiped out by you.
A large majority of software development today happens in very high level software. Applications written for interpretation in a virtual machine to be run on commodity hardware where one does not have to care about most resource constraints. The same goes for engineers, the “fungible software engineers” who should be mythically replaceable. There is very low dependence on traditional compilers, optimizations and struggle for resources. More freely available CPU and memory resources have eliminated the need for antiquated optimizations. But now in 2023 there is a major shift towards edge processing and it is becoming increasingly popular to autonomous devices. This is bringing back the need for mindful, constrained software development. The industry however is not ready for it and we do not have enough many generalists who can build software for the increasingly complex systems.
So, I asked Alex, “what are higher level software developers missing that makes it challenging to work in a low level environment?” Here’s a brief list we came up with that distinguishes embedded engineers. Not all embedded engineer are proficient with this list, but, most would have encountered them at some point.
Most engineering students who don’t specifically study Electrical or Computer engineering will have no formal knowledge of these topics. These are by no means hard to learn. There are excellent free resources, especially with blogs such as Embedded artistry or Memfault Interrupt. They’ll take a few weeks to learn and experiment with. But, most high level software developer have little or no practical experience with most of these topics. That puts them at a disadvantage.
What about the other side of this coin? What do high level developers bring to the table that pure embedded engineers struggle with? Here’s a list.
Of these I find fluency with data structs and applying algorithmic approaches key to any software developers success. As you grow and handle large projects object oriented design and system design very important to avoid creating blunders. Embedded engineers are also averse to setting up and using unit testing effectively. I have found this pervasive even in large scale systems almost always to the detriment of the products built. This is not hard, it is just a change in mindset low level developers are unwilling to make.
Continous integration and development is accepted in the web development community as a necessity. There is always ambiguity when it comes to analyzing the need for it in the low level world. This ambiguity is enough to deter engineers and have them push it off to a later day problem. This decision however has deep implications of how software build systems are setup and quickly become intractable for automated testing. With a few 10 engineers working on a project the quality goes to hell very quickly.
I have found engineers fluent with a mix of both lists operate at their best when working a complex product (the scale of an autonomous robot). Specialists have their place to dive deep and resolve gnarly problems. But their scope often limits them from understanding the system as a whole. Deeply disconnected product problems are inevitable where walled specialists operate to build different parts of a complex system.
So, when you encounter your next generalist who can work their way around both these lists, hire them or at least make friends with them. You’ll need them one day.