Mark Youngman's Website

Who to trust when learning programming?

21 May 2020

My experiences online suggest two people from separate programming subcultures, each with multiple decades of experience, who articulately defend themselves, can disagree on fundamental issues. Both are convinced that their way is unquestionably correct. Their group's ideas are generally viewed as sacred, and yet they often contradict those of others groups.

Why do they disagree? And how do we decide who is right? I don't think there are any clear answers, but it's interesting to think about.

When experts disagree

I think experienced programmers disagree because programming is (a) still in its infancy and (b) hard.

A programmer can only work on a handful of large projects in their lifetime, and it can take years before they receive feedback from their choices. This means that even a veteran programmer has little evidence to determine what's best -- and ambiguious evidence at that. Further more, since programmers tend to follow a specific culture, it's likely that the projects they've worked on all followed a similar methodology, and so they have limited experience of other approaches.

I think in most instances, programmers don't know much beyond the culture they were raised in. An embedded systems programmer is unlikely to understand the problems of a web programmer and vice versa. One is working in a jungle, and the other is working in the desert. One gets incredulous that the other doesn't wear suncream.

How do we choose who to trust?

A better question might be "Who shouldn't we trust?" Teaching is a big industry: books, conferences, bootcamps, university degrees. Anyone can give a talk or write a book. "Those who can, do; those who can't, teach". And the teachers themselves disagree with each other.

If you dig into the CVs of well-regarded programming teachers, they often don't provide strong evidence for why you should trust them.

I have more respect for those that survey across projects, but that's still weak data and anecdotes. I think there are too many variables between different teams and projects to isolate what works. There are no double-blind studies here, and I think if we're going to reach sound conclusions, that's the rigour required.

What qualities should a trusted authority have? Maybe they've programmed much more than others, completed multiple large, complex projects to a high standard? How high is writing lots of books and giving lots of talks on that list?

If we trust people with a long, proven track record, that is no guarantee in itself. Even if correct, they will tell you what worked for them when solving their problems, when working within their environment.

The fundamentals of programming

An approach that has served me well when learning a new subject is to start at the beginning. When the foundations are dodgy, the elaborate the structure on top is irrelevant.

Currently, my personal conclusion, rightly or wrongly, is that programming is primarily about solving data transformation problems. I find the more I focus on the data, the clearer things are. Code is easier to read and maintain if the transformation performed naturally arises out of the problem being solved, when self-expression is minimised.

Programming is an engineering discipline, not an art. Any solution should say more about the problem than it does about the person who solved it.

The power of "I don't know"

I like people who are ready to admit that they don't know. This subject is so large that nobody can know it all, and so varied that there are few absolutes. Like with anything else, an ego prevents learning, causes overconfidence, and leads to ostentation.

With so many experienced programmers disagreeing, I think the only sensible approach is to be slow to judge, to avoid dogma, to expand your horizons, and to always doubt your own approach.




Code readability and data flow

25 April 2020

While working on a test framework a while back, a particular piece of functionality inspired some thought about the relationship between the flow of data and code readability. I found that iterating towards less code meant the code, as written, no longer represented the flow of data. It was therefore left to the reader to work out the flow themselves.

The functionality in question

You can see code at the time of writing here.

The test framework allows users to create test suites by subclassing the TestSuite class. They can then add tests to that subclass by creating a new method and prefixing test__ to its name.

To implement this in Python, we need to:

It turns out this can be achieved concisely:

for test_name in filter(lambda name: name.startswith('test__'), dir(self)):
  self.tests_to_run.append(getattr(self, test_name))

After starting with something much longer, it's satisfying code to iterate to, but is it good?

Clever code is dumb code?

If the reader has knowledge of Python -- understands filter, dir, etc. -- they may not need a search engine. But even so, they still have unpacking to do.

If we put the code aside and consider how the data is processed, it starts at the dir(self), which returns a list of method names for the current class. It's the first thing we need to progress towards what we want: a list of test methods to run. But this isn't the first thing a reader would see. Instead, dir(self) is at the end of the first line, nested inside a function call.

This means that to understand this code, the reader is forced to read the code multiple times, jump back and forwards, to identify the flow of data for themselves.

Target audience

Even for an experienced Python developer it would require some mental overhead, and that all adds to the overhead of the code's larger context. People can only hold so many things in their head at once.

This reaches to a larger topic: who is the target audience and what does the coder wish to communicate to them. I believe the same issue is found in writing.

Consider The School by Donald Barthelme. On the surface the writing seems simple, but if you try to emulate his style it become apparent that behind that simplicity is great skill.

Compare Barthelme with Irene Iddesleigh by Amanda McKittrick Ros. Ros is out to impress, to the comical detriment of the writing itself.

Writing often wants to be stylish, to entertain, but I think code should to be workman's prose. It shouldn't draw attention to itself.

The narrative of the data

For me, the problem with the above code is that it prioritises conciseness over accurately representing the flow of data. The reader wants to understand the narrative of the data -- its journey from start to end.

Here is the code I ended up with:

method_name_list = dir(self)
is_test = lambda name: name.startswith('test__')
test_name_list = filter(is_test, method_name_list)
for name in test_name_list:
  method = getattr(self, name)
  self.tests_to_run.append(method)

The transformation of data can be followed in order by reading from top to bottom -- it matches the steps I gave earlier. And with the additional variable names, it's possible for a reader unfamiliar with the in-built functions called to infer what the code does.

I like the idea that code should be accessible to the inexperienced. I'm going to try and stick to this when coding in the future.




Better pull requests with interactive rebasing

18th April 2020

I am like most developers in that, rightly or wrongly, the most I pause before creating a pull request is to create a cleanup commit. PRs are often littered with unnecessary commits, and the commits themselves don't intentionally demonstrate the thought process behind the changes made.

There is much scope to improve. In a time when documentation is frequently trumpeted, I'm inclined to think logically grouping, ordering, and labelling the commits made in a PR can probably help more. And as it could be part of a workflow, it may be easier to stick to than an additional task that is easily forgotten like documentation.

Clear commit messages

Commit messages are the obvious starting point to improve a PR, but you can only have a good commit message if the commit changes themselves are explicative -- even with no message.

That said, it is the lowest hanging fruit -- particularly if, like me, you have created many-a-commit like

git commit -m"TKT-999 Fixes the thing"

There are already good guides for writing commit messages -- and interactive rebasing can help reword commit message before creating a PR.

Beyond the commit message

There are others ways to improve PRs that aren't often discussed:

Interactive rebase can be used to manipulate commits on your branch before creating a PR. A commit that fixes a typo, for example, can be squashed. Consider this example:

$ git log
commit f001584020e873aa89a3063c29c7f29fcf87317a (HEAD -> master)
Author: Mark Youngman
Date:   Sat Apr 18 20:04:57 2020 +0100

    Add even more text to test.txt

commit 5ad7fb8c8a632cf36863534104fb1bfb1d27fc59
Author: Mark Youngman
Date:   Sat Apr 18 20:04:23 2020 +0100

    Fix typo in test.txt

commit 42c9d470c400f4502031dfea23977693b9973023
Author: Mark Youngman
Date:   Sat Apr 18 20:01:42 2020 +0100

    Added more text to test.txt

commit b3e17ee7b2f605482e67f812ff6fea313a74fd0f
Author: Mark Youngman
Date:   Sat Apr 18 19:40:04 2020 +0100

    Added test.txt

This can be fixed as follows:

git rebase -i HEAD~3

On starting the interactive rebase, instructions are provided:

 6 # Commands:
 7 # p, pick <commit> = use commit
 8 # r, reword <commit> = use commit, but edit the commit message
 9 # e, edit <commit> = use commit, but stop for amending
10 # s, squash <commit> = use commit, but meld into previous commit
11 # f, fixup <commit> = like "squash", but discard this commit's log message
12 # x, exec <command> = run command (the rest of the line) using shell
13 # b, break = stop here (continue rebase later with 'git rebase --continue')
14 # d, drop <commit> = remove commit
15 # l, label <label> = label current HEAD with a name
16 # t, reset <label> = reset HEAD to a label
17 # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
18 # .       create a merge commit using the original merge commit's
19 # .       message (or the oneline, if no original merge commit was
20 # .       specified). Use -c <commit> to reword the commit message.
21 #
22 # These lines can be re-ordered; they are executed from top to bottom.
23 #
24 # If you remove a line here THAT COMMIT WILL BE LOST.
25 #
26 # However, if you remove everything, the rebase will be aborted.

In this instance, you change pick on the relevant commit to f for fixup:

1 pick 42c9d47 Added more text to test.txt
2 f 5ad7fb8 Fix typo in test.txt   
3 pick f001584 Add even more text to test.txt

In a different situation, evidence of the typo may still exist in your PR. To erase all evidence, you can use rebase -i to amend the appropriate commit:

git rebase -i master
[replace 'pick' on the appropriate commit with 'e']
[fix the typo]
git add [file]
git commit --amend
git rebase --continue

Commits as steps towards a complete feature

After squashing unnecessary commits, a PR can be further improved by having each commit be a logical step towards the completed feature. This means the PR can be read commit by commit, showing why the feature was developed as it was.

This ordered grouping of changes often isn't the case with PRs. The steps towards the completed feature are spread amongst different commits, so rather than

Commit 1: Finish step A
Commit 2: Finish step B
Commit 3: Finish step C
Commit 4: Finish step D

you have

Commit 1: A little bit of A and C
Commit 2: Finish A and start B
Commit 3: Finish B and C and D

Ideally, making your PRs more like the former than the latter is done by having the discipline to identify the steps before starting work on a feature. Back in the real world, some cleanup will be required before creating your PR.

With rebase -i, this cleanup can largely be done by rewording, squashing, and editing commits. However, in some instances you may want to split a commit.

Splitting a commit

Changes across two commits are probably simpler to follow than both combined. For that reason, alongside padding your commit stats, splitting a commit can make sense.

A commit can be split by following these steps:

git log
[make a note of the hash of the commit you wish to split]
git rebase -i master
[replace 'pick' to 'e' on the commit immediately _before_ the commit you wish to split]
git checkout [commit_hash] -- . # Assuming you're in your project's root dir
git restore --staged .
[restage the files with the changes you want to appear in the 1st of the two commits]
git commit -m"Your commit message" # Note we create a new commit -- not amend!
git rebase --continue

If you're making such changes before creating a PR, it's probably best to clone your branch before making them, so you can check the two branches for unintentional differences.

Worth the effort?

I'm only starting to think about this, and it is uncertain to me whether time spent is worth it. But I can imagine several benefits:




Standard Format for Blog Posts - This Website Part Two

28th October 2018

After being dissatisfied with using markdown to write these posts, and despite knowing nothing about creating a standardised format, I decided to design my own format for blog posts, and write something to convert those files into HTML. My ineptitude gave me confidence that, by tradition, my new format would soon be standard across the web.

My new post file is inspired by LaTeX for no more reason than I wanted a simple yet extensible markup language that wasn't markdown or XML, and no other syntax came to mind.

Despite my static website relying on Python, I thought I'd write the post-to-html program in C, ostensibly because Python strings are immutable, but really because I like the idea of writing a compiler in C and this is a step in that direction. It was also an opportunity to learn how to use C with Python, which was, for me, one of the attractions of Python. I also had already written some C string manipulation functions when working through Advent of Code.

To write it in C, I needed it to play nice with my generate_website.py script. There were a few options here:

I didn't want to compile the entire Python runtime each time I pushed a change, so I started by looking at an extension.

Creating a Python C extension

Turned out that creating a Python C extension is as easy as copying and pasting this example, so I was up and running in two shakes. I then tested to see if it would work with my GitLab pipeline, which it did.

Things were looking good, but I knew debugging was going to be essential, and after looking at the hoops I'd have to jump through, I decided to try ctypes instead.

Using ctypes

The ctypes module allows you to call a C dynamic library -- something I'd previously messed around with -- in Python. It also meant I could debug with QtCreator, as I usually do with C.

I quickly printed "hello world" with ctypes, and then managed to work around unicode to pass and return ASCII strings from the compiled C code.

I previously had an idea to write C library to handle unicode strings with a kind of built-in memory management, but that had to be postponed in the name of progress. It was time to write some C.

Writing post-to-html.c

I had no idea how to proceed, so I decided to bang out some code to produce the About page as a start, and resigned myself to the fact that the code, even if it works, would be a mess. As usual, I tended to avoid adding dependencies, as this is ultimately a learning exercise.

To say the result is a mess is an understatement, but the emphasis was on getting something working quickly, and the About page does now generate from a .post file. I expect the clean up will be a complete rewrite, if not several rewrites, not just of the C code, but of everything about how this website is currently generated.

Before I start cleaning the code up, I think first I'll add functionality for paragraphs, sections, lists, blockquotes, images, and tables, so I can change all the existing posts from .md to .post and remove the dependency on the Python markdown library. Then it'll be time to settle down and read about parsing, check out a few examples, and hopefully write some code that isn't bat-shit insane.




Shuffling into Serial -- Behind the Libraries Part Five

21st October 2018

We're slowly understanding various parts of the Arduino Uno and, more specifically, the ATmega328p chip. Next on the agenda are the various Serial commands we used, such as Serial.begin and Serial.print.

Before we learn how to do serial communication without using the Arduino libraries, firstly we need to learn how functions like Serial.println work behind the scenes.

What is USART/Serial?

USART is hardware that allows two or more devices to communicate. In this instance, we're interested in one particular mode of the USART called asynchronous mode. This mode uses two wires to allow communication between two devices. One wire is used to receive data and the other to send data. Pins 0 and 1 on the Arduino Uno, which receive (RX) and transmit (TX) respectively, are used for this type of communication. Also for convenience the Arduino has a Serial-to-USB chip, separate from the ATmega328p, which converts serial data communicated over pin 0 and 1 into something that can be transmitted over USB, which is what allows the Arduino to connect to a computer and use the Arduino IDE.

How can two devices communicate?

Just as two people speaking must use the same language to communicate, two devices, to communicate data over one wire (or more), need to use the same protocol.

The protocol dictates how the communication occurs. On a single wire, we only have two options, on and off. So the communication depends on how we structure those periods of being on or off such that data can be communicated.

Let's look at an example. We might keep a wire HIGH permanently when communication isn't occuring. If we want to start transmitting data, we bring it LOW for one second to indicate to the receiver that communication is about to begin. Then we set the wire HIGH or LOW depending on what the particular bit we want to transmit is, and leave it at that for a second, and then repeat that until we've transmitted 8 bits after 8 seconds. Finally, after the data has been transmitted, we go HIGH to indicate the transmission has finished and either stay HIGH or go LOW again to transmit another byte of data.

communication example 1

With this method, we have transmitted 8 bits in a "frame".

NOTE: The least signficant bit is often sent first, so my example wouldn't have transmitted the number 0b01101001, but that number backwards, 0b10010110.

It is important that the sender sets the wire HIGH or LOW at a different time to when the receiver reads the wire. In the above example, at precisely two seconds the sender set the wire to HIGH. Ideally, the receiver would read the wire at 2.5 seconds. If the receiver reads data from the wire at the same time, it is possible it will grab the wrong value.

One way some protocols achieve this is by having a separate "clock" wire. A clock wire operates like the Arduino Uno's crystal oscillator, in that it toggles on and off at a set frequency.

communication example 2

In the above example, the sender writes (W) the next value to the wire each time the clock wire rises, and the receiver reads (R) each time the clock falls. If both sender and receiver can see the clock wire, then it makes it trivial to avoid writing and reading from the data wire at the same time.

No clock wire

USART uses no clock wire that both devices can see. This saves the need for a third wire, but adds the problem of synchronising the two devices. The way this synchronisation is achieved is through both devices having their own internal clock, and both being set to the same rate.

This is why the Serial.begin command is needed and why the Arduino IDE's Serial Monitor prints gibberish if you set the wrong rate. In my example above that didn't include a clock wire, the communication only occurs if both devices read/write every second. If the device receiving data instead read every 1.5 seconds, for example, then the sender's data wouldn't be recieved.

communication frame

This is taken from p223 of the ATmega328p datasheet, showing how USART sends bits. USART can send a different number of bits each frame, but in the above example, it sends nine bits (0 to 8). At the end of the frame, it can either idle or immediately send another frame.

You'll notice it works almost the same as my example of how eight bits could be sent. The one difference is the addition of the parity bit. This isn't something we'll be using in this article, but it is used to help check whether the data bits were sent correctly -- it tells the recepient whether the intended transmission is even or odd.

Example Code

Now that we've explored how Serial works behind the scenes, we can now learn how to use USART directly. Here is an example program that repeatedly sends the character 'A' using the USART:

void setup() {

  //Baudrate to 9600

  UCSR0A &= 0b11111101;

  UBRR0 = 103;

  // Use 8 data bits, UCSZ2:0 = 0b011

  UCSR0B &= 0b11111011; // UCSZ2 = 0

  UCSR0C |= 0b00000110; // UCSZ1:0 = 0b11

  // Enable transmission

  UCSR0B |= 0b00001000; // TXEN0 = 1; }

void loop() {

  // Wait until ready for new data, i.e. UDRE0 bit == 1

  while( (UCSR0A & 0b00100000) == 0);

  UDR0 = 65; }

Before we breakdown this program, let's cover what registers we are using.

UCSR0A/B/C are three USART Control and Status Registers. They allow us to configure the USART mode, whether transmitting and receiving are enabled, the number of data bits, whether we use a parity bit, and so on. They also give us information, such as whether it's ready for new data or whether it has transmitted the data it was previously given.

UBRR0 is a 16-bit value contained in two registers UBRR0L and UBRR0H, like we saw with TCNT1. The value it contains controls the baudrate.

UDR0 is used to both hold the data we want to send and read any data we've been sent. Behind the scenes, this label actually refers to two different 8-bit registers, but which we access depends on whether we read or write to that label. If we set UDR0 to a value, then we're setting the USART Transmit Data Buffer Register. If we read from UDR0, we're reading from the USART Receive Data Buffer Register.

NOTE: In the above program, there are some configuration options that we're not setting because they're already set as we want by default. For example, the USART is by default set to asynchronous mode. This article isn't intended to be an indepth guide to everything about USART. If you're interested in going deeper into what these options are, the datasheet has the answers.

UCSR0A &= 0b11111101

UBRR0 = 103;

This configures the baudrate to 9600. What to set this value to would be confusing, with us having to crunch numbers into a formula from the datasheet, but thankfully the datasheet also contains a large table of common clock rates and the value to use for particular baudrates on p241. As we discovered in a previous article, the Arduino Uno runs the 328p at 16Mhz, so all we need to do is look at the relevant section of the table and use those values.

UCSR0B &= 0b11111011; // UCSZ2 = 0

UCSR0C |= 0b00000110; // UCSZ1:0 = 0b11

This sets the three bits that determine the number of data bits. Like in previous cases, it's unclear why these three bits are spread across two registers -- presumably a hardware design reason.

while( (UCSR0A & 0b00100000) == 0);

Here we loop continuously until the UDRE0 bit is 1. This bit is automatically set to 1 when the USART is ready for the new data.

UDR0 = 65;

Serial Monitor uses ASCII, which basically translates the 8-bit value we send into its associated character. In this instance, 65 relates to the character 'A'. You can see which values relate to which characters by looking in the ASCII table.

Screw USART. Let's Bit Bang

Since this series is about understanding how things work on a hardware level, we can use everything we've learned in this article to transmit data without using USART and instead using digitalWrite and delayMicroseconds. We can do with software what USART did with hardware.

If we want to communicate at 9600 baud, that is 9600 bits per second or 1 / 9600 = 0.000104166666666667 seconds per bits or approximately 104 microseconds. We also know that pin 1 is what Serial uses to transmit data. Using this information, we can write our program:

void setup() {

  pinMode(1, OUTPUT);

  digitalWrite(1, HIGH); // Start HIGH to indicate no transmission occurring

}

void loop() {

  delayMicroseconds(104); // 104 microsecond delay for 9600 baud

  digitalWrite(1, LOW); // start bit

  // 'A' == 65 == 0b01000001

  // Least significant bit first

  delayMicroseconds(104);

  digitalWrite(1, HIGH); // 1

  delayMicroseconds(104);

  digitalWrite(1, LOW); // 0

  delayMicroseconds(104);

  digitalWrite(1, LOW); // 0

  delayMicroseconds(104);

  digitalWrite(1, LOW); // 0

  delayMicroseconds(104);

  digitalWrite(1, LOW); // 0

  delayMicroseconds(104);

  digitalWrite(1, LOW); // 0

  delayMicroseconds(104);

  digitalWrite(1, HIGH); // 1

  delayMicroseconds(104);

  digitalWrite(1, LOW); // 0

  delayMicroseconds(104);

  digitalWrite(1, HIGH); // stop bit

}

This should repeatedly print 'A' to the Serial Monitor. To prove this isn't a fluke, you can change the transmitted data to print other characters. For example, if you change the HIGHs and LOWs so it transmits 66 or 0b01000010, it will print 'B' repeatedly.

End of the series?

I originally intended to go further with this series, but then I got a job as a tester and learning about the ATmega328p was no longer my priority. So while I've put these posts up, and it was fun to work this stuff out, you can consider this the end of Behind the Libraries.




Follow Mark on Twitter