Quick note about the new fetch() API

If you're running into a problem where the server is properly returning a Set-Cookie header, but subsequent requests from your browser aren't sending them (either via the fetch() API or through normal requests), I have the answer!

Your call to fetch() must have the following configuration option: credentials: 'same-origin' The documentation states that this option is required for the fetch() API to send cookies, but it is also required for the browser to accept cookies returned from the server. Lesson learned the hard way.

A Warning About Windows 10 and Dual-Booting Linux

I have had a lot of positive experiences with Windows 10, and figured it was time to upgrade the last machine before hitting the "not free anymore" deadline. This was the first one that didn't go well.

The machine had 4 primary partitions: Lenovo recovery, Windows 7 NTFS, Linux (ext3), and Linux swap. If you dig back into the bowels of history, MBR-style partition tables can only support 4 partitions. Apparently, during the upgrade Windows 10 slightly shrinks its own partition and creates a new one for Windows 10 recovery. This would put the number of primary partitions up to 5, which there just plain is no room for in the MBR. From what I can tell from my failed recovery efforts, this process involved Windows creating an extended partition and trying to move the ext3 and swap partitions into it. Well something went wrong there, the partition table got mangled, and the ext3 partition disappeared.

In the process, I learned a lot about the TestDisk [1] program that comes with Kali [2]. This looks like a great disaster recovery tool, but try as it might, my disk was sufficiently mangled that it couldn't recover the Linux partition.

Lesson learned: Don't install Windows 10 until you're sure you've got room in your MBR for an extra partition!

My plan going forward is to nuke the remnants of the Linux partitions, try to install the Windows 10 boot loader from a USB key, and then re-install Ubuntu afterwards. Luckily I keep all of the important stuff from the Linux partition in git, so I didn't lose anything other than time.

Coming soon: a post about Windows and Linux full-disk backup solutions. The vast majority of my work is stored in git repositories or Dropbox, so I haven't spent a lot of time thinking about machine recovery, but I've sunk enough time into fixing this mess that I figure it's worth it.

A Simple Serial Packetization Protocol - The Wire Protocol

I was having a chat with a friend the other day about reliable serial data transfer, and I want to share a scheme that has worked well for me in the past. It's split into two layers: I'll call them the "wire protocol" and the "command protocol". One of the great things about this protocol is that it is really simple for a low-memory microcontroller to implement.

The wire protocol is used to encode the start and end of packets in the serial data stream. In this scheme, these are single bytes. You can use whatever you want, so long as it's consistent on both ends. For the rest of this example, I'll use 0x02 ("start of text" in ASCII) and 0x03 ("end of text"). Let's say the packet you're going to transmit is "Hello". The resulting output is going to be:

0x02 H e l l o 0x03

Super simple! The microcontroller can sit and read bytes from the UART, and when it sees 0x02 it knows that a new packet is being received. There's a problem with this scheme though: what to do when the packet contains 0x02 or 0x03? This would confuse the receiver, since they'll be half-way through receiving a packet and will either terminate prematurely, or think that a new packet is starting.

The solution is an escape character. I'll use ASCII ESC (0x1B). The escape character is used to indicate to the receiver that the next character should be used as-is and not be treated as a control character. This must be applied to any instances of the start byte, end byte, or escape byte in the data to be transmitted. A simple approach is to pass through the following character untouched. So transmitting a packet of

0x32 0x02 0x1B

would result in this output:

0x02 0x32 0x1B 0x02 0x1B 0x1B 0x03

That's pretty decent, and straightforward to decode, but it makes me uncomfortable that 0x02 and 0x1B are transmitted multiple times, and have different meanings depending on where they are relative to escape bytes. One of the rules that I have about serial data streams is that it's quite possible for them to be in an unknown state (e.g. a power glitch while a data stream is running, or someone bumps a cable, etc). If the microcontroller missed the first byte in this example, it would still lock on to the following start byte (0x02) and decide that a packet of (0x1B) had been transmitted.

A simple tweak to that algorithm is to give the escape character the following meaning: the byte that follows the escape character is the bitwise-not of the byte to be written. We still escape the same three characters (start, end, and escape), but they only ever appear in the data stream when they have their intended meaning. For reference, ~0x02 = 0xFD, ~0x03 = 0xFC, and ~0x1B = 0xE4. Going off the previous example, encoding:

0x32 0x02 0x1B

results in this output:

0x02 0x32 0x1B 0xFD 0x1B 0xE4 0x03

In this example, no matter where the receiver starts receiving, it will never inadvertently receive a start, end, or escape character unless the transmitted byte has that intended meaning. And that's it!

Summary

To summarize, this protocol has 3 special bytes:

  • A start byte (0x02) to indicate the start of a data packet
  • An end byte (0x03) to indicate the end of a data packet
  • An escape byte (0x1B), which is used when the data packet contains one of the three special bytes. The escape byte is followed by the bitwise-not of the special byte, to ensure that the special bytes only ever appear in the data stream in positions where their special behaviour is warranted.

Pseudocode

Here's some C-ish pseudocode for a transmitter:

#define START_BYTE 0x02
#define END_BYTE 0x03
#define ESCAPE_BYTE 0x1B

void tx(char* buf, int len) {
  putchar_uart(START_BYTE);
  for(int i = 0; i < len; i++) {
    if((buf[i] == START_BYTE) || (buf[i] == END_BYTE) || (buf[i] == ESCAPE_BYTE)) {
      putchar_uart(ESCAPE_BYTE);
      putchar_uart(~buf[i]);
    } else {
      putchar_uart(buf[i]);
    }
  }
  putchar_uart(END_BYTE);
}

And here's some receiver code:

int rx(char* buf, int maxlen) {
  int i = 0;
  bool started = false;
  bool flip_next = false;
  char c;

  while(1) {
    c = getc_uart();
    if (c == START_BYTE) {
      i = 0;
      started = true;
      continue;
    }
    if (c == END_BYTE) {
      if (started) {
        break;
      } else {
        continue;
      }   
    }
    if (c == ESCAPE_BYTE) {
      flip_next = true;
      continue;
    }

    if (flip_next) {
      c = ~c;
      flip_next = false;
    }
    buf[i] = c;
    i++;
    if (i == maxlen) { break; }
  }
  return i;
}

I just typed that in by hand and haven't tried compiling it. Let me know if there's something wrong with it.

Solving my power problem on the Surface 3

I got a Surface 3 a few weeks ago, and I've fallen in love with it, except for one thing: the batteries dies overnight if it's not plugged in.

I started out trying the Sleep Study, which had potential. It shows a nice graph that I'm right... the battery level drops off overnight. If you want to get the report (which is great!), run "powercfg.exe SleepStudy" from an Admin Command Prompt. It'll generate "sleepstudy-report.html". If you want more days, try "powercfg.exe SleepStudy /duration 7" to get the last 7 days worth of stats.


Combing through it, all I really got out of it is that my Wireless card is on while the Surface is "sleeping", and so is the "Intel SD Host Controller". But figuring out what's going on beyond that has been elusive, until tonight!

Running "powercfg /requests" is supposed to identify which applications have made power requests.


Aha! Media Sharing!

Now, I thought I could turn that off through Windows Media Player (since, you know, it says "enabled through Windows Media Player"). Turns out that won't do it.

Go into Control Panel, Administrative Tools, Services:


Find the Windows Media Player Network Sharing Service:


And then... Kill it with fire. Stop it, and change it to Disabled.


Checking back at our trusty "powercfg /requests" list:


Huzzah! Maybe it'll sleep now!

Edit: Yup! Woke up this morning 7 hours later with 95% charge, and I did do a bit of stuff after I had unplugged it last night. Success!



Bringing up an MRF89XAM9A radio

I'd like to document the process I went through recently when I was bringing up an MRF89XAM9A (link at bottom of post). This is a pretty cool little radio module that costs lest than $10 from Newark and has FCC/IC modular certification, which should dramatically reduce the amount of hassle required to put it in a product.

Lately I've been using the ST Micro STM32F042 for a few projects, and this one is no different. I've already got a solid USB stack put together for it, and I'm at the point where I pretty much just drop it into designs without thinking too much about it. Looking at the datasheet for the radio, there's two chip select pins, two interrupt pins, and standard SPI pins for communicating with it.

Starting out, the thing I care most about is verifying that I can even talk to the thing. For that, the interrupts don't matter, just SPI and the CS pins. So that's where we start!

First, let's initialize all of the clocks, GPIO pins, and the SPI controller.


That's all pretty generic stuff. Note that this all goes in its own function, not right in main(). Even when it's a prototype and simple, don't put it right in main(). Get your project structure in place first, because there's a good chance that this codebase is going to end up in your final product.




Test Post

Is this thing on?