A first look at Kafka

19 June 2017

I've heard a few people mention the Kafka project recently, and as a good computer scientist I thought I ought to give it a once-over. I read through the high-level description and thought it was interesting, and I wanted to collect my thoughts here.

What is Kafka?

Kafka is a distributed streaming platform, a message queuing system optimized for stream processing.

The word distributed should tell you that Kafka runs as a cluster, making it more fault-tolerant. Kafka vocabulary: - record: A message delivered to Kafka - topic: An ordered sequence of records, this is how you group related records together

Kafka stores the records in each topic for a configurable length of time. This means that a record essentially remains "available" or "in flight" for this retention period, after which it is presumed to have been consumed by anyone who wanted it and is discarded. From this angle, Kafka can be seen as a special-purpose distributed file system optimized for processing relatively short-lived data.

Kafka defines four types of users: - Producers: Produce or publish records to a topic - Consumers: Consume records on a topic - Streams: Consume records from one or more topics and Produces new ones to one or more topics. Essentially makes it easy to write Consumers that monitor a variety of topics. - Connectors: To help you get existing systems like databases to interact with Kafka. The introductory docs don't go into a lot of detail on Connectors.

Kafka also supports grouping Consumers into Consumer Groups. Kafka delivers each new record to one Consumer in each subscribed Consumer Group, so you can increase the number of Consumers you have of each type for scalability or fault tolerance, and Kafka will ensure that only one of them gets each new record.

Note that consuming a record does not destroy it; as noted earlier, records remain available to any interested Consumers until the retention period expires. In addition, Consumers may review records in any order. They are not required to be only forward-looking.

How do I talk to Kafka?

A peek at the docs suggests that there are Java classes to make it easy to talk to a Kafka cluster. Of course, since Kafka is a distributed application, you can talk to it from any network-capable program provided you follow its protocol. Kafka uses a binary TCP protocol.

Why is it named Kafka?

Speculation

I figured it had something to do with the author Franz Kafka. Kafka's most famous work, at least to graduates of American high schools, is probably The Metamorphosis. Since Kafka enables stream processing, whose goal is the transformation (i.e. the metamorphosis) of the data, I'm guessing this is what the namer had in mind.

Reality

Alas, the namer, Jay Kreps, begs to differ, saying simply that Kafka was a writer with a cool-sounding name.

Towards Securing an Ubuntu Desktop

14 Mar 2017

The Hack

Over the weekend, Virginia Tech's IT Security staff alerted me to unusual traffic leaving my desktop at work. Gigabytes of encrypted data were headed to China, which they deemed "unusual", and rightly so. They were polite enough to ask my advisor (who owns the machine) whether this was expected behavior.

After disconnected the network cable, a member of VT's TechStaff and I wandered through the machine looking for evidence of a hack. We found some -- some very odd-looking files in /tmp belonging to root. We applied the strings utility to one of the binaries and found a long list of IP addresses. Another file was a Python script that referred to accessing a "back door", and it turns out they had added a /usr/bin/sshd to my machine that I guess had a back door in it. As a final note, they had added a file to /etc/init.d/ to automatically load their attack binary when the system booted, ensuring they would have continuous access to my machine.

The Aftermath

Happily, the attackers don't seem to have damaged any of my data. I've spent the last two days checking over my data, taking a backup, installing a fresh version of Ubuntu, and restoring my data on the new system.

Here are the steps I took, for future reference:

  1. Create a Linux live USB stick. The disks utility made this fairly straightforward.
  2. Boot from the USB (make sure you've updated GRUB to boot from USB before hard drive), then mount your hard drive. The OS in your USB should automatically detect the hard drive for you.
  3. Find a big external media onto which you can copy your data. I dug up a 300GB external drive and brought it into the office. Taking tiny "sips" on a small USB drive is not a good idea; use the gallon bucket.
  4. Identify the data you want! This is presumably the content of your home directory.
  5. Copy it over to your external media, sync && sync && sync && sync && sync; then safely unmount your external media. I created tarballs to reduce the amount of overhead in FS metadata (e.g. inodes) on the external media. I figured the directory hierarchy in the tarball was lighter weight than thousands or millions of inodes.
  6. Bonus points if you now confirm that the backup on your media looks OK, e.g. by un-tarring it on some spare space in your machine's hard drive and looking at a few files.
  7. Take notes on any system configuration details. e.g. packages you've installed, your network configuration (mine had a static IP and VT-specific DNS servers, needed to get a wired ethernet connection to work), etc.
  8. Install Linux afresh, blowing away your old installation.
  9. Restore your backup.

There might be easier backup solutions available if you trust your OS and can boot into it, but as the attacker had root access I did not want to pursue this route.

This process would have been easier if I cleaned up after myself better and did a better job creating README files to label things. There were about 30GB of unnecessary data in my home directory that I deleted before I took the backup, and a bunch more that seemed redundant ("X.backup.tgz", "Y_orig", etc.) but that I didn't want to risk throwing away.

Becoming More Security Conscious

I'm used to maintaining systems that don't spend a lot of time online, and that sit behind routes, like my laptop. But my desktop at work has a static IP and can be accessed from anywhere in the world. Since it's on 24x7, that's a lot of time for an attacker to spend brute-forcing the system.

Here are the steps I've taken to improve the security of my machine so far:

  1. I chose a new, longer password.
  2. I changed the ssh port to something other than 22. This was also done on the previous incarnation of my desktop, but seems to have been inadequate.
  3. I tweaked /etc/ssh/sshd_config in two ways to decrease the attack surface: I set PermitRootLogin to 'no', and I added an AllowUsers rule to limit login attempts to only my account.
  4. I enabled Linux's Uncomplicated FireWall (UFW), which is simpler to use than traditional iptables. As shown by nixCraft, I configured it to limit login attempts to "No more than 6 every 30 seconds". This requires a brute-force attacker to spend way, way longer trying to crack the password.

Well, that's it. I hope this is secure enough. I certainly didn't enjoy spending my time repairing the system, but

March Madness Squares

10 March 2017

Introduction

My wife's research group runs a March Madness Squares pool. She wanted some advice on which square to pick. See this GitHub project for an explanation of March Madness Squares and the extract-squares.pl tool I wrote to improve your chances of winning.

The idea is to identify the best squares, either per-year, or across all years. I believe an aggregate analysis will be most useful, as there is little reason to suspect that a particular set of scores would be meaningfully more common in one year than another.

extract-squares.pl | sort -k 3 prints the hit count for each square, sorted from fewest matches to most. The squares at the bottom of the list are your best bets. The squares at the top, your worst bets.

$ ./extract-squares.pl `ls history/*.txt` | sort -k 3 ... squares[1][4] = 6 squares[1][6] = 7 squares[2][8] = 7 squares[4][1] = 7 squares[7][5] = 8

Based on my analysis, you should pick square 7,5 this year.

Further analysis on the 2012-2013 through 2015-2016 tournament data

Counting the 4 play-in games, there are 67 games per tournament. With 100 possible squares, at least 33 squares will never have a hit. Over 4 years (467 = 268), the expected number of hits for each square is roughly 2.7. Some squares *never had a hit!

squares[0][2] = 0 squares[2][1] = 0 squares[2][3] = 0 squares[4][5] = 0 squares[7][2] = 0 squares[7][7] = 0

Here's the entire distribution. Each row expresses the number of squares (first #) that got that many hits (second #).

$ ./extract-squares.pl `ls history/*.txt` 2>/dev/null | sort -k 3 | awk '{print $3}' | sort | uniq -c 6 0 21 1 24 2 20 3 16 4 8 5 1 6 3 7 1 8

If you look at the corresponding histogram, you'll see that it looks decidedly not like a normal distribution. The mean number of hits is about right (between 2 and 3), but the right-hand tail is much longer than I expected.

Charts square hit count against frequency

I've been making pull requests to the node-webworker-threads project, which is a module that lets you offload work in Node.js. The native cluster and child process modules, and derivatives thereof, let you coordinate or offload work to other processes. node-webworker-threads, on the other hand, lets you coordinate or offload work to threads, somewhat reducing both the data transfer overhead and the OS resources consumed. My largest contribution (not particularly large, if we're being honest) was to convert some of the V8 API calls to NAN calls to eliminate compiler warnings on Node.js v7.4.

This week I got some exciting news. The project owner, audreyt, invited me to be a collaborator on the repository. Essentially this means that I have write access to the repository, and can merge pull requests, add labels to or close issues, and other administrative activities. I've never been a collaborator on a project before, so I consider it an honor. I must use this power only for good...

I've been on a Node.js kick of late, and I wanted to learn how to write C++ code that can be called from my Node.js JavaScript. This is known as a C++ add-on and is a great way to accomplish any of three ends:

  1. Get blazing-fast performance (though Node.js JavaScript itself is claimed to have pretty good performance thanks to V8?).
  2. Talk to an existing C++ library.
  3. Interact with libuv, e.g. to offload work to the threadpool. You could also use child processes for this, but it can be more efficient to use libuv's threadpool instead.

To this end, I got a copy of Scott Frees's C++ and Node.js Integration. While there are a lot of different blog posts on how to write C++ addons for Node.js (and the worthy official documentation), they seem to stop at "Hello World" or assume you're a wizard. Frees's book is pretty cheap compared to the pain of trawling the Internet for help. I have received no compensation for this recommendation.

Anyway, the book has been useful and I'm learning a lot. I won't bore you with the details of writing a C++ add-on -- you should read those blog posts or Frees's book. However, I had some trouble debugging my toy C++ add-ons, and I couldn't find a helpful source on the Internet for guidance (though I found other askers, e.g. Alexander List and Ryan Cole). In this post I will explain how to debug a C++ add-on on a UNIX-style OS (Linux, Mac) with gdb.

First, compile your add-on using node-gyp with the --debug flag.

$ node-gyp --debug configure rebuild

Second, if you're still in "playground" mode like I am, you're probably loading your module with something like

var ObjModule = require('./ObjModule/build/Release/objModule');

However, when you rebuild using node-gyp in debug mode, node-gyp throws away the Release version and creates a Debug version instead. So update the module path:

var ObjModule = require('./ObjModule/build/Debug/objModule');

Alright, now we're ready to debug our C++ add-on. Run gdb against the node binary, which is a C++ application. Now, node itself doesn't know about your add-on, so when you try to set a breakpoint on your add-on function (in this case, StringReverse) it complains that the specific function is not defined. Fear not, your add-on is part of the "future shared library load" it refers to, and will be loaded once you require() your add-on in JavaScript.

$ gdb node
...
Reading symbols from node...done.
(gdb) break StringReverse
Function "StringReverse" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y  

OK, now we just have to run the application:

(gdb) run ../modTest.js 
...
Breakpoint 1, StringReverse (args=...) at ../objModule.cpp:49

and we can list the source code, step through the operations, etc. just as we normally would in a gdb session.

I for one am looking forward to debugging my modules without printfs!

As I discussed in an earlier post, I maintain a website. You've probably figured that out by now because you're currently on it, reading this blog.

Now, the blog doesn't use a "blogging engine". Until today, it was just just HTML+CSS. This meant that when I wanted to write a new blog post, I also had to add formatting tags for things like paragraph breaks, bulleted lists, links, and so on. My friend Ayaan suggested I look into Markdown, a way to deal with this annoyance that apparently the rest of the civilized world has heard about already.

For the uninitiated, the premise of Markdown is that the content of information should be emphasized over its format. This was exactly my frustration with blogging in HTML: I spent so much time worrying about the format that I couldn't focus on the content. Markdown is a simple language that can be easily converted (by Gruber's perl script) into HTML; it's syntax sugar that makes writing HTML content easy.

Embedmd

Great, Markdown rocks, end of story? Well, not quite. At the moment, my blog has a research section and a technical section, and each section has its own HTML file chock full o' HTML-fueled blog posts. With Markdown I could rewrite the blog posts, but how could I cleanly translate the blog posts back into HTML to embed in the appropriate HTML file? I didn't turn up any good solutions on Google, so I whipped up the embedmd tool.

The idea of embedmd is to let you easily embed Markdown in HTML. See the GitHub page for the latest info on how to do this, but the premise is simple:

  1. Extract the content from your HTML file and put it in one or more Markdown files.
  2. Replace each piece of content with '#INCLUDE markdown.file.name'.
  3. Run the embedmd tool on the HTML file; it will locate the #INCLUDE lines, convert the corresponding Markdown files to HTML, and substitute that HTML in place of the #INCLUDE line. Example: embedmd blog-research.mdh > blog-research.html
  4. Since the "HTML" file is no longer really an HTML file, you should probably change its suffix from .html to .mdh (Markdown-HTML).

How do I make sure my Markdown looks OK?

This SO post pointed me to the grip project, a lightweight way to view Markdown as HTML.

  1. Install grip: pip install grip
  2. grip my_markdown.md
  3. Paste the link it supplies into your browser.

One of the great things about being a student at a large public university is that there are all sorts of goodies available to you if you know where to look. For example, if you're a student of a more studious bent, you probably spend time in the library. It turns out that on the main level there's a 3D printing lab -- seriously, you can't miss it, since most of the walls are glass and it's full of about 5 3D printers. Here's the webpage if you're curious.

The gist of it is that you can walk in with a CAD drawing on a flash drive and they'll print it for you, provided your object will fit in a 6x6x8 inch cube. It's pretty amazing! There's usually 1-2 student workers in there helping "customers" get acquainted with the system and taking 3D printing orders. 3D printers and printing isn't cheap, so I'm assuming there's some grant money behind it.

I've never been particularly good with my hands, but I am pretty good with software, and I've been itching to give it a try since I learned about it last year. One of the tricky parts about 3D printing, though, is that you need to have something in mind to make. There's an abundance of little tchotchkes you can make, tiny horses or dogs or what-have-you, but it's harder to think of something that I might actually want to have in my house.

I've been mulling this dilemma -- what to make that wouldn't just be a waste of plastic -- over for a few months, and recently I thought of something! My wife went to a wedding whose favors were airplants. These little guys don't need any soil, they just want to be spritzed or soaked in water every so often. They've been sitting naked on her desk, and it occurred to me that I could 3D print a home for them.

I stopped in to the 3D printing lab to ask what software they recommended, and they sent me to TinkerCAD. This CAD (that's Computer Aided Drawing) website offers a web-only interface and a set of tutorials to learn you some CAD. There's a shallow learning curve to get started with basic geometric shapes (rectangular boxes, cylinders, cones, etc.), but to do anything really fancy you need to code it up in JavaScript or make it in another CAD tool and upload it. I really enjoyed the magic of being able to "turn" an object and look at it from all sides as I edited it in major ways and gave it minor tweaks. It was kind of like Mario 64, in which you could use the yellow buttons on the Nintendo 64 controller to rotate the camera, except that in the CAD software you could actually rotate it to get all of the angles you wanted (always a frustration in Mario 64). You could even hollow out an object and sneak a peek from the inside. Overall this was an exciting first CAD experience for me, although if I do it again I think I'll try another piece of software that offers a bit more control from the GUI without needing to wander into JavaScript.

What does the air plant home look like, you ask? Well, I brought my CAD drawing to the 3D printing lab, and they said it would be ready in 1-2 weeks since the line was pretty long. I'll upload a picture once it's done!

I recently heard rumor of a convenient way to access files on a remote machine. Though rsync is nice if you have a directory tree you want to deploy all-at-once, sometimes you want to edit files on your local machine and on a remote machine concurrently. sshfs lets you do exactly that. It uses Linux's FUSE (File system in Userspace) to allow a user to access ("mount") a directory tree on a remote machine provided that you have ssh access. Hence the name -- sshfs, or SSH File System. I wanted to access my Virginia Tech personal account, readily available via Virginia Tech's rlogin cluster. I have passwordless ssh set up already, so access was a piece of cake.

$ mkdir /tmp/davisjam
$ sshfs davisjam@rlogin.cs.vt.edu:/home/grads/davisjam /tmp/davisjam
$ mount | grep davisjam
  davisjam@rlogin.cs.vt.edu:/home/grads/davisjam on /tmp/davisjam type fuse.sshfs (rw,nosuid,nodev,user=jamie)
$ ls /tmp/davisjam | wc -l
  30

I've set up NFS before, and sshfs was definitely way easier. Most machines already have ssh installed and sshd running, so I didn't need to install new packages or muck around with config files in /etc. One obvious drawback of a remote file system is that performance is cruddy. Observe.

$ dd if=/dev/zero of=/tmp/10M bs=1M count=10
  10+0 records in
  10+0 records out
  10485760 bytes (10 MB) copied, 0.0153374 s, 684 MB/s
$ dd if=/dev/zero of=/tmp/davisjam/10M bs=1M count=10
  10+0 records in
  10+0 records out
  10485760 bytes (10 MB) copied, 15.5244 s, 675 kB/s

It's approximately 1000 times slower to write into the sshfs. Your mileage may vary, and depends on things like upload speed (on your end), download speed (on the remote end), and what the I/O rates are on the remote end. Once you're done accessing the sshfs, you should unmount it. Since it's in userspace, you shouldn't need to use umount; instead, try fusermount -u.

  $ umount /tmp/davisjam
    umount: /tmp/davisjam is not in the fstab (and you are not root)
  $ fusermount -u /tmp/davisjam
  $ mount | grep davisjam
  $

And there you have it. An easy-peasy way to interact with a remote directory tree.

I recently had to profile my application. StackOverflow recommends gprof or Callgrind) for C-based applications on Linux. To use gprof you have to both compile and link with -pg. After suitable modifications to my Makefiles, I tried gprof. You just run your program as normal, and it produces an output file that you can examine with gprof. This must mean that -pg adds code to your application, but to do what? Immediately my program crashed with SIGPROF: Profiling timer expired. Since my code doesn't typically receive SIGPROF, this must be the effect of -pg. At a guess, -pg tells the compiler/linker to add instrumentation to your code, plus a signal handler for SIGPROF. Then your program is sent SIGPROF repeatedly, and the signal handler records what your program was up to at the time. With a long enough runtime and a large number of signals, such a "sampling profiler" can give you a statistically good sense of what your program is up to. Given the crash, it seems that part of the application outside of my control wipes signal handlers, overriding whatever gcc inserts to handle SIGPROF when you compile/link with -pg. This seems to be a wildly unusual issue, since google returns almost nothing useful.

I decided to try a simpler program.

void func1 ()
{
  return;
}

void func2 ()
{
  return;
}

int main()
{
  int i;
  for (i = 0; i < 1000; i++)
    func1();
  for (i = 0; i < 10; i++)
    func2();
  return 0;
}

Profiling with gprof is simple.

  g++ -pg sample.c -o sample

Profiling with callgrind. Callgrind is an add-on to valgrind, and you can visualize the results using a graphical tool called kcachegrind.

  valgrind --tool=callgrind ./sample
  kcachegrind callgrind.out.3715

These produced results as you might expect. On to the real McCoy, using callgrind this time.

UV_SCHEDULE_MODE=RECORD UV_SCHEDULE_FILE=/tmp/f.sched valgrind --tool=callgrind `which node` fs_nondet.js

annnd...crash! -- Profiling timer expired. Whoops. I was still compiling+linking with -pg, and Profiling timer expired as explained in the SO post referenced above. After removing -pg I could get down to business. Profiling is an iterative task; you remove one hot spot and then re-run, identifying the new hot spots that manifest.

-  mylog(LOG_LIST, 9, "list_elem_looks_valid: begin: list_elem %p\n", e);
+  DEBUG_ENTRY_EXIT_LOG((LOG_LIST, 9, "list_elem_looks_valid: begin: list_elem %p\n", e));

Without compiling with -DJDLOGEE (EE: entry-exit), this call to mylog will never take place.

Things I learned:

  1. kcachegrind is a great tool.
  2. Performance profiling can be surprising. I wouldn't have guessed that the calls to the logging function were so expensive, nor did I imagine that my inefficient linked list size implementation would actually matter later on.

I've spent the last two weeks chasing a bug in a C program.

My program has an internal tracing facility (that's a fancy way of saying I wrote a mylog() function that has classes like MAIN, SCHEDULER, LIST, and so on, and offers control over verbosity levels). Sometimes when I ran the program on sufficiently complex input, the output would be missing newlines. The output was also confusing -- based on the location of the calls to mylog, it seemed like the execution was disobeying program order. The program would also hang sometimes at the end in what, according to the traces, was an unexpected place.

After consulting with some wizards, I believed the issue was stack corruption. How else could the program traces indicating "entry" into a function halfway through it be correct? gcc offers some aids to deal with stack corruption, essentially inserting canary values onto the stack and testing them prior to function return. However, adding -fstack-protector-all and -fstack-protector-strong to my compilation flags didn't change anything. However, there are ways to corrupt the stack that wouldn't be caught by such canary values (namely, unlucky writes elsewhere in the stack), so this was not conclusive.

I then spent a few days looking up sources of undefined behavior in C programs. I also enabled -Werror (treat all warnings as errors) to force myself to clean up a few hundred warnings that had piled up. Some of these warnings were actually potential causes of undefined behavior. In particular:

With the warnings removed, the bug remained.

After that I began to add unit tests, magic numbers for corruption detection, and asserts, to the various modules I had added: list, map, tree, scheduler. I even added unit tests and asserts to my tracing module, and that's when I found it!

My mylog is implemented (of course) via printf/fprintf. printf's man page reads "If an output error is encountered, a negative value is returned" and I had been ignoring the return value of printf. I know checking the return codes of function calls is Programming 101, but it hadn't occurred to me that printf could fail. It turns out that it can! In my bulletproof'ing of mylog, I had added an assert that printf actually printed the same number of characters as were in the string. Most of the time this passed, but it would sometimes fail with -1. Interestingly, the printf man page on my system is silent on the types of errors that can be encountered. I checked the man page for putc, which was similarly silent, but as strace indicates that the underlying system call is write, 'man 2 write' suggests a list of potential reasons. One of them was a match: my printf call was failing with errno EAGAIN: "The file descriptor fd refers to a file other than a socket and has been marked nonblocking (O_NONBLOCK), and the write would block." The unexpected jumps in execution were actually a sequence of failed printf's!

I tried changing the fd for stdout to be blocking, but that didn't change anything. I opted instead to put the printf call in a loop until it had printed the full length of the string. This eliminated the "funky output" and made it clear that my program was hanging in a reasonable spot, and I was able to fix the deadlock.

Interestingly, I would get EAGAIN even if the entire string except for the trailing newline was successfully printed. In this situation I would have expected a return code of N-1 where N is the length of the output string. Anyway, this means my output sometimes looks like "XX\n": the output string X is sometimes repeated. My program now has a stutter, but this is much better than losing its voice.

Lessons learned:

Recently I discovered that the internal behavior of Node.js varies depending on where the output is going.

src/node.js:
    case 'FILE':
      var fs = NativeModule.require('fs');
      stream = new fs.SyncWriteStream(fd, { autoClose: false });
      stream._type = 'fs';
      break;
...
    case 'TTY':
      var tty = NativeModule.require('tty');
      stream = new tty.WriteStream(fd);
      stream._type = 'tty';
      break;

and so on. The upshot of this is that the program would behave differently whether I sent the output to the terminal (node foo.js), to a file (node foo.js > /tmp/out), or to a pipe (node foo.js | tee /tmp/out). Specifically, when sent to the terminal the program would crash, while when sent to a file or a pipe it would work fine.

I needed to examine the traces emitted by the program when it crashed, but since they had been spewed to the terminal this was tricky. A bit of googling turned up this interesting linux tool. Problem solved!

Before:
1. node foo.js ---> crash
2. Scroll around for awhile trying to do error determination.
3. Eventually give up.
After:
1. script
2. node foo.js ---> crash
3. C^d
4. Examine the file 'typescript' which contains stdout from my program run.

A neat little tool indeed.

I've wondered about the ioctl syscall for years now. ioctl stands for I/O ConTroL, and is the standard way to interact with hardware and other low-level stuff. Today's the big day -- I actually need to use it.

Problem: Determine how much data is pending on an fd, without calling read().

Solution: First of all, a quick google turns up the answer:

Search text: "determine how much data is ready on an fd without calling read"

First hit: http://stackoverflow.com/a/5409705. A similar search on Bing yields: http://stackoverflow.com/questions/6979769/linux-ioctl-with-fionread-always-0.

At IBM this would have been sufficient, and I would have gone off and implemented the solution. Since I'm in grad school, I was unsatisfied with copy/paste code. I turned to the man pages and google to try to learn more about what was going on. Consulting 'man tty_ioctl' (ioctls for terminals and serial lines [i.e. including TCP/UDP streams]), I discovered what FIONREAD does:

   Buffer count and flushing
       FIONREAD  int *argp
              Get the number of bytes in the input buffer.

This Linux Journal article describes a similar exploration for someone trying to determine whether or not a machine has an ethernet cable in its port.

Well, after a few hours of tinkering I've made a discovery. Kodi's API has really limited support for playlists. In fact, playlists as you might think of them (the jazz playlist, the oldies playlist, and so on) aren't even a proper object in Kodi's API.

When you call Playlist.GetPlaylists, you always get 3 playlists back: audio, mixed, and video. These playlists corresponding to what is actually being played -- literally the "play list" or queue of items that will be played next.

What I was hoping to find was an interface to a list of saved playlists so that I could request the playlist of my choice. Instead, it appears that I'll have to add support for playlists on top of the Kodi APIs, for example by clearing the audio playlist and calling Playlist.Add a few times.

In other news, the Kodi wiki is sadly lacking in playlist examples for those of us with a headless media server. There are tons of examples of playlist creation through the GUI, of course. Here are the details I scraped together:

Sample dumb playlist:

[playlist]
PlaylistName=MyFirstPlaylist
File1=/media/RPi external/Admin Music/Andrew Peterson/Love and Thunder/01 Canaan Bound.mp3
Title1=Canaan Bound
Length1=5
File2=/media/RPi external/Admin Music/Andrew Peterson/Love and Thunder/08 High Noon.mp3
Title2=High Noon
Length2=5
NumberOfEntries=2
Version=2

Sample smart playlist:

While smart playlists seem more usable than dumb ones (hence the name?), the Kodi wiki is shy about links to the playlist specifications for manual editing. Nevertheless, I can see both of these playlists through Yatse and play them. Yatse is presumably using the Files APIs to parse these playlists, since Kodi's GetPlaylists APIs don't return these playlists.

At home I've got a raspberry pi running the Kodi media server. I wanted to be able to control it using my voice, so I've been working on voice control (using CMU's sphinx project) and Kodi control (using Kodi's JSON-RPC API). I have just posted the latest version of my kodi.py python module to github. It's pretty close to Arn-O's project, but my version is rather more object oriented. Also this way I get to learn more about python, JSON, and reading documentation.