Zen2: Credit where credit’s due

So, AMD has released its new Zen2 architecture, codenamed ‘Matisse’, and now available commercially as the Ryzen 3000-series.

And I think I can keep this blog short: Intel has its work cut out, because AMD is back. AMD now has TSMC build their x86 CPUs, using their new 7 nm process. But Zen2 does not stop there. As in: it’s not just a so-so architecture that leverages a superior manufacturing process to push clockspeeds or transistor count into an advantage.

No, Zen2 is actually a very good architecture in its own right. It can compete head-to-head with Intel’s best offerings on IPC, and performance-per-watt is very good as well. Combine that with AMD’s standard weapon of value-for-money, and it feels like we’re back in the late 90s/early 2000s, where AMD’s Athlons would compete head-to-head with Intels Pentium III and 4.

Perhaps most interesting is that they chose not to use a single die, but to spread the functionality over multiple dies, where they can combine different manufacturing processes (something that Intel has done in the past with CPUs and GPUs/IMCs as well, such as the 32nm/45m Westmere).

It seems that Intel’s biggest problem at the moment is that they are still on 14 nm for their mainstream offerings. Now that is a first, as far as I know. Intel has always had the edge in manufacturing, even when their architecture itself was not superior. This allowed Intel to compete back in the Athlon days. But this time, Intel is on the back foot in manufacturing. And for the first time since the introduction of the Core2, Intel no longer has the more efficient architecture either.

So, I suppose we will now wait and see what Intel can do to strike back. What will their 10 nm process bring? And can they introduce a new architecture that once again takes the IPC/performance crown?

But for now, AMD seems to be in a very good place. I think that is good news. I have always said, since AMD had taken over ATi, that problems in the CPU department would hurt the GPU deparment, for the simple reason that the CPU market is larger, and would have their priority. And I think this is what we have been seeing in recent years. With AMD struggling since Bulldozer, their Radeon line was slowly losing grip on nVidia. So hopefully we will now see the reverse: with Zen2, AMD should be able to get back a lot of CPU marketshare, and boost their revenue considerably. And with that, they should be able to invest more in R&D for their GPU line as well, and close in on nVidia once again.

Posted in Hardware news | Tagged , , , , , | Leave a comment

Just keeping it real at Revision 2019

I visited Revision again this year. And I took my IBM PC/XT 5160 with me. And I made a quick release. That release was a plan that I’ve had for a while, and which I mentioned in my previous blogpost: a music disk for the IBM Music Feature Card.

And here is the pouet.net page. You could download the binaries, but not many people will have the required hardware, I guess. And it will also break many emulators, because IBM Music Feature Card support is not a common feature in emulators (yet). I made sure that the code runs on my DOSBox pre-release with IMFC support, and AMAME, as described in my previous blogpost.


Code was done by me, graphics by VileR of 8088 MPH fame, and the music was done by UP-C.

If you look closely, the graphics aren’t just rips of the original Sega title screens, but they are an impression that combine the title/logo with some in-game view. All this in colourful composite CGA of course.

Likewise, the music is not a straight rip, but is tuned specifically for the IBM Music Feature Card/Yamaha FB-01 with custom SysEx messages embedded in the MIDI data.

System requirements

The minimum requirements are: IBM PC or compatible, 8088 CPU at 4.77 MHz, 128kb memory, composite CGA, IBM Music Feature Card, 360kb 5.25″ floppy drive, PC-DOS 2.0.

Technical info

The code is reasonably straightforward. The MIDI player is based on my pre-processed data format, combined with my streaming disk routine. Why the streaming disk routine, you might ask? Well, I originally developed that for the PCjr MIDI player for the DreamBlaster S2P. The problem with MIDI data is that it has no structure. It is just a linear sequence of note data and delays. So when a part of a song is repeated multiple times, the data is duplicated. The PCjr only has 128k of memory, which is shared with video memory, and your average MIDI file will be too large to fit in memory entirely.

Since that code was already done, and it worked well, even from floppy, I figured I could just replace the code for the DreamBlaster S2P with IMFC code and use the rest as-is. That way, this music disk will run even on PCs with low memory.

In the future I might experiment with data compression. I have tried Trixter’s LZ4 code on some MIDI data, and it compresses really well. For example, the Hang-On song is about 112k, but compresses down to less than 7k. That might be very interesting, especially for PCjr, since it lacks the DMA controller to load from floppy in the background.

There is not too much to tell, other than that. Perhaps that the interrupt controller uses Auto-EOI mode, as discussed earlier. And there’s two cute tricks for the graphics. The first is that I am using a cute trick that is possible in CGA composite mode 6: fade in/out. I had already used that in the sprite part of 8088 MPH. Namely, by adjusting the background colour, you can affect the luminance, so I get 4 levels of luminance to fade the screen. The second trick is made possible because of this: I start with the screen faded to black. Then I load the graphics directly from disk into CGA memory. And then I fade in. This way I do not need any graphics data in system memory at all, which again is very nice for low-memory configurations.

That’s about it for now, hope you liked it!


Posted in Oldskool/retro programming, Software development | Tagged , , , , , , , , , , , , , , , , , , | Leave a comment

The IBM Music Feature Card and Yamaha FB-01

This has got to be one of the most obscure pieces of hardware I have covered so far. A sound card for the IBM PC, released in 1987. And by none other than Big Blue themselves! I am talking about the IBM Music Feature Card:


I suppose this card needs some explanation. As you can already see from the picture, it is a large, complicated card, with a lot more components than an AdLib or a CMS/GameBlaster.

That is because the card features a full-blown MIDI sound module, with its own Z80 microprocessor. The MIDI sound module is based on the Yamaha FB-01:


The rest of the card contains a proprietary MIDI interface. In fact, the entire card is designed and built by Yamaha, for IBM. In concept, it is very similar to the later Roland LAPC-I, which is also a combination of a MIDI interface and a MIDI sound module on a single ISA card, except in that case, it is an MPU-401 MIDI interface and an MT-32 module.

For more background information, you can read the excellent coverage of the IMFC on Nerdly Pleasures.

So, it is an FM sound card then. Is it any good? Technically, yes. Where the AdLib uses the low-end YM3812 chip, with 9 voices and 2 operators in mono, this uses the more high-end YM2164, which has 8 voices and 4 operators, in full stereo. The entire card is well-made, and was quite expensive at the time, aiming more at semi-professional musicians than gamers. Here is an example of Sega’s Afterburner theme converted to FB-01/IMFC:

However, like the AdLib, the card suffers from the fact that most musicians couldn’t really be bothered to make the most of the FM synth, as discussed in an earlier blog.

Anyway, I was fascinated by this card. Firstly because it is so rare, and not supported by a lot of software. Secondly, because it is an official IBM expansion card. And lastly because apparently nobody had bothered to add support for it yet, in DOSBox. So I decided to give it a try.

Is it really a Yamaha FB-01?

The approach to emulating this card appeared to be straightforward: there is the MIDI interface, and then there is the MIDI module, which is allegedly a Yamaha FB-01. Well, as usual, with PCs, it depends on how you look at it.

The MIDI interface is reasonably simple, and it didn’t take me too long to emulate enough of it to get games to detect the card, initialize it, and send MIDI data, which I could intercept. I simply made use of the MIDI code that was already present in DOSBox for the MPU-401, and sent the MIDI data out via the same interface.

This worked, and the only way I could test it, was with a real IBM Music Feature Card, with its MIDI in connected to the MIDI out of my DOSBox machine. But when I sent out an early test-build of this DOSBox for people to test with a Yamaha FB-01, they reported problems. Some instruments were wrong or missing in Leisure Suit Larry 3. What was going on?

Different firmware, that’s what was going on. While the hardware is mostly the same (the IMFC is obviously missing the LCD display and the front panel buttons), the IMFC apparently has an updated version of the main firmware. We studied the manuals of the IMFC and FB-01 closely, and mapped out all SysEx commands. We found that the IMFC supports one command that the FB-01 does not, and Sierra was using this command.

The command in question is the “Parameter List” command. Yamaha implemented an interesting type of SysEx command, which will likely confuse a lot of modern MIDI software. They basically have variable-length SysEx commands, which effectively switch the device into a certain state, after which you can send it a variable (in theory endless) amount of custom commands within that SysEx command.

The FB-01 supports one type of this list command, which is the “Event List”. It basically switches to a Yamaha-specific proprietary extended MIDI mode, where you can send key on/off, control change, program change, after touch, pitch bender and parameter change events, which allow some fine-grained control that normal MIDI does not.

The “Parameter List” however was apparently added to the firmware after the FB-01 was released. Similar to the “Event List”, it allows a variable number of commands. These commands modify the parameters of instrument settings. The FB-01 does of course support changing instrument parameters via SysEx messages. It just does not support the list format, only individual SysEx messages.

So the fix was relatively simple: the state machine has to be implemented in the emulation code. Whenever it detects the start of a Parameter List command, it will go into the parameter list-state. Then every command is in the list is translated to an individual SysEx command on-the-fly (that last part is extremely important, the list could be virtually endless, and you do not want to buffer the whole list before you start converting it). When an end-of-SysEx is detected, it switches back out of the parameter list-state, and Bob’s yer uncle.

Funny enough, we found out that this difference between the IMFC and FB-01 was known in the old days. Firstly, apparently someone once wrote a TSR for King’s Quest 4 and related Sierra games that modified the IMFC driver to be used on a MPU-401 and FB-01 instead. When I inspected the code, I found that this driver also did the same SysEx translation for compatibility reasons.

Secondly, Sierra also provided their own FB-01 driver for some games. Ironically enough, this driver also fails with some games. Why does it fail? Because apparently they translated the IMFC parameter list incorrectly. Namely, the parameter list works on the basis of instrument number. Sierra however translated to parameter change commands based on MIDI channel (the FB-01 supports both variations). The problem here is that there is not necessarily a 1:1 mapping between MIDI channel and instrument, so some parameters are set incorrectly with the FB-01 driver.

What’s a YM2164?

Right, with the basic MIDI problems out of the way, the next big step was to actually emulate the synthesizer part, so that people would not need a real Yamaha FB-01. I wanted to go with a similar approach to Munt, which is an emulator for the Roland MT-32. A Windows driver, which installs itself as a MIDI input device, and outputs the emulated sound to your audio device in realtime. This means that you can use it as a standalone synthesizer as well, using Windows MIDI tools.

The Yamaha FB-01 is a reasonably straightforward device: it has a Z80 microprocessor, some firmware in a ROM, and a YM2164 chip. The firmware for the Z80 contains the MIDI interpreter, which takes care of all the MIDI events, manages the voice banks and configuration, and drives the actual YM2164 chip.

So I figured I could re-implement the MIDI interpreter part myself, based on the SysEx documentation. The biggest problem is the YM2164 itself. This is a very obscure chip, mainly because it was not a chip that was sold to third parties. It was only used in Yamaha’s own products, and a few Korg synthesizers. As a result, you cannot find any datasheets or manuals online, because they were never made public. I tried contacting Yamaha, but they could not provide me with the documentation for the chip either, so it seems to be lost forever.

However, we still have the internets! And as luck would have it, Yamaha actually built modules with the YM2164 chip for the MSX computer range, the SFG-05. BUT! The YM2164 is only used in the later revision of the SFG-05. Earlier revisions used the YM2151. And there is our clue!

Namely, people have studied these different modules, and they are basically identical in terms of sound and operation. Upon studying the firmware, they only found a few superficial differences, in terms of undocumented registers and timer resolution.

And unlike the YM2164, the YM2151 is a common chip, with available documentation, which was used in various arcade machines, and has proper emulation support in MAME. So we can just use the YM2151 emulation from MAME and hook it up to our MIDI interpreter, and it should be a good virtual FB-01!

This was the initial plan. However, due to other priorities, I never finished my own implementation. Recently someone pointed out that MAME actually includes an entire Yamaha FB-01 machine (which indeed uses the YM2151 emulation code to approximate the YM2164). And there is also the AMAME fork, which focuses on using MAME synthesizers, and wants to make them into VSTis.

So I tried experimenting with MAME and AMAME a bit, to see if the FB-01 actually worked. And indeed, it did! This was somewhat of a breakthrough for my IMFC emulation project. Namely, I can now offer full software emulation. There is no physical FB-01 required anymore. This makes the project a lot more interesting for the average DOSBox user.

Here is a quick video I made to demonstrate the whole thing:

Early release

So I have decided to put together a quick release of what I have so far. This is my custom DOSBox build with IMFC support, and a compiled version of AMAME to offer you an emulated FB-01 to complete the chain. You can download it here.

In order to set it up, you need to do 3 things:

  1. AMAME needs two ROMs for the FB-01 emulator, which I did not include because of copyright issues. You will have to find them yourself. One is the FB-01 firmware, which should be in a file called fb01.zip. The other is the firmware for the LCD controller, which should be in a file called hd44780_a00.zip. Place those two files in the ‘roms’ subdirectory under AMAME.
  2. DOSBox needs to be configured to output to the correct MIDI device. In this version, there is no IMFC-specific section in the dosbox.conf file yet. I am re-using the MPU-401 configuration. So set up the MPU-401 as usual: under the [midi] section, you use the midiconfig=n option, where n is the index of the MIDI output device you want to use.
  3. AMAME also needs to be configured to get input from the correct MIDI device. This is done by name. Run “amame64 -listmidi” to get a list of the names of the possible MIDI in and out devices. Then use the -midiin and -midiout parameters to pass these names. I have included an FB01.bat as example, just change the names to the name of your device, and you should be up and running.

A loopback will be required between the MIDI out-device that DOSBox uses and the MIDI in-device that AMAME uses. I have tried LoopBe1 and loopMIDI, but found problems with both. The first seems to filter SysEx commands, which effectively filters out custom voice data from games, which makes it rather useless. The second seems to crash the FB-01 on large SysEx commands. I ended up using my E-mu 0404 USB interface with a physical loopback cable between its MIDI out and MIDI in ports. This works perfectly.

You could also use a real Yamaha FB-01 instead of AMAME of course.

Make a demo about it

The emulation still needs some work (it currently does not fully emulate the 8253 timer on the IMFC yet, MIDI in is not working yet either, and not all commands are fully implemented yet). But in its current state it should be good enough for all Sierra games at least.

I also still want to complete my own virtual FB-01. Firstly, because it should be more convenient to use than AMAME. Secondly, because my code will be a clean reimplementation of the MIDI interpreter, and will not require any copyrighted ROMs.

But for now, I think it is most interesting that the IBM Music Feature Card is now available to a much wider audience. And the next thing I want to do is probably some simple demo/music disk, targeting an 8088 at 4.77 MHz with CGA and an IBM Music Feature Card.

Posted in Oldskool/retro programming, Software development | Tagged , , , , , , , , , , , , , , | 1 Comment

The DreamBlaster S2P: General MIDI from a parallel port

In the interest of catching up, here is the second blog that is long overdue. I want to talk about the DreamBlaster S2P from Serdaco. I received mine over a year ago, and it actually was an early prototype.

The DreamBlaster S2P is based on the DreamBlaster S2 WaveBlaster module. It uses a Dream SAM2695 synthesizer chip. This chip has two MIDI interfaces. The regular DreamBlaster S2 uses the standard serial MIDI interface. The DreamBlaster S2P makes use of the special parallel interface, which allows you to send MIDI data one byte at a time over a parallel bus, such as an ISA bus, or in this case a printer port. It uses a similar approach to the OPL2LPT and OPL3LPT.

When I got it, it was still in the early stages of development, and nobody managed to get it to work properly yet. I tried a few things to see if I could get it working, but sadly it didn’t work. As it turned out, a patch wire was required. Pdewacht got it to work eventually, and he modified SoftMPU to support the DreamBlaster S2P. Based on his working code, I could get mine to work as well. In the final version that is now available, the problem with the patch wire is fixed in the design. A reset button was added to the design as well.

So what can you do with it? Well, the most obvious use case for now, is to use SoftMPU to play games with General MIDI support over an MPU-401 interface. This requires a 386 or better CPU. This is a very early video made by Serdaco when the device was working with SoftMPU for the first time:

The SAM2695 also has an MT-32-compatible mode, and I added a patch to support it in SoftMPU. Here are two early videos I made, to demonstrate.

The first video is an early test, where I use my own MIDI player with a captured MIDI file from Monkey Island (a game with MT-32 music, not General MIDI). First you hear it in General MIDI mode, then you hear it in MT-32 mode:

Pay no attention to the haphazard timing, it was still an early version of the MIDI playback routine. What you can hear is how the instruments sound very wrong in General MIDI mode. In MT-32 mode, the instruments sound much more like how the composer intended.

The second video is with my modified SoftMPU, running the actual game in MT-32 mode:

Also, like with the OPL2LPT, it is possible to patch native support into games. This means you will not need SoftMPU, so you also will not need a 386 or higher if the game does not require it. It can also work with protected mode games, which SoftMPU does not currently support. You can find S2PPATCH here. It currently works with DOOM and other games using the same sound engine (Heretic, Hexen, Raptor: Call of the Shadows, etc.), and with games that use the Miles Audio Interface Library, both 16-bit and 32-bit.

Going retro

However, I think the most interesting application of this device would be with newly written games, demos and other software. The same goes for the OPL2LPT and OPL3LPT. If you write new software, and you already want to support MIDI/OPL2/OPL3 music, it is trivial to add native support for these devices. You will not need SoftMPU, and you will not require a 386. The parallel port protocol is very simple, and takes barely more CPU time than a regular ISA device. So even on an 8088 at 4.77 MHz, these devices are a very realistic target, when used natively.

So I decided to add support for the DreamBlaster S2P to my MIDI player, and tried it on an 8088:

That worked quite well. All the complex timing is precalculated into a simple stream of delay values, adjusted for the PIT resolution, and blobs of MIDI data to send directly to the interface. No problem for an 8088.

So I decided to see if I could go even more low-end. I fine-tuned my assembly-optimized player routines, so they could run on my 128k IBM PCjr:

Even that works fine! It actually streams the data from floppy, because the MIDI file was too large to fit in memory. If you listen closely, you can hear the floppy drive reading the next bit of data. Because the PCjr does not have a DMA controller, it actually pauses the music for a bit. On a regular PC, the read would have happened in the background, and the music would continue playing. I suppose there are only two ways to avoid that: making smaller MIDI files, or using compression to be able to fit everything into memory, and decompress on the fly.

Basically, as far as sound devices for oldskool machines go, I would say that the DreamBlaster S2P is by far the best option on the market. It is cheap, and its sound quality is well above FM synthesis, comparable with General MIDI devices such as the Roland Sound Canvas. And because it is a complete synthesizer chip with built-in MIDI interpreter, it also has an extremely low CPU load for playing music. Much lower than an FM chip. So you have low cost, ease of use, good quality and light on resources. That is a tough act to follow.

Also perhaps good to point out: although these devices come from the DOS scene, they can obviously be used on any machine with a standard Centronics parallel port implementation. In fact, you could probably adapt the interface to various other bus types quite easily as well. All you need is an 8-bit parallel data bus (write-only is enough), and a few control lines that you can toggle (2 to 4 lines is enough). I can imagine a simple adapter for a Commodore 64 userport for example.

Serdaco SDK

To help others to support the Serdaco LPT interfaces, I dediced to make my routines available, in a simple Serdaco SDK. This contains some C and ASM code to initialize the devices and send data to them. It should make it very easy for people developing their own games and demos to add support for these devices.

I have actually had this idea of an oldskool DOS SDK for much longer. I have collected a reasonable amount of code for low-level hardware access on early PCs during my travels. The current Serdaco SDK is a subset of that. I want to release a bigger SDK at some point, which also includes support for regular OPL2, OPL3, SAA1099 (C/MS and GameBlaster), SN76489 (PCjr and Tandy), Sound Blaster, IBM Music Feature Card, Roland MPU-401, CGA, EGA, VGA, MDA, Hercules and related stuff.

I am basically trying to model this somewhat after the Amiga NDK. This is a collection of headers for C and ASM for the Amiga, with definitions for all the custom chipset registers and flags, and also the Kickstart and WorkBench libraries. I want to do the same, but for DOS-based PCs. It should bridge the gap between reading some technical manuals and getting the proper values for registers and such in actual code. I also want to include some basic helper functions and example code, so people can experiment with some code that is known to work on real hardware, and get up and running more quickly. I hope it will also aid the problem of people only developing on emulators, and creating code that doesn’t actually work on real hardware.

Posted in Hardware news, Oldskool/retro programming, Software development | Tagged , , , , , , , , , , , , | 1 Comment

Keeping it real for Outline 2018

It’s been a while, but I’m back. Or at least, I hope to have some more time to blog again from now on. And not just to blog, because many of my blogs are the result of experimentations with software and hardware. So I hope to have time to experiment, write new demos, and generally play around with stuff.

First things first though… A backlog of items has accumulated in the past silent months. And it occured to me that I never even wrote a blog about my last demoscene production, which was the invitation for Outline 2018, released at Revision 2018:

Granted, it is not too spectacular… Some graphics, a nice animation, and just a simple text writer with vertical scrolling, accompanied by some OPL3 music. But still, I’d like to describe some of the development behind it.


The music was done by Diode Milliampere in Adlib Tracker II, making fine use of the OPL3 functionality. Getting the music to play was a technical challenge. Adlib Tracker II may be open source, but it is written in Free Pascal. This makes it difficult to interface with other languages such as C and ASM, which I prefer. The code also requires 32-bit DOS or Windows.

Initially I went looking for an alternative player, and I found a routine in C. There were some problems with it though. The code was rather outdated, which meant it was not compatible with the later versions of AT2. One problem was that AT2 switched to a different compression algorithm. I tried porting the Pascal code for the decompressor myself, and eventually got it working.

However, because of the way the format is designed, you need huge buffers to decompress the pattern data, and the format is also extremely wasteful. Storing entire patterns in memory works fine when you use 4-channel mods, with 64 rows per pattern. However, OPL3 is 18 channels, and AT2 supports higher resolution patterns, up to 256 rows each. This means every single pattern takes up about 130k in memory, so things get very memory-hungry fast.

The same goes for the macro feature that the format supports. This has similar memory-hungry buffers. In short, just to decompress an AT2 file, you already needed about 5 MB of memory. So 16-bit was never going to be an option. I tried skipping the macro feature, and using some simple compression scheme for the pattern data (a lot of values tend to be 0 in every row, so you can store it efficiently by just making a linked list of the used data, and skipping the unused rows). It did eventually play some simple files in 16-bit mode, but you quickly ran out of memory still when there was more pattern data.

Another problem is that the replayer logic is also rather time-consuming with all the instrument and effect parameters running over 18 channels. The lower bound for playing AT2 files is more or less a 486SX-25.

So, I figured that it might be better to just go with my own preprocessing VGM player, which is at the other end of the spectrum in terms of CPU and memory usage. I just captured the music with DOSBox, and used the preprocessed capture data in the demo.


The graphics effects are rather simple. The main text is just a Mode X screen, where I scroll vertically by just updating the screen offset. I simply draw the next screen of text to the page that is currently invisible.

The other ‘effect’ is the animation at the start. This animation was drawn by visionvortex. He supplied it as a set of PNG files. I simply include all raw pixeldata in the EXE file, and copy the frames to the screen one by one, with the appropriate delay. Nothing fancy.

Final version

The party version was rather quick and dirty, but we wanted to release it at Revision anyway, because its goal was to invite people to the Outline party. It worked, but there was a small bug in the initialization routine for the OPL3 chip, which messed up the music.

Another problem was that the animation had quite a few frames, so it took up a lot of memory. This led to an EXE file of 1.7 mb. It also meant that I had to make it a 32-bit DOS release.

In the final version, both things are fixed. The music now sounds the way Diode Milliampere intended it. And I implemented my own simple Mode X RLE compressor and decompressor for the animation frames. The compression worked incredibly well on this animation, so the resulting EXE is only 87 kb. This meant that I could also make the code 16-bit, and fully 8088-compatible.

Posted in Oldskool/retro programming, Software development | Tagged , , , , , , , , , | 5 Comments

An emulator that 8088 MPH doesn’t break?

Breaking news! Or actually non-breaking news!

Just now I have managed to run 8088 MPH in an emulator for the first time, with all effects showing, and giving a reasonably accurate rendition of real hardware. The emulator is 86Box, and I have created a simple ZIP file here for you to try it. Everything should be pre-configured. Just wait for DOS to boot (can take a while), enter the date and time. Then type:

b: <enter>

8088mph <enter>

And you should be good to go. It’s not entirely perfect, the timing seems to go up and down a bit, the cycle-exact end-tune seems to play too fast. But it’s the best I’ve seen by far, and a lot better than nothing.

Posted in Oldskool/retro programming | Tagged , , | 9 Comments

Read any good books lately?

Recently, I have written a few things about Agile methods and Scrum. I feel that they are not the solution to everything, because in certain scenarios they have severe risks and drawbacks. I warned about how certain (sub-)problems cannot easily be solved in an iterative fashion, and require proper thinking, design and perhaps even prototyping before you move into Scrum.

Now, I have been reading the book “Requirements Engineering” by Axel van Lamsweerde. I did not expect the topic to pop up, but there it was. And in retrospect, it makes sense: requirements engineering is very closely related to the problem I was describing. You would not generally take a Scrum-approach, but you’d rather have a more spiral approach, where you refine your results in ever larger iterations (much like how you would first develop a prototype with a small team, and then refine it to the end-product with a larger team).

Van Lamsweerde does one better than that, however. He specifically goes into Agile development and points out that Agile methods only work under specific assumptions. In other words: they should only be applied in specific scenarios, and are not the solution to everything. He lists the following assumptions:

  1. All stakeholder roles, including the customer and user roles, can be reduced to a single role.
  2. The project is small enough to assign to a single small-size team on a single location
  3. The user is available for prompt and efficient interaction.
  4. The project is simple and non-critical enough to give little or no consideration to non-functional aspects, environmental assumptions, underlying objectives, alternative options and risks.
  5. The user can provide functional increments quickly, consistently (so that no conflict management is required) and gradually from essential to less important requirements (so that no prioritization is required).
  6. The project requires little documentation for work coordination and subsequent product maintenance. Precise requirement specification before coding is not an issue.
  7. Requirements verification before coding is less important than early release.
  8. New or changing requirements are not likely to require major code refactoring and rewrite, and it is likely that the people who develop the product, will also maintain it.

He specifically points out that mission-critical software is one area where agility is a poor fit. He also points out that ‘agility is not a binary notion’: For each phase of the project, you can choose to use less or more agility, depending on what is required for that phase. I think this is quite interesting, because Scrum-consultants seem to claim the exact opposite: any derivation from the strict rules of Scrum is considered a ‘Scrumbut’, and they always advise to apply Scrum more strictly.

Now, getting back to the example I gave earlier, where the team developed a system based on queues, where they should have used a database… Quite a few of the 8 above assumptions did not apply in that scenario, so Scrum was a poor choice from the start. The queues-vs-databases example itself immediately points to a violation of assumption 8: major code refactoring was required because of the ‘new’ requirements for prioritizing. But also, the project was a very mission-critical project for the customer. It was also a fixed-price, fixed-deadline project. This would violate assumption 4. The customer was in another country, so 3 did not apply either. I would also say that at least assumptions 1 and 6 were violated as well.

When I first heard of Agile and Scrum, it was mostly related to web-application development. And in retrospect, this makes sense. I suppose (small) web-applications tend to be a reasonable fit for Scrum. However, as we also discussed in the comments of the earlier articles, there are also projects that are highly technical and experimental in nature, and they are a very poor fit for Scrum. A spiral-like approach is much more natural there, where one or two people start by collecting the requirements, and iteratively working out the risks and problematic areas, perhaps developing some proof-of-concepts/prototypes during that time, and then refine the requirements, design etc, so that the project will gradually mature, and it starts making sense to add more people to the project, because there are now reasonably well-defined goals and workloads to be distributed.

The irony of it all, is that Scrum never claimed to be the solution to everything. Take the original paper by Ken Schwaber: Figure 6 clearly shows the Scrum Methodology, defining 3 different phases:


  1. Planning & System Architecture (Pregame)
  2. Sprints (Game)
  3. Closure (Postgame)

The way phase 1 and 3 are drawn in the figure, show that they are not really part of what Scrum is about (the sprints). They are shown in a way similar to the Waterfall methodology in Figure 1.

Schwaber further remarks:

The first and last phases (Planning and Closure) consist of defined processes, where all processes, inputs and outputs are well defined. The knowledge of how to do these processes is explicit. The flow is linear, with some iterations in the planning phase.

So basically Scrum never meant to solve everything with sprints. Not the planning, not the (intial) requirements, and not the system architecture. For some reason, everyone seems to have forgotten about this, and thinks Scrum is about doing everything in sprints, even all the design. That is a recipe for disaster with any non-trivial project.

Anyway, it’s always nice when your findings are confirmed independently by academic sources.

Posted in Software development | Tagged , , , , , , | 2 Comments

Intel tries its hand at a discrete GPU again?

As you may have heard a few months ago, Intel has employed Raja Koduri, former GPU-designer at AMD’s Radeon division. Back then the statement already read:

In this position, Koduri will expand Intel’s leading position in integrated graphics for the PC market with high-end discrete graphics solutions for a broad range of computing segments.

But at the time it was uncertain what exactly they meant with ‘discrete graphics solutions’, or what timespan we are talking about exactly.
But now there is the news that Intel has also employed Chris Hook, former Senior Director of Global Product Marketing at AMD.
And again, the statement says:

I’ll be assuming a new role in which I’ll be driving the marketing strategy for visual technologies and upcoming discrete graphics products.

So, there really is something discrete coming out of Intel, and probably sooner rather than later, if they are thinking about how to market this technology.

See also this article at Tom’s Hardware.

I am quite excited about this for various reasons. It would be great if NVIDIA would face new competition on the (GP)GPU-front. Also, Intel was and still is Chipzilla. They have the biggest and most advanced chip production facilities in the world. I’ve always wondered what an Intel GPU would be like. Even if their GPU design isn’t quite as good as NV’s, their manufacturing advantage could tilt things to their advantage. I’ve also said that although Intel GPUs aren’t that great in terms of performance, you have to look at what these chips are. Intel always optimized their GPUs for minimum power consumption, and minimum transistor count. So they only had a handful of processing units, compared to the thousands of units found on high-end discrete GPUs. The real question for me has always been: what would happen if you were to take Intel’s GPU design, and scale it up to high-end discrete GPU transistor count?

Perhaps we will be seeing the answer to this in the coming years. One other thing I had pointed out some years ago, was that Intel appeared to have changed course in terms of drivers and feature support. In the late 90s and early 2000s, Intel really had very minimal GPUs in every meaning of the word. However, when DirectX 10 came around, Intel was reasonably quick to introduce GPUs with support for the new featureset. Sadly it still took months to get the first DX10 drivers, but they did eventually arrive. It would appear that Intel had ramped up their driver department. DX11 was a much smoother transition. And when DDX12 came around, Intel was involved with the development of the API, and had development drivers publicly available quite soon (way sooner than AMD). Intel also gave early demonstrations of DX12 on their hardware. And their hardware actually was the most feature-complete at the time (DX12_1, with some higher tier support than NV).

Let’s wait and see what they will come up with.

Posted in Hardware news | Tagged , , , , , , | 7 Comments

The “Visual Basic Effect”

The programming language of BASIC has always been a popular option for beginners in the past. BASIC is an acronym that stands for “Beginner’s All-purpose Symbolic Instruction Code”. So its designers were clearly targeting beginner programmers. As a result, the language is quite easy to understand for people with little or no prior experience with computers. Its syntax is close to simple English phrases, and users do not generally need to worry about data types much.

On the Windows platform, Microsoft offered Visual Basic, in two flavours:

  1. The ‘full’ Visual Basic for stand-alone Windows programs.
  2. Visual Basic for Applications, a ‘scripting’ language for use inside MS Office and various other Windows applications.

Visual Basic was mostly used by inexperienced programmers, and people from outside the world of software development. Especially VBA, which was used by (non-programmer) people to add functionality to their Word and Excel documents, for example. Some even made ‘complete’ (and sometimes business-critical) applications this way. This resulted in poor average quality of Visual Basic solutions. This is what I would describe as the Visual Basic Effect.

Namely, in the hands of an experienced and properly educated software developer, Visual Basic can be used for decent software development, and can produce applications of good quality, even though Visual Basic obviously has its limitation (then again, don’t all programming languages? To a significant degree, software development is all about working inside limitations).

This is not limited to just Visual Basic, I think there is some ‘Visual Basic Effect’ in every programming language or environment. I think the most obvious example these days is in JavaScript. A lot of JavaScript is written by people who aren’t necessarily software engineers, but rather web designers who have learnt a bit of JavaScript to implement their designs.

As a result, many websites have highly inefficient, unstable, poorly written and bloated scripts. Now, I grew up with 8-bit computers of just 1 MHz and a handful of kilobytes for memory. And these machines could respond to user input ‘instantly’. I simply can’t get my head around the fact that today there are many websites that take a second or more to respond to a simple button-click, even on a computer with many cores, many MHz, and many gigabytes of memory. And certainly, there are fine examples of JavaScript-driven web pages that are very elegant and efficient. Of course, these are designed and implemented by proper software architects and engineers, not necessarily ‘people who build websites and know JavaScript’.

A related thing I encountered was with image processing, where it seems common to use Python to interact with a library such as OpenCV. Now, OpenCV itself is written in C++, by people who understand the intricacies of image processing, and know how to optimize it at a low level. The people who use OpenCV with Python are likely similar to the BASIC and JavaScript-crowd: People who aren’t necessarily software engineers themselves, but use Python as a simple tool to access OpenCV as a toolkit.

When people at work were looking for people to hire, they found that it was easier to find Python programmers with OpenCV experience, than people using C/C++. I pointed out the ‘Visual Basic Effect’ and said: “Are you sure the Python programmers are what you want though? They will know how to use OpenCV, but if what you are looking for is actually a custom re-implementation of certain functionality that is in, or similar to, OpenCV, these Python programmers are probably not going to be up to the task.” You will want the type of programmer that can develop something like OpenCV, and those are usually C/C++ people. These people think at a different level of abstraction, the level required to implement and optimize image processing routines such as found in OpenCV. Simply using OpenCV and rigging some processing routines together in a Python script is a completely different affair. Clearly, anyone well-versed in C/C++ is capable of picking up Python in a matter of days, and doing those tasks as well. But not the other way around.

I suppose that brings me back to the ‘aptitude’ that Joel Spolsky pointed out in his Guerrilla Guide to Interviewing. People who interview others for a job by just asking if they have experience with certain tools, or just asking a lot of basic facts, like a ‘quiz show’, are missing the point. You don’t want to know if the person you are interviewing happens to be familiar with the buzzword-of-the-day in the niche that your company’s latest project happens to be operating in, of if he or she happens to be familiar with the flavour-of-the-month tools and languages that your company is currently using. No, you want to know if the person has the aptitude to pick up the relevant knowledge and apply it in ways that generates value for your company.

And in that sense, it may often be better to hire people who are smart and experienced in other areas, than those who happen to know the tools you’re currently interested in, but have limited understanding and experience otherwise. In the case of the Visual Basic effect, you’re probably better off hiring non-VB programmers, because the developers that use more difficult and advanced languages such as C++ and C# will generally also be able to design and implement better VB code than people whose only exposure to software development has been VB.

There is just this huge difference between people who ‘know how to do X’, and people who have a solid understanding of relevant technologies, and can apply them to a wide range of tools and problems.


Posted in Software development | Tagged , , , , , , , , | Leave a comment

What is software development? An art? Craft? Trade? Occupation?… Part 3

In the previous two parts we have seen that software development is a very large collection of technologies and methodologies, and an area that has been in constant development since the advent of computers. However, even though every few years there are new ideas that are all the rage, and presented as the ‘silver bullet’ to solve all our problems and massively improve the quality and productivity, it seems to fall flat on its face everytime, once the initial hype dies down. In this part I would like to look at this from a different angle, and try to find some things that DO actually work.

Are new things really new?

In past blogs, I have given some examples in the past of ideas that aren’t really new, but can be found in papers or books published decades ago. I recently found this presentation by Kevlin Henney, that goes much deeper into this, and gives a great perspective on these ‘new’ ideas:

He gives a number of interesting historical examples of things that people consider ‘new’ today, but apparently were already being discussed back in the 1960s and 1970s. Some examples include abstract datatypes and objects, testing and iterative development.

This brings us back to Parnas (and in fact, a quote from Parnas from the 1970s appears in the above presentation by Henney). Because in his ‘Software Aging’ article that I mentioned last time, Parnas mentions exactly the type of thing that Henney discusses: A lot of ideas are already around, but they are not being communicated well throughout the software development community. The ideas are either not known at all, or they are not understood properly, in the context they were meant. As a result they are also not taught to new developers.

Now, when I talk to the average Agile/Scrum-fanatic, it appears that they think they finally found the answer to software development, and all those old developers simply were just messing around with waterfall models, and didn’t know what they were doing.

I think Henney proves a very important point in his presentation: These people DID know what they were doing. It is those people who think that these ‘new’ ideas are really new, that don’t know what they are doing. They are ignorant of the past, and are reinventing the wheel to a certain extent. And it seems that it comes down to the usual ignorant-arrogant pattern that we often see with the Dunning-Kruger effect: In their ignorance, they assume that they know more than the others, so they assume that their ideas are new, and the people before them weren’t as smart as they are.

It had to come from somewhere…

Now, if you think about it some more… Object-Oriented Programming (OOP) has been around for a long time now. The whole concept of Object-Oriented Programming had to come from somewhere. People didn’t just get together and think: “Let’s design some new programming language… here’s an idea! Let’s put objects in there!”. No, the concept of objects existed long before that. OOP languages are basically syntactic sugar for programming in a way that people even did with Pascal, C and assembly: working with objects.

It just seems that these ideas were forgotten or poorly understood, so when new generations of developers started learning object-oriented programming, they were not taught the right way. They understood how to write a working program in an object-oriented language, but they don’t know the languages that went before them, or what problems object-orientation tried to solve, let alone how best to solve them.

Now, Design Patterns will give you some examples of good solutions, but still I am missing something: nobody seems to tell you what the idea behind it is at the lower level. Let me try to give you a quick (and somewhat simplified) example:

An object can be seen as two sets of data:

  1. The actual data that is stored inside the object
  2. The methods of that object, laid out in a table

Methods are data? Well yes, they are function pointers. Methods are nothing other than procedural functions where the first argument is a pointer to the object. That is how inheritance and (virtual) methods work: You can store a function pointer inside the object itself. So if you pass an object to a method as a parameter, you are also passing the functions that should be called. So the object decides which function is called. The interface decides how the table of functions is laid out, so that index N in the table is always DoSomething(A, B, C) for all objects that implement that interface: it is a ‘contract’.

And when you understand objects at that basic level, it makes sense to just use objects and interfaces to pass function pointers around, so you can perform specific callbacks on that object from other methods and objects. And then Design Patterns such as Strategy and Bridge may suddenly make a lot more sense.

And of course you will also understand that people were doing this long before the Gang of Four published their book in 1994, and long before object-oriented programming languages were around.

As for hype… I have mentioned the No Silver Bullet article by Fred Brooks many times already. Even way back in 1986 he already warned about ‘new’ technologies and methodologies, surrounded by a lot of hype, which never seemed to deliver on their promises.

So clearly, there are some very good lessons that can be learnt from older sources. They are out there. You just need to look them up. Stand on the shoulders of those giants.

Get to the point already!

Brooks describes the ‘great designer’. Donald Knuth titled his most famous works ‘The Art of Computer Programming’, and has described programming as an art and a science. There is a very interesting study done by Barry W. Boehm in his book ‘Software Engineering Economics’, where he asserts that the key driver of software cost is the capability of the software development team. There is the concept of the ‘10x programmer‘, which was first seen in the paper ‘Exploratory experimental studies comparing online and offline programming performance’ by Sackman, Erikson and Grant in 1968. Joel Spolsky has pointed out that you should be looking for an ‘aptitude’ in programming, in his Guerrilla Guide to interviewing.

The constant here appears to be the ‘human factor’, not some kind of ‘magic’ technology that you have to use, or some kind of ‘magic’ methodology that makes everyone a great developer. It is about individual people, their talents, and their knowledge, skills and experience. The talent is something you cannot control. Knowledge, skills and experience you can improve. You can train yourself to become a better developer. You can train others to become a better developer.

And that brings me to the actual point here. As the title suggests, you can view software development in many ways. Viewing it as an art or a craft makes a lot of sense in this particular context: just like other arts and crafts, you can improve, and aim to master your craft. And you can guide less experienced developers to also become better by giving them proper advice and guidance.

This brings me to another recent movement in the world of software development. One that perhaps does not quite have the marketing skills of the Agile/Scrum movement, but their ideas are certainly no less valuable. On the contrary. I am talking about ‘Software craftsmanship’.

They try to make you view software development as some kind of medieval guild. And of course they have also drawn up their own Manifesto. For now I will leave you with Kevlin Henney once again, with a talk he has done on this subject. It aligns nicely with some of the things I have said on this blog before, and some things I haven’t:

Posted in Software development | Tagged , , , , , , , , | 1 Comment