A BRIEF HISTORY of SMSQ/E
Le Grand Pressigny, FRANCE – Tony Tebby
“Incompatibilities & Improvements, Bugs & Features”
“Was It All a Terrible Mistake?” I knew right from the start I should not do it, but so many people were asking for SMS to come out from under its covers that, in early 1992, I outlined a strategy (with Miracle Systems and Jochen Men) to make a “QL compatible” version of SMS available.
The outline was quite simple. A QDOS compatible SMS kernel existed (and had been in regular use since 1990). A complete set of SuperBASIC procedures and functions existed. A complete set of (extended QL style) device drivers for the Atari ST series existed as well as “portable” disk, serial and parallel port drives for other hardware. There was an environment which supported QLiberator compiled programs. All that was required was the core of a SuperBASIC interpreter.
Following the success of the Gold Card, Miracle Systems were looking for a legitimate operating system for their (as yet undefined) forthcoming computer and Jochen Men needed an operating system to legitimise the QL emulator for the Atari ST series. It was clear that it would be best to provide a version for the Gold Card as well but as Miracle Systems did not want to get involved in selling software, the Gold Card question was left.
I embarked on trying to find out what Sinclair’s SuperBASIC interpreter did. It was not difficult defining what it should do, but Jan Jones had built it on the principles of a GIGO (garbage in, garbage out) machine. With limited ROM space, there was no room for real error checking so Jan just tried to make sure that whatever rubbish SuperBASIC was asked to deal with, it carried on and did something. The “something” was not always obvious.
While WE (QJUMP, those wonderful people at QVIEW, Jochem Merz, Albin Hessler etc.) would never deliberately exploit “holes” in SuperBASIC, the same was not necessarily true of other software suppliers or contributors to PD libraries. In addition, even WE had been known to fall through a hole by accident.
Compatibility, therefore, meant not only reproducing SuperBASIC as it was intended, but reproducing as many oddities as would be necessary to execute most QL software. The two “compilers” for SuperBASIC programs provided a starting point. The aim was to provide a BASIC interpreter which would provide:
1. better compatibility with SuperBASIC than either compiler, 2. execution at least half the speed of QLiberator, 3. an environment supporting both QLiberator and Turbo compiled programs.
Clearly, as some software for the QL will not even work on all QDOS ROMs, total compatibility with a particular QDOS ROM can only be provided by copying that ROM code. Even slight re-ordering of the QDOS ROM routines (as in the Thor XVI) can cause considerable incompatibility. At this stage, there was no intention of providing improvements. The “Minerva Experience” had shown the extent to which the slightest improvements could give rise to extensive incompatibilities. So much for intentions!
The Birth of the QXL
“They should not have done it either!” In principle, the implementation of SMSQ on a 680×0 processor embedded in a PC should have been fairly straightforward. The Gold Card used the IBM PC disk controller, the IDE hard disk interface is not very different from the Hardcard used in the Miracle QL Hard disk and the serial and parallel ports on the PC are much the same as the serial and parallel ports that you find anywhere except on the QL.
If the QXL had been designed as a card which plugged into a standard AT motherboard (no processor or memory) and provided with drivers to drive a standard keyboard interface, a standard multi IO / floppy / IDE card and a standard Super VGA card, it would have been simple.
To have done this, however, Miracle would have had to have supplied the PC hardware (at a cost of about half the QXL card for a single floppy / 110 Megabyte HD configuration this would have seemed, to me, the obvious way to do it).
However, it would have meant that the machine would not have been usable as a PC (sigh of relief) and would have ruled out the use of portables and notebooks. In addition, second hand PCs were widely available either free or for less than $30.
As one of the most likely reasons for a second hand PC being available at a very low price was that the hardware was not quite a perfect clone, then there was no possibility of having a version of SMSQ which would access the IO devices directly. Also, if the QXL were to be put into a real “working” PC, it would not only have to co-exist with its host but would have to work through whatever low level software (Stacker, Doublespace, Hypercache, Smartdrive etc.) was used to improve the IO performance of the PC.
As a result, any direct access from the QXL to the PC was ruled out and the QXL was to be hosted by a DOS program. A logical decision, maybe, but, from the point of view of the operating system software, it was a disaster.
Where is the problem? The PC comes complete with device drivers for all of its peripherals all that needs to be done is to pass data from the QXL to the PC device drivers (using BIOS calls) and vice versa.
There are three problems with this.
1. The design of the PC BIOS does not take account of the requirements of multitasking (it is, for example, impossible to write something to disk while you are waiting for input from a serial port).
2. While the accuracy of the reference manuals about the QDOS operating system entry points left a lot to be desired, the (in)accuracy of “reference” manuals for the PC gives a whole new meaning to the word reference.
3. All the reference manuals (so far examined) for the PC were written in the days of the PC and PC/XT. The PC BIOS also dates from this period. The BIOS has been considerably, and incoherently, changed while the manuals have been superficially updated to take account of AT keyboards, hard disk drives larger than 10 MByte and 3.5″ and HD floppy disks.
Take, for example, formatting a floppy disk on the PC. On QDOS, it is a single operating system call. On the PC, however, part Of the format operation is performed by the application. The format routine for the Gold Card floppy disk driver took a couple of hours to write. As the PC BIOS does most of the work for you, it should be easy to write a format routine on the PC.
There are a variety of DOS and BIOS calls to help you do this: setting the device parameters, formatting and verifying tracks etc. I have three reference manuals which give example format programs.
I look at the first one, and think “this is very strange”. There seems to be no way in which you specify the density and there seem to be no checks for whether the tracks have been correctly formatted: it appears to be automatic. So I try it. Fine, it makes all the right noises and tells me that my DD disk has 1440 sectors. I try an HD disk. Fine, it makes all the right noises but tells me that my HD disk has only 1440 sectors. It try without a disk at all. Wonderful, it formats much more quickly and tells me that I have 1440 sectors! – AU FREE!
I try the second program: this one checks the error return from the “Format track and verify” call: it even allows me to specify the density. I try it with a DD disk. Fine! it tells me that there are 1440 sectors. I try it with an HD disk. Fine! it tells me that there are 2880 sectors. I try it without a disk: the format fails, excellent! I try it with a bad DD disk telling that it is HD. Fine! it tells me that it has 2880 sectors. Suspicious, I try to copy some files to it using DOS. DOS refuses to recognise it. I try the other two disks: neither is readable. Over to QDOS to look at the disks: there are no sectors 1, 2 or 3 on any of the tracks I look at. This would explain why the first program did not bother to check the error return from “Format track and verify”: it does not verify!
On to the third program. This is similar to the other two, but uses the old “INT 13h” BIOS calls rather than the more powerful “DOS function 44h” calls (wonderful this DOS terminology). This requires the use of a separate “verify sectors” call. The verify sectors call seems to work: this routine gets an error on every track: it is right, after formatting, none of the tracks are readable on any type of disk.
So, I try it myself. After a lot of experimenting with the BIOS calls described in the various manuals, I am able to write either DD or HD tracks and verify them. The only problem is that I write too many sectors to a track: the last sectors overwrite the first sectors on the track. After a week of work, I can select the density and I can nearly format a track.
Thinks! Microsoft can do it, I should be able to as well. Now we start to see the problem: the DV3 floppy disk format routine is less than 512 bytes. Microsoft’s FORMAT program is greater than 32 kbytes, almost the size of QDOS, all its device drivers, SuperBASIC and all its procedures and functions. No wonder all the manuals are wrong. It would take about 200 pages just to list the MSDOS FORMAT program, without trying to explain how it works!
Disassembling all of this program could take months. I decide to trace the two paths of interest: 720k and 1440k formats. It turns out that what you need to do is to poke special values into various undocumented locations in low memory. I note all the locations to be poked and set up a format routine. Within this routine I poke all the required locations, format the disk and restore all the locations to their previous values.
Success, I can now format DD and HD floppy disks. The only problem is that, despite my care in restoring all the poked locations, after a format the PC refuses to recognise any disk change until you hit the reset button. Two weeks have passed and I still do not really know how to format a disk using MSDOS.
Do I spend another 2 weeks finding out how to restore the BIOS after a format operation? Even if I succeed in making it work on my PC, will it work on any other PC? How can anyone succeed in selling an operating system where it takes two weeks to write a routine using the operating system calls when it would only take two hours to write the same routine accessing the hardware directly?
Now we find the real cost of the QXL in development. Even though the IO performance of the QXL is well below the levels that it would be reasonable to expect, the implementation of the QXL device drivers has cost between 10 and 20 times the cost of equivalent drivers for other 680×0 platforms. As a result, all the time that had been set aside for the development of the SBASIC interpreter has been swallowed up. For the first purchasers of the QXL, things looked grim: poor IO performance, no SBASIC interpreter. Not a very promising debut for SMSQ.
“You Take the High Road and I’11 Take the Low Road” The QXL hardware strategy was not the only problem to be faced. Miracle Systems, for reasons which should be obvious, wanted the QXL to seem as much like the Trump Card and Gold Card as possible, while Jochen Merz wanted an operating system which was not just developed along the same line as the Atari QDOS extended device drivers but one which went much further.
One man’s improvement is another man’s incompatibility. Now we have the problem of developing (and maintaining) two different variations of SMS: SMSQ, the basic QL-like version and SMSQ/E, the extended version which is likely to diverge evermore and more from SMSQ. Jochen Merz, therefore, decided to supply SMSQ/E for the QXL as well as the Atari and Gold Cards. Easy for him to decide: it was me that had to do the work!)
More problems. It seems that computer users are not very sensitive about how much they have to pay for their operating system. They are, however, very sensitive about how much other users pay! Gold Card and Atari users do not complain about having to pay for SMSQ/E (we told then it would be necessary back in 1990), but they do object that QXL users get a “free” version of SMSQ with their QXL. QXL users do not seem to mind being asked to pay extra for SMSQ/E (at the moment the differences are fairly small so it is not usually worth “upgrading”) but they do object that Gold Card and Atari users are not being asked to pay more.
Even worse, there are some QXL users who seem to think that they are being provided with a specially naff version of SMSQ to oblige them to cough up a few extra pennies for an upgrade!
Then to cap it all, Miracle Systems produce a Super Gold Card which looks like a Gold Card, but turns out to be rather different. We now have implementations of SMSQ on three distinct hardware families, seven different hardware variants, four different display types, with four different 68000 series processors, in three (and sometimes more) languages. So far, there are more possible combinations than there are users.
To avoid the necessity of producing a different version of SMSQ/E for each user, SMSQ now uses a module structure which has been borrowed from the Stella (Stella????) operating system. This allows operating system modules to be selected (or ignored) as the system is booted. In principle, a single version of SMSQ could be delivered which would autoselect the right modules for any hardware combination. In practice, each hardware family (Gold Card, QXL and Atari ST/TT) requires its own special loader, so that it is not worth incorporating all the modules in each version.
Just as the number of users starts to take off, so does the number of variations. Jochen Merz ships a copy of SMSQ/E to a Gold Card user: the next day there is a message “SMSQ/E does not work with the XXX keyboard”. Not surprising, the XXX keyboard uses a patched version of the JS ROM. The cure? Another keyboard driver module for the Gold and Super Gold Cards and another language module (the keyboard tables). The net result is one new user and four new variations. Counting variations is soon going to be like counting marbles in a cookie jar.
“‘Till Death us do Part” In the days of easy divorce for reasons of mutual incompatibility, it is surprising to find so many QL users wedded firmly to the old software packages of the “use it at your own peril” style. Then I started the evaluation of SBASIC, Miracle Systems sent me a bundle of diskettes (about 10 Mbytes worth) of the type of software that they thought might provide a test for the compatibility of SBASIC.
I started looking at these disks on the Atari ST with JS and the E level drivers. After resetting the Atari ST for the tenth time without having found any software which even started to work, I gave up and tried using a Gold Card.
After a day or so, I found two programs that could be executed, played with and removed without crashing the system. All the rest either crashed right at the start, could not be made to do anything sensible, or could only be removed by resetting (I began to understand why some users have been asking for a quick reset). I have been told that a lot more of the software would have worked if I had set the memory size to 128 kbytes, but if you are going to reset you machine to 128 kbytes use one program and then reset again, there is no point at all in using SMSQ: you might as well stick with QDOS on your old faithful QL. Seriously, does anyone use this type of software anymore?
The first compatibility tests were very encouraging: all the programs which crashed on a JS QL crashed with SMSQ. It seemed that we had obtained better than 95% compatibility. Moreover, one of the two programs that worked on the QL worked with SMSQ: the figure was up to 98% compatibility.
“New Lamps for Old” One of the best ways of checking the originality of software is to investigate the bugs. If two items of software perform the same functions correctly, one could be a copy of the other, or they could both be written to the same specification. If, however, two items of software exhibit the same bugs, it can be assumed that one is copied from the other.
There are very few “first level” bugs (bugs which prevent the system functioning correctly under “normal” conditions) in QDOS. Because of the GIGO policy and the desire to limit error checking to a minimum to maintain efficiency, there are a much larger number of “second level” bugs (where the system misbehaves when passed incorrect parameters or data structures) and even more “holes” (where calling a system function with deliberately incorrect parameters has a reproducible if bizarre effect).
During testing of SMSQ and SBASIC, a large number of second level bugs were uncovered in the JS ROMs. Many of these showed up also in Minerva, none in SMSQ or SBASIC. From time to time, users have uncovered a number of second level bugs in SMSQ and SBASIC. All of these were entirely new and have no connection with old QL ROM bugs: SMSQ and SBASIC are entirely original!
Streamlining code has the effect of removing, altering or introducing holes. It is not surprising, therefore, to find that many of the holes that are exploited by some common software, have either disappeared or been altered in Minerva (giving rise to complaints of compatibility problems).
One such case is the xx.xxxxx SuperBASIC vector which is the same in all QL ROMs. This vector is intended to be used with data structures set up by the SuperBASIC interpreter. This has three defined paths controlled by the value of one byte (0, 2 or 3). Someone discovered that it could be made to produce a bizarre effect if the passed a value of 1 in the control byte. The resulting code fragment (which takes longer than using a legitimate call) has been incorporated into a utility, which has found its way into a large number of programs for the QL. The streamlined Minerva code no longer had this hole so a large amount of software stopped working on Minerva. The Wizard did not manage to find the real villain in the code, but succeeded in restoring “compatibility” by setting a register to a value which it would not normally have with SuperBASIC. This, in turn, altered another hole and introduced different compatibility problems.
In SBASIC, however, the hole never existed. Once the villain code had been identified (a week’s work) it was, therefore, a simple matter of detecting the villain case and emulating the hole directly. It was a waste of time and effort, and it slows down SBASIC, but that’s what it’s about, isn’t it?
The boundary between a bug and a hole is a very fine one and if some software relies on a bug in the QL ROM do I need to reproduce this bug? Unfortunately, the answer is sometimes GRRRRRR YES.
Twice recently, I have received reports of “bugs” that have appeared in the string handling in recent versions. These “bugs” have been introduced into SBASIC to improve compatibility with SuperBASIC (there are still three “bugs” in SuperBASIC string handling which are not emulated in SBASIC). Neither of these users was aware of that the bugs existed in SuperBASIC: SBASIC is now being used where SuperBASIC never went before.
“Whose Fault is it Anyway?” One rather tetchy letter complained that SBASIC was very fragile by comparison with SuperBASIC: using a well-known piece of commercial software: “SBASIC crashed”. This was misdirecting the blame. As the software was invoked correctly by SBASIC and as it never returned to SBASIC, SBASIC could hardly be to blame.
This well-known SuperBASIC extension started off by trying to identify a fragment of the QL ROM, and, when it could not find any QL ROM code (there is none in SMSQ), it jumped to a completely arbitrary location. BANG. The cure: I re-wrote the extension and incorporated it (with improvements) in SBASIC.
In fact, SBASIC is more robust in this respect than SuperBASIC: error trapping is much more thorough (and forceful). If it had happened in a daughter SBASIC: it could simply have been removed with no harmful side effects. (Perhaps I should implement a keyboard “restart” for Job O.)
Another difference between QDOS and SMSQ which might give the impression that SBASIC is more fragile than SuperBASIC is the default error handling: QDOS carries on but SMSQ stops to allow a debugger to be started. If the job is already being monitored by a debugger, there is no difference. In the normal state, however, allowing a job which has produced illegal instruction or address errors to continue could easily result in widespread damage to the system data structures, possibly resulting in the loss of part or all the data on a hard disk. SMSQ is, therefore, much safer, even if jobs appear to stop more often.
Many programs compiled with the current version of Turbo are wonderful examples of this. At the start of these programs we find some code which sets a location in memory to O. Several times later on, the value in this location is moved to register A2 and then there is the instruction to move the contents of address O (A2) to 04, D2 is compared against the new value in D4 and then there is a conditional branch.
On the QL the MOVE sets D4 to 3 so the operation of the code is dependent on whether D2 is greater or less than 3.
On Atari STs modified for the old QL emulator, the MOVE sets D4 to 24,622. The behaviour of this Turbo code will, therefore, be significantly different on these STs as the operation now depends on whether D2 is greater or less than 24,622.
On unmodified STs with a patched version of the JS ROMs, the MOVE will cause a “bus error” which QDOS ignores and so execution will continue without changing the value in 04. 02 is, therefore, compared against an unknown value: this will give yet different, and rather unpredictable, behaviour of Turbo programs.
On unmodified STs with SMSQ, the MOVE is trapped and these Turbo programs just stop. It is possible to set a special “Turbo mode” (PROT_MEM O) which emulates the QL ROM access by setting 04 to 3 and continuing. The behaviour of Turbo programs on these STs is, therefore, the same as on a QL: SMSQ is more compatible with the QL than QDOS!
“I See no Ships” While SMSQ with SBASIC marks a great improvement in performance and capability over the old QDOS ROMs, this has not been achieved without creating a few problems. The original SMSQ and SBASIC had very few intrinsic bugs, but many incompatibilities. Unfortunately, it is one of the facts of life of computing that making any changes to a existing software borders on vandalism. The neat structure of the original conception begins to crumble and soon each little “fix” risks introducing a host of new problems. Fixing each one of these introduces more. It is a tribute to Jan Jones original SuperBASIC conception that during the early days of “active development” when new features were being added every day, the ratio of changes to bugs introduced was better than 10:1.
SBASIC is a much more complex piece of software which has the disadvantage of being required to emulate all the quirks of someone’s first attempt at writing a BASIC interpreter. It, therefore, starts off being not very neat and it is prone to degenerate more quickly. SBASIC’s ratio of changes to bugs introduced is closer to 5:1 – small enough to be convergent, but too large for comfort.
Fortunately, the bugs introduced from version 2.11 (experimental SBASIC) through version 2.25 (the first “release” version) to version 2.42 (current at time of writing) have usually been smaller than the ones they replaced.
Although some compatibility problems with some hardware variations remain to be resolved and there are one or two program which still refuse to function with SMSQ, the original aims for compatibility and performance have been well exceeded and SMSQ is now establishing itself as a living replacement for QDOS.
So, is SMSQ/E at last stable? The answer must be no. As more and more people start using SMSQ/E there are more and more requests for improvements (i.e. decreasing compatibility). SMSQ/E has now passed the point of no return: there are more requests for improved capability than for improved compatibility. The current versions of SMSQ/E are at least as reliable as any QL ROM version and are getting as close to 100% compatibility as is possible while providing better performance and more facilities.
SMSQ/E is a commercial product and as such needs to meet users demands. If users require changes, and it is commercially feasible to provide them, they will get them. SMSQ/E cannot, however be developed in all directions at once.
For software development, a 16 MHz 68000 based Mega STE (1 MIP with 2 Megabytes of memory running under SMSQ/E (my “standard” configuration) is more than a match for a “standard” 50 MIP 32 Mbyte workstation. A TT or a QXL on a good 486 machine is more impressive still. The Gold and Super Gold Cards have the same standards of raw performance, but suffer from limited display capabilities, poor keyboard and IO.
How much more is it reasonable to do with SMSQ which, by its need for compatibility with a 10 year old computer, is locked into a 10 year old design? Is there enough interest in the type of operating system concepts pioneered by QDOS to make it worthwhile producing a completely new system? The future depends on your response.
March 2014 M T W T F S S « Feb 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31