Open Source Depot Development

The Ubuntu Switch and Prima

2008-06-10T04:27:00.000-07:00

The switch went well. I can pretty much do everything I did with Windows, and then some. It is especially nice to have a POSIX environment. Don't get me wrong. Cygwin is very good, but slower than Linux and subject to the arcane Microsoft ways, especially when it comes to drive mapping and security.

Once settled in, I modified the Prima build environment to include Linux. It had all the hooks in place. I just never finished the work. Now that Linux is on the laptop, it was convenient to do. Besides, the old "server" has to go anyway so I don't want to go sit in the computer room to work on it.

If there is any demand, I'll post the changes. My next step is to add some real device drivers. Again, I'll publish the work if there is any demand. After that, I'll finally attack the API. There may be some surprises.

Switching to Ubuntu

2008-05-19T04:04:00.000-07:00

After a long, painful WIndows experience, I finally switched to Linux for my laptop. This was partly due to continual security problems, even with the gargantuan Symantec tools, which leads to my second issue: speed. There seems to be countless services being dropped in by every little software distributor in existence. Mostly, they lurk looking for updates to download, but who knows what other purposes some of these may have in mind.

Software manufacturers, I am putting you on notice. I own the hardware, and your software is purchased for my use. The hardware is not yours to do as you please. So, you have now lost a source of revenue thanks to your disregard of my personal property.

I know Linux also has its share of daemons. A simple ps shows lots and lots of running processes on an idle system, but I can point to a specific operation of feature that each of these perform, and I can look at the source to make sure it does what its supposed to. Also, the underlying operating system is fundamentally more secure, so I don't have to worry about architectural design flaws making my system a playground for a script kiddie or other hacker. Plus, it boots faster, runs faster, and is generally much easier to use.

The switch took place about a month ago, and while I consider the switch a success, I did have to make the system dual boot because there are three programs for which I cannot find Linux equivalents. I would gladly buy an upgrade to Linux in each case, since I bought the original licenses and they were not cheap (several hundred dollars each). Unfortunately, the software developer does not yet support Linux. Until then, I occasionally will have to boot to Windows, but for that 99% of the time where I do not have to, I am much happier.

I can now say that my computing environment is pretty much Microsoft free.

More Thoughts on a Target

2008-01-06T05:00:00.000-08:00

I finally got back my last bit of feedback. It definitely seems that FreeDOS and DOS in general does not seem to be a worthwhile target. There simply isn't enough interest to make the effort worthwhile. I won't rule it out for some time in the future, but clearly it is not of enough interest to warrant a high enough priority to be an essential or early feature.

Having said that, what I am going to do may seem contradictory. For at least the Proof of Concept release, the native API will be DOS based. It is easy, since the code is in place and already running as of the last POC release. Keeping it going for the immediate future will certainly not hurt. Instead, it will help me get to a funtional POC much earlier than any other API would.

I have another old project I may tap into. It is "UNX" and is unix based. It was a new kernel written to take advantage of the Unix v7 and v32 open source releases. I wrote it after porting the original v32 kernel, only to find that adapting it to modern computers would require such radical changes as to be just shy of a rewrite. It was, in fact, faster to write a new kernel than it was to do the rewrite. In UNX, I have an inode based FAT files system and an API compatible with Unix v7. It may be a good resource for Prima v 0.10.

In the mean time, work on the next POC release continues. ACPI memory detection is a pain in the rump. The simple fact that the specification allows for memory holes means that I need to manage an array of memory and not just a simple model as end of kernel to top of memory for our pool of memory. Oh well ...

P.S., The project was called UNX as a nod to the old Digital Equipment Corporation facility where I worked through both Compaq and HP acquisitions. It was where VAX System V was written, as well as parts of Ultrix. Much of OSF/1 was also written there, which eventually became Digital UNIX and later Tru64 UNIX. That facility was shut down by HP in 2006.

Prima Gets A Memory Manager

2007-12-22T06:09:00.000-08:00

There's an underlying theme that I have yet to write about. No, it's not the memory manager. We'll get to that shortly. What I'm talking about here is KISS. For those unfamiliar with the acronym, it stands for Keep It Simple, Stupid. While it may be taken as a slight, it's really good advice. When given choices of solutions, choose the simpler. It's easier to code and debug. It's easier to maintain. It's faster to develop, and so on. Unfortunately, we seem to forget this, and create huge, difficult to understand and maintain programs, especially in the operating system space.

A good example of this is Linux. I know this is heresy, but how many people really understand the current kernel? I haven't looked at it since 2.6.10, and I know that there have been changes, so that I can't make the claim unless I go back and spend several hours examining the changes. I'd like to avoid it here. I know that if this operating system gains any popularity whatsoever, it will be difficult to avoid bloat, but I'm going to try. Now, maybe, you'll understand the choices behind the memory management.

Prima's Memory Management Design

Previously, we examined the region abstraction that we'll use to encapsulate the underlying hardware details. We haven't looked at the bigger picture, which is managing the processor's memory space in general. For this, we'll fall back to DOS. While there are many people who will pooh pooh this, the methodology is a simple linked list management system. A processor page of 16 bytes is used, thanks to the relationship of segments to memory, and it can be allocated and freed with a minimal amount of code. Seems to me that it meets all the requirements of a KISS design.

DOS-C/FreeDOS isn't the only operating system to use a linked list for free memory management. There's a similar system used in Minix and others. For Prima, we'll use the original DOS-C code, with some changes. For example, the original code used paragraph offsets. This was by design, and made to match the MS-DOS design. I did not make the original design decisions, but the use of a 16 byte paragraph offset allows the os designers to perform simple increments and decrements of the segment portion of the far pointer to move about memory in a rapid fashion. For Prima, a similar choice should be a 4096 byte page. This will help keep memory manipulation in check with smaller size integers for use in computations. It also allows a simpler migration to paging for future releases. That size should also be configurable for future ports to other platforms.

Another adaptation needed to my original code is to the functions and macros designed to adjust the segment:offset addresses to C far pointers. Since we no longer work in a segmented architecture, that is no longer necessary. The reference type will take care of the physical/logical conversion and there is no far attribute in C outside real mode 8086 compilers. What is necessary is a group of support functions to move data to/from user space and between different users spaces. These functions are in assembler and reside in lowcore.asm. Between this and the free space linked list, we should have everything we need to manage our physical memory.

What will significantly differ from DOS-C is freeing of memory. In DOS, the processes are also in a linked list, and freeing process memory is as simple as acquiring that pointer and calling the correct routine to free memory. That can't happen here because of our region abstraction. Instead, we'll pass the region handle back to the region code and expect back a linked list of deallocated memory. We'll then go through the list and free each piece of the list. This is necessary because when we move to paging, the logical, linear memory space may be highly fragmented pages. We cannot assume that the pages will be consecutive. So we must ask the region management code to convert it for us so we can then manage the free space normally.

Final Notes

What I did not mention is that a rudimentary process structure will be necessary to continue this work. I need it so that I can test the region allocation and deallocation code. I also need to create a user space to move data to and from in order to test the support code. I have some ideas, but I'll reserve that design for a later discussion.

You may also be interested in the current status of the code. I just completed converting the thread code from kernel threads to a more generic threading mechanism that will work for kernel and user space. I've done some testing on the code, but without a user space I cannot test the user space aspect of it. Again, another reason to create a rudimentary process. Alas, a complication typical in many projects. However, the progress I am making is good and I'm looking forward to completing the proof of concept release. I will make full use of the early process code to test ideas for the final code.

Prima Needs a Target

2007-12-13T17:12:00.000-08:00

I've been giving considerable thought to the design of Prima. There are a few design aspects that need to be considered such as processes and threads, memory management, and other details critical to an operating system before you can begin coding. While it may be fun, I want to begin testing some of the ideas I have with respect to these topics.

Adding Functionality

It makes sense to set a target in order to test these new designs. While I'm not ready to make a formal release, I do think it is prudent to set a short term target that can readily be handled as a small project. The definition of this new release is simple: it will be a new proof of concept release with a specific capability to run a single binary. I want to keep it simple so I won't select a specific file format because I am still undecided as to the final goal. I would like to use FreeDOS as a platform for development and djgpp is the best choice for that. In that case, the djgpp coff format would make a good choice. If I decide to support win32 in any way, then PE make sense. Of course, there's always elf with its large base of support in the open source community. So for now, it will be a simple stripped binary image, somewhat of a 32-bit "com" file. This is great, but not something I want to keep for the future.

The Project

I now have the beginning of a project: a defined, measurable target. I still need to do more planning, but this is a small project whose goal is to achieve a step to the final goal of a stand alone operating system. As strange as this may seem, this is my preferred way to handle roadmaps. I like to create individual projects that meet each of the roadmap targets and "phases" that create small, achievable steps along the path to the final goal. This is one of those phases. While the roadmap may not be clearly defined, I know one target and this is one of those phases. I need to have in place processes and their relationships to threads and regions. This is how I will do it.

Acknowledgement

I do want to thank Christopher Giese who runs a web site with lots of handy little snippets of code. You can find it at http://my.execpc.com/~geezer/osd/index.htm and it's worth taking a look at. I've borrowed a couple of his code snippets, such as the interrupt handling, but I have adapted the concepts to fit my generic design. For example, I have my own getvect() and setvect() functions modeled after the DOS versions. I also have newer functions that eliminate the passing of the flags. Also, while Chris looks at tasks, my design is thread based, an update of the original task based design from earlier multitasking operating systems I developed.

If you want a good reference to start your own OS coding, I definitely recommend spending some time at this site.

Prima Design Decisions: Regions

2007-12-05T05:36:00.000-08:00

As you may have noticed from earlier articles, I am somewhat compulsive about portability. Of course, this can be difficult to achieve in a kernel. The kernel is where you get your hands dirty. It is where details such as pages, bits and special registers must all be dealt with.

Fortunately, a microkernel-like design, if not a true microkernel, allows a significant advantage because we can partition our code so that only the kernel itself is aware of these details. The rest of the operating system can access these dirty bits and bytes through abstractions. That's where regions come in.

For Prima, I've decided to model core as regions. Each region has a property of size, virtual start address, and granularity. We can allocate and dispose of regions, and resize them. We will also be able to assign some sort of signaling to each regions so that whenever some sort of need arises to signal a process or thread, the kernel can do so. What we get back is an identifier or "handle" that we can pass to the kernel to allow it to operate on that piece of memory.

There will also be system calls that allow you to transfer data into and out of any region. This is to allow me to avoid one of the costly operations seen in some microkernels, the copying of data in to and/or out of messages. I would much rather hand off a region handle and virtual address to read from or write to in a message than copy the data itself into and out of the message.

Of course, the devil is in the details, and I'm not sure of all those details. But, the design makes sense from an architecture standpoint, regardless of whether or not the design becomes a hypevisor or just a typical, more mundane microkernel.

Of course, this is just one of many design decisions that must be made during the design of an operating system, so don't be surprised if you see something detailed later on that doesn't entirely agree with this. Any complex software architecture will need to be adjusted here and there as the whole design solidifies.

Expect to see some aspect of regions in the next proof of concept release.

Prima: Moving Past DOS-C

2007-12-02T10:22:00.000-08:00

Funny Story

When I first wrote the operating system that was to become FreeDOS, I really didn’t know what to call it. The commercial software that spawned it was known as DOS/NT, but I wanted to call this version something different. I couldn’t decide what I wanted to call it, so it went unnamed for a very long time. I did promote it as the shareware version of DOS/NT, so in that respect it is part of the old DOS/NT family, but it had no official name.

Along comes James Hall. I’m reading usenet and I see his PD DOS posting. At first, I shrugged it off, but I later changed my mind. I wanted this piece of code to have some purpose. I then contacted Jim, and offered him that operating system. The remainder of this story is well known and is documented on Wikipedia and other sites.

What About Now

Well, I recently became restless. I had not done any operating system work for a few years, either personally or professionally, and while VoIP is interesting, it’s not the same as a kernel. I thought of ways to advance FreeDOS by extending the kernel to 32- and 64-bit. In what I had in mind, the kernel is a new one in which the user would not know the difference. The user would run all his/her applications as always, but new applications would take advantage of newer features. The user would also gain the ability to run new devices through new device drivers.

So I went back and tried to gauge interest on the FreeDOS mailing list. I published a little poll, and what I got back didn’t really match up with what I wanted to do. I wanted to move the DOS paradigm forward. Most of the responders questioned the need to do this. They’re satisfied with running FreeDOS under some sort of virtualization, not as their primary operating system.

What they’d like:

Better device driver support
Better connectivity via TCP/IP
Better support for modern platforms, such as ACPI and other power management functionality available on modern motherboards
Some support for new file systems.

Another FreeDOS developer list question asked whether or not FreeDOS could run on ARM. The responses, of course, indicated that you couldn’t. While this is true of the current distribution, the original source code was portable and the new kernel would certainly allow such a port. Even the last release of DOS-C is relatively portable, although some small changes are necessary. Surprisingly, this was the only embedded reference I got back.

Announcing Prima

Prima is the new operating system I am putting together. It is an extension of my original DOS-C, so in some respects it is a branch of the current FreeDOS project. Funny thing is, I have not totally defined the project. I may or may not want to support 16-bit applications. If current DOS users are content with what they have and use other operating systems for their day to day activities, this certainly won’t be something they want to do. I may or may not want to support a POSIX API. There are many very good POSIX based operating systems around, free such as *BSD and Linux, and commercial such as MacOSX and Solaris. I may or may not want to support win32. This would be a huge undertaking.

What this is a proof of concept release. I am taking the original DOS-C code and putting it into a stand alone application that allows me to boot it up and test it. It is a test of the compatibility of the old code with modern compilers. It is also a launching point for the new project.

If you examine the code, you’ll notice some code that was never before released. I am placing a messaging system into this project. I am contemplating a design somewhere between microkernel and virtualization. I like the concept of a microkernel and want to take advantage of the design principals in this project. You’ll also notice a minor shift from the current DOS device driver design. Since there are very few new DOS device drivers these days, I don’t see any advantage in trying to access real mode drivers. I may want to reconsider this at a future date if there is a need to run old device drivers.

For now, here is the proof of concept. Next, I’ll implement a set of coding guidelines, followed by new portability guidelines. By that time, I’ll have decided on a direction and defined the first project for Prima. In the mean time, you can get the source code from ftp://opensourcedepot.com/Prima/Prima-POC-071202.zip.

Enjoy.

FreeDOS: Lost in the Sauce

2007-11-22T04:03:00.000-08:00

For all US readers, Happy Thanksgiving. For non US readers, today is a day in which tradition dictates that you get together with family, and stuff your face. You eat turkey, stuffing, trimmings and pie. Tradition also dictates that you do nothing but eat left overs for several days afterwards. And let's not forget "Black Friday," the day after Thanksgiving where everybody runs out and does their Holiday shopping, because Thanksgiving is traditionally the start of the "Holiday Season." This is signaled by the traditional Thanksgiving Day Parade where Santa Claus comes down and greets all the parade watchers after many floats and balloons of popular cartoon and comic figures, as well as other pop culture icons.

Fact is that Thanksgiving is a day of giving thanks. The original settlers started this in 1619 to thank God for their arrival to the new colonies where they could enjoy freedom unavailable to them in their original homeland. Fact is that in the nearly four centuries that have followed this first day of giving thanks, the original meaning of the celebration has become somewhat obscured.

Seems that the same is true of FreeDOS. For those readers who do not know who I am, FreeDOS began to take shape in 1994 after I donated a version of my original commercial operating system to the project. The first version of FreeDOS was quite literally written by me. It consisted of the kernel, a loader subsystem that was derived from a multiboot loader used in the commercial version, a command.com shell and a sys program to make bootable disks. The commercial version was portable among many different platforms, most notably 8086, 80386/486 and 680X0 processors. A lot of care went into the design of this commercial operating system which was used either in part or in whole in various applications such as FAA Flight Inspection systems, telephone switching systems and other embedded applications.

Over the years since that first code was released, many more of what I had termed "MS-DOS Warts" were written into the operating system, my command.com was abandoned in favor of another version that was written from scratch by other project members, and many new device drivers and utilities were added to make it not only a true MS-DOS look alike but extends it with a distribution that truly makes it a usable operating system for small systems. Unfortunately, it is no longer portable as much of the code base was either changed to save 3 bytes here and there, processor specific code is no longer contained in a few files, and special types defined in portab.h meant to make the code portable are either improperly used or not used at all.

Unfortunately, this makes moving the design forward to new platforms very difficult, if not nearly impossible. There is no way that you can simply recompile many of the files and expect it to operate properly on another platform, much less a non-Intel processor. I'm not saying you can't back out some changes and rewrite others. What I am saying is that it is a much bigger project than I had hoped for.

Here's some insight into the original design: the original FreeDOS kernel was an embedded operating system kernel surrounded by two compatability layers. The upper most layer implemented the DOS API and the lower layer was the machine interface, which included processor support code as well as virtual machine (VM) and I/O support. The embedded OS was highly influenced by Unix, as is apparent by the use of an fnode, which was modeled after the the Unix inode. Same is true of separate block and character I/O, also part of the design of Unix V7. The VM support was replaced by macros MK_FP and friends, although some of the original code such as fbcopy() remained. This was because the original memory manager was replaced in the FreeDOS version of DOS-C by one that implemented the MS-DOS linked list. The use of weird macros COUNT, BYTE, etc., were meant to make various types transparent, allowing me to keep objects such as bytes, signed and unsigned 16 bit integers, etc., straight across target processors.

Now, I don't want you to walk away from this thinking that I am criticizing what Jim Hall and the FreeDOS team have done since I stopped actively participating. I just wanted to shed some insight into where I have been and where I'm heading next. I will have to make certain decisions, such as whether or not to use any or all of the FreeDOS code in my new project. It may be easier to just use my original code and write extensions to handle virtual DOS tasks within the new OS. I may also go back to a POSIX compatability layer instead of DOS, given some recent discussions I've had with folks about FreeDOS and the availability of new DOS applications, but that's material suitable for another article. Suffice it to say that I found it interesting how, over the years, some of the original design goals were "lost in the sauce," just like the meaning of Thanksgiving. Now let me take the dog for a walk because the Thanksgiving Day Parade will be on TV shortly and I'd like to catch some of it before heading out to my sister-in-law's house for Thanksgiving Dinner.

Start of something new

2007-11-20T16:51:00.000-08:00

It's been a while since I've done anything different here at Open Source Depot. It's not that I haven't wanted to, but rather that it seemed that I never really had the time. Since I started the site, I worked on a couple of variations of my original DOS-C that were intended for release as open source, but never got released. I also ported a version of AT&T Unix V32, a 32-bit variant of the famous v7 version. Early versions of that one are available from the ftp area, but the latest version is not. There's also a port of the ACK C compiler. As I stated earlier, I've been busy; just not communicating.

I want to change all that because of some recent developments. I conducted a short poll on the FreeDOS developer mailing list to help me decide what to do with my work on kernel internals. I had been thinking about trying to create a new DOS kernel, one that would allow DOS to live on, but the results of the poll were somewhat inconclusive. Well, at least that's my opinion. I don't necessarily want to continue down that path.

I will decide which way I'm going shortly. I will either continue to to work on my Unix port, or I'll take some of that work and create a new DOS kernel that extends into 32- and 64-bit processors. When I do decide, you'll read it here and be able to follow the development.