PalmOS on a 14″ PDA

A long time ago I talked on IRC to someone who seemed to have a lot of design-level knowledge on PalmOS despite being totally clueless about programming. He claimed that PalmOS kernel was really a true multitasking OS’es kernel with lots of features like desktop OS kernels have and that it was a descendant of some other proprietary OS, of which I think this person was a great fan (funny how every, even the dumbest of OSes must have a sort of cult around them). But as far I can tell this person was right and PalmOS really has a full-blown kernel, making it a royal PITA to debug or disassemble. Despite the complete OS being so dumb and looking even dumber than it is, in the kernel there’s a scheduler, there are mutexes, semaphores, completions, locks, memory pools and other nasties. Figuring out what a process is waiting for in a given moment is very difficult.

How do we know there are semaphores etc.? This is the cool part: with a little scripting you can get a full trace of all instructions that are executed and see what functions are being called. That wouldn’t tell you much if not for the PalmOS symbol tables that were leaked a couple of years ago (not sure what they are really, it’s like a long list of all standard calls with only the name and number for every syscall and it applies to every device because there’s a standard way of invoking the syscalls across all PalmOS versions on ARM. There are only a few device-specific calls). How they leaked was also an interesting story that I was once told by someone but don’t remember well, but the plot involved elements like spies, a microfiche, Russia (maybe not exactly, but close). Here’s a heavily stripped sample of the execution tree cut out of a PalmOS run with the symbols inserted where appropriate, and non-branch instructions removed (the program that generated the tree got it slightly wrong because it assumes that the log begins at reboot, not somewhere in the middle – the whole trace would count in terabytes – according to the script there were 1172429829 instruction executed in the boot-up, but only 96183 unique, i.e. 96183 different r15 values – which is not to say that all the 117… were loops – the script detects loops and “rolls” them back into a loop form). You can see calls like WinPaintRectangle towards the bottom.

Another interesting detail about PalmOS is in the UI it uses DMA to draw solid rectangles and other shapes. For example the boot-up animated logo is all done by the DMA and the CPU only waits for the DMA to finish between calls (imho defeating the whole point of using DMA, but it’s still an interesting idea). The animation is visible in the emulator screen and you can even appreciate some artistic details that are not seen on the real hardware, or I didn’t notice them before. Everyone knows that the animated Tungsten logo is the only reason people still use PalmOS.

To draw a rectangle it sets the DMA frame size to be the width of the rectangle in bytes and sets the frame index to be the difference between the addresses in the framebuffer of the right edge pixel of a rectangle and the left edge pixel on the next row. The channel’s source is constant-addressed and points to the variable that holds the pixel’s colour value.

I still haven’t had much time to play with the stuff. The most tricky information we can get from PalmOS is probably what it does during suspend/resume because that’s what’s difficult to get right without documentation, and even with it. Linux battery-life-in-suspend is currently drastically shorter on this device than in PalmOS.

For some reason during boot-up, its kernel also reads some three values from a place in memory that’s part of Flash chip-select 2 area, and afaict there is no chip there. I made some tests on the real hardware, from under Linux and I couldn’t read any sane values from there, but I will keep looking. It would be nice if there were more peripherals connected that we didn’t know about. The way MMU is set up by PalmOS also suggests that there’s something there. (Maybe left over from an older devince or from a development ROM.)

That’s about it. First thing I did after PalmOS booted successfully, which was about 3am Monday two weeks ago, I started looking at how PalmOS manages the MMU tables because it appears to perform some funny MMU-acrobatics when it boots,.. but I was interrupted by the Second thing.

Second thing I did after PalmOS booted successfully was throwing stuff into my backpack in a great hurry and getting the hell out of my home and running to the airport where I planned to be at 4am to catch the 6am flight to Birmingham for GUADEC 2007 where I spent that whole week. The flight was, of course, delayed by some 4 hours and I only got some sleep after finally getting onboard, and then later after a nice dinner with Nokia and the rest of OpenedHand.

The OMAP emulation has landed

About a month ago I started a new project on my harddisk, that was supposed to, as usually, solve all existing and non-existent problems (of porting Linux to new, undocumented devices).  But, as usually, I got sidetracked in the middle of the solving of all problems, and it lost priority.  I should probably again appologise to everyone whose all problems will not be solved ;) – sorry I’m too busy with other stuff.

I got sidetracked by an (of course) brilliant idea, that I had on Friday or Saturday two weeks ago, just before GUADEC, about why my qemu-based emulator for the Palm Tungsten|E never worked (kindof).  Palm Tungsten|E is a PDA with an OMAP series cpu in it and the hard part is emulating this OMAP chip. The emulator was an idea I started implementing like one and a half years ago in order to run the PalmOS binary that ships with the device and then spy on what it’s doing to the hardware and this way learn about the hardware itself.  Unfortunately PalmOS never booted in the emulator, until recently, even though Linux booted just fine, but that’s a whole lot less useful.  Nevertheless the time I spent on it was not all lost – applying structured procrastination (cool name I learnt about recently) I always avoided coming close to the proper PalmOS and was rather starting all the time new projects of automated disassemblers, debuggers and other funny names, that would partially do the task of getting PalmOS to run for me, some of these programs may turn useful some time and even if not, I used them as my university projects for some two courses and actually tricked the lecturer into believing the code did something useful (haha).

Back to the “brilliant idea”, it turned out to work and after two days of almost continuous coding, PalmOS booted and seemed to work really nice.  It took a couple of horrible hacks that I’m not even sure why they might be needed. In some places I totally raped the ARM specs just to keep PalmOS happy. In addition, the source tree was based on a prehistoric qemu snapshot and now that the basic objective – booting PalmOS – is reached it will take some effort to port all of the code to the current tree and try to make it usable.  I already started and today committed to CVS an initial Palm Tungsten|E / OMAP310 support that boots Linux but doesn’t do anything more.  So far it was more like a rewrite because my old code was so bad :(  I didn’t have time to play with the PalmOS yet but I will post later about whatever interesting could be observed so far.

ALP ctd.

I’m spending this week at GUADEC 07 in Birmingham, UK and it’s great fun so far. The theme seems to be in a big part about two things (not counting GNOME, Linux, etc.): desktop web integration and mobile Linux. One of the presentation explained that with a slide that said “Go where the money is: 1. Web, 2. Mobile”. Unfortunately a lot of the talks, especially the keynotes, talk about money-oriented development being the best thing, complying with the needs of corporate users, competing with ms-windows, and even saying deadlines help you make better code (that maybe actually true but it’s so pragmatic and put in my face). Oh well, probably that’s normal in the desktop development world, which I’m not so much into. Everything’s pragmatic here.

In the same spirit was a talk about Hiker & ALP that the ACCESS Systems people had yesterday. The presentation gave some new insight into the whole ALP thing, but not a terrible lot of it. The Hiker guys had working demos of ALP simulator (basically Xephyr) on their laptops and the presentation had a couple of screenshots.

• The look: ALP looks pretty much like PalmOS with a new, blue theme (default; there’s also a theme called Pony). The UI ideas are also taken from PalmOS but that’s probably an advantage for ALP, because that’s what PalmOS did right. The UI integrates nicely with GTK, has keypad navigation across all widgets that are on screen, and is closed-source.
• The openness: as could be expected, ALP is horribly littered with proprietary software, in fact you could say it’s a whole new proprietary OS with the Linux kernel and some free projects unnaturally fitted into it, and the Hiker part open-sourced in order to keep appearance.
• Hiker is not necessarily evil but imho it’s also not a piece of software that solves any current or serious problems of mobile computing, it’s just an alternative to a couple of other opensource frameworks for mobile apps (fortunately the Hiker authors made impression of being aware of that).
• How does it compare to them? It probably stresses security more than the other projects and has this part covered more extensively (at least the presentation had). The approach to security is definitely original but it’s hard for me to tell if it’s totally correct. The idea is that the system assumes there’s only a single user (which is not a terribly bad assumption and the existing competing mobile distros also have this assumption in a lot of places), and since then the UNIX user and group IDs would be practically unused, they are instead used for process control, i.e. different programs run with different UIDs/GIDs and that allows for easy access control like allowing different apps to use different sets of resources (like /dev nodes, iptables) and preventing e.g. malware from spreading this way. It’s cool that they thought about malware but is this approach ok? UID stands for user id after all. They mentioned a kernel patch they had sent to mainline, related to this approach, which wasn’t accepted, so maybe that’s why. One thing that’s surely positive for Hiker is the clearly defined (ripped off PalmOS) applications behaviour policy (e.g. ensuring only a single instance of most normal apps at a time, etc.), this is where OpenMoko is probably lacking.
• How does the rest of ALP compare? ALP looks much like a rewrite of PalmOS but this time based on a Linux kernel and other pieces of modern software. Unfortunately ALP is currently not yet even at the point where PalmOS left off, the implementation is just slowly caching up. It’s not any more open than PalmOS was and ACCESS is willing to go for all kind of compromises with hardware vendors and implement all those lock-ins whose avoiding is one of the objectives of a Free Phone (OpenMoko). They will also do a lot to promote their stuff, as is shown by the leaflets they handed out at their stands, full of typical marketting babble except in some places being completely untrue, rather than unclear.

There was some talk about allowing only certified apps to be used on ALP phones, depending on what the particular phone vendor decides.

The .prc loader depends on something called GarnetVM which is completely proprietary and which also in a way depends on Hiker, so probably quite useless for reusing on other devices. I was told it does load native ARM .prc’s as well as the m68k ones, although the Hiker people were not satisfied by the current compatibility level. They said it probably should run “properly written” PalmOS apps without modification, so this can mean many things. There is no connection that they know of, between their .prc loader and Palm Foleo .prc loader and compatibility layer, or any other parts of the Palm Foleo software.

ALP has the arguable advantage that it’s written with ease of making 3rd party apps in mind. There’s a set of Eclipse + CDT templates for ALP apps for everyone’s use (though they are making it sound like a whole new development environment based on opensource solutions, whenever they can) and the simulator thing (Xephyr). There’s obviously no hardware level emulation because they are not a hardware company and they don’t know what hardware ACCESS Linux will first be shipped with yet. They are looking for proposals. I saw one development phone that they use, it was a PXA running at about 300 MHz, CF and miniSD slots, 64 MB of RAM and 64 MB of Flash, a nice white case. If I guessed correctly, it had two cameras, one facing the user and one on the other side. The camera is supported under ALP. It had no touchscreen.

A Maemo person asked if they have had problems with startup times on ALP because that’s one of the things Maemo earlier struggled with and they confirmed they did and still do have problems.

It was said that they talked to Mickeyl about using Hiker in OpenMoko and Mickey liked the idea. I’m pretty sure that won’t happen before the Neo1973 release because Hiker seems to be currently even less complete than OpenMoko, also I can’t see any problem that switching to Hiker might possibly solve. Note, however, that the 0.9.1 tarball I downloaded from their site before GUADEC, was mostly a 2006 code and there must have been some progress since then. (This also shows that it’s not a normal open project because the development was going on in some internal repositories except the two released tarballs. The whole developer network (ADN) for which you have to register, also hints on palmsource.com, msdn and the likes).

Since yesterday Hiker is double-licensed under MPL and LGPL, fact that was announced during the talk.

ALP status

I visited the website of ACCESS Co., a company that I’ve always only heard of in connection with the unfulfilled hope of PalmOne’s PDA users, the permanently soon-to-be-released Access Linux Platform (ALP) – to see if they have abandoned it or not yet. The status is that they haven’t or at least are not admitting it, currently the FAQ says Q1 of 2007 and the whitepaper says Q3. Here’s what I could gather.

They already released something called “application framework” for ALP, a project named Hiker. Current Hiker version is 0.9.1 and the 0.9 sources can be downloaded too, both are 2006. It is a 3.5 MB tarball mostly under MPL, containing: a short doc about building the thing and an ALP whitepaper PDF, code in C partially something that may be an early “application framework” and partially lots of empty stub functions, a reasonably small amount of C++ code (unit tests) and doxygen generated docs for that – not informative. Not all of ALP is going to be opensource and there’s no information on the status of the other parts. Of the pretty “software stack” diagram found in the docs, around.. one eighth? of the bubbles is orange, designating opensource parts in ALP. About half of the diagram surface is covered by popular FOSS project names, so that’s not so bad. The build system for Hiker is autotools based and the whole package has a few dependencies like sqlite, glib, dbus.

What’s that application framework? It is supposed to be a set of libs including UI (this part is not present at all in current version), IPC, security (these two constitute most of the tarball), settings registry, etc, divided into daemons with names known from PalmOS (Application Manager, Attention Manager…). I found no hints about how the UI will look, the UI toolkit is called MAX. Apps running on ALP can be: MAX native, GTK+ native, PalmOS m68k .prc’s (what about ARM?) or Java. The whitepapaer admits many UI ideas are taken straight from PalmOS, hopefully the good ones. Hiker doesn’t depend on GTK, but there’s some code that uses GTK, a thing called Native Process Launchpad Daemon.

Apart from the system libs ALP is supposed to manage packaging, packages are called bundles. Looking at the code, bundles appear to be cramfs images with names ending in .bar, and containing one obligatory XML file describing the bundle. Two alternative formats (schemes) are allowed: ghost (?) and java .jar/.jad’s for which the Manifest.xml is generated on the fly. Installation involves loop mounting and some SQL queries. Sandboxing is mentioned too.

IPC has separate APIs for C and for C++. The whitepaper says that writing ALP apps is easy as making Java apps.

All different formatting styles are present in the C code. And here’s a sample from NLPD:

	//
	// This is just some magic I found
	// via google.  Man pages do not
	// cover it.
	//
	if (prctl( 15, NewName, 0,0,0) != 0)
		returnvalue = 1;

Dear ACCESS, prctl() has a man page on my system, and I can send it. Also, the call alp_prv_strcpy2, that is introduced “because C strings a pain in the tail”, as we are told in the comment, is already in glibc known as strndup().

I can fly?

My long awaited private pilot’s license (PPL) for paraglider finally arrived and I’m offically a pilot. It came by mail straight from… Slovakia. Why Slovakia? Turns out it’s the nearest place that has some sane law regulating the air traffic. The license is acredited by Fédération Aéronautique Internationale and comes in three documents, two of them looking really neat and one that appears to be photocopied in b/w and filled in with a true old-style typewriter plus an unreadable signature of a representative of the Letecká Amatérska Asociácia Slovenskej Republiky in Žilina. This is however totally compensated by the fact that all three papers contain both my name and surname spelt without a single typo! This is an unprecedented achievement in the field of office work and the history shall remember this moment. Hereby credits go to the Slovakian beaurocracy and, of course, to my paragliding instructor.

When I grow up I want to be a Slovakian office worker. (I bet their aeronautic association has a whole department to deal with poles going for the PPL in Slovak Rep. in face of the lack of sane regulations locally).

Unrelated to all that the license was sent to my parents’ house where I went by bike today and when I was coming back a storm was just starting. Let’s just say I had never seen the average of people on the bikeway speeding that fast on their bikes. It’s normally full of annoyingly slow and lazy bikers of all ages. Now, right before the storm everyone, including the moms with children, was beating the shit out of their machines, and it was a real delight. I took the bike path that goes through siekierkowski bridge over Vistula and there’s a place in the middle of the way where there’s no civilisation in at least 10km in any direction and you don’t want to be there when the storm starts. Btw the path connects two slightly suburban but important Warsaw districts and it is most of the way in a nice forest with the bridge in the middle and the view of the city was great from there, especially during the rainstorm on an otherwise sunny day (you could say it almost looked good, which is impressive considering that it’s known as the ugliest city in this part of Solar System).

OpenMoko to start taking orders

So around this week OpenMoko is supposed to start selling the Neo1973 developer preview (aka. Phase 1) devices. It’s hard to say if they will make it this week but either way it shouldn’t take very long now.

It just so happens there is a Phase 1 Neo1973 phone sitting on my desk and I have a couple of comments on it for those who are planning to get one for themselves too. Firstly, at least one person expressed amazement about the fact that now, three days before shipping, things like SMS support are being worked on and are nowhere near finished on the software side. Apparently it wasn’t made clear enough that this Neo1973 release is *really* a developer version and is targeted for ambitious developers who would otherwise be disappointed if the device they get comes packed with all software ready to use. And I’m sure there will be a lot more people who didn’t get it and are going to be giving OpenMoko bad reviews and this is annoying, because what they will be reviewing is not even the OpenMoko platform yet, it’s a developer snapshot. (This is similar to those folks who beg you to let them taste what you have in your pot when you’re in the middle of cooking something, and then they’ll say something about not liking the soup or worse, about your cooking skills, based on this. The thing is they are tasting a development snapshot of the soup, not the soup. It might even be poisonous, and the final dish still be perfectly fine. Guess I’m hungry…)

Another thing, when you get your device it’s a good idea to back up the factory contents of the Flash chip so you don’t lose some of the files on it before flashing an update later. Updates will overwrite all of the data in the Flash and there is a couple of files there that are not available from internet (don’t ask me why). A simple way I found to make a backup is using netcat from under Linux after it boots for the first time. It will take about 30 minutes and here’s how it’s done: connect the device to your PC with a USB cable or Bluetooth and run the following spell on the PC side:

$ for i in 0 1 2 3 4; do netcat -l -p 20000 > mtdblock$i; done

Netcat will listen for the backup data on TCP and write the five chunks (Flash partitions) to files named mtblockN in the current directory. Log into the phone through ssh and do:

# for i in /dev/mtdblock?; do cat $i | nc 192.168.0.200 20000; done

These files should be enough to restore the original state of the phone’s memory at any later time.

One more comment I have is that with the current Neo1973 kernel it doesn’t make much sense suspending and waking the system up when you’re coding something on the PC and are periodically testing it on the phone. Leave it running, power the device off entirely after you’ve finished. There’s something seriously wrong with power usage in suspend/resume. I have not made any measures and the battery I’m using is not 100% compatible but I risk to say it draws more power when suspended than when fully on. There is a curiously looking patch on the mailing lists that might explain this but I haven’t tried it yet. On the other hand I²C peripherals alone shouldn’t have that much impact.

For comparison, a different ARM-based device I own can live ~4 hours powered on and over two months suspended (i.e. with only RAM constantly on), from a full battery. The Neo will have the additional sucking from the GSM modem, but it didn’t seem to make much difference when I powered the GSM off through GPIO (assuming I did it correctly).

The thrill of jee-pee-el vee-three’ing my files

Today I was adding new files to a new project (more information when it starts to work, which is hypothetically possible). I decided to try GNU GPL v3 this time, to see what will happen. I glanced around to make sure nobody was watching, quietly typed http://gplv3.fsf.org/ into the browser, quickly copied the necessary paragraph and put it in the headers of two or three files. Already with shaky hands I saved the changes, and.. guess what? Nothing happened!

Screw this.