Are you annoyed by Twitter’s grammatically incorrect “Who to follow”? Here is a bookmarklet for you: javascript:$(‘#global-nav-whotofollow%20a’).text(‘Whom%20To%20Follow’);$(‘a[href=/#!/who_to_follow/suggestions]%20span’).text(‘Whom%20to%20follow’);undefined
It’s been quite a while since I released duckprxy, the duck typing (well, as close as it gets) library for Java.
I had a bug fix that I wanted to release, but never got around to do it. The bug fix was sitting there uncomitted (I don’t really know why) and now I forgot what it really fixed. Anyway, all the tests (including some new test) pass. So I guess it works correctly.
Anyway, I pushed everything to github now: haselbach/duckprxy.
Maybe I’ll bundle this up to a new release, but I don’t think there are a lot users (read: probably none) out there, so I don’t think it is worth the effort.
Duckprxy has been mentioned on discussions on sites like Stackoverflow. So having the source out there and linkable might not be a complete waste.
Do you want to know more about duckprxy or have feature requests? Then let me know c-duckprxy@mr-co.de.
I’m not a big blogger. I just write a post every few month about stuff that I don’t want to forget. I did not want to maintain my own blog software installation any more. So I’m now transitioning away from Cyrus Harmon’s nuclblog to a hosted solution: tumblr.
I’ll move over my old blog posts within the next few days and set up redirects.
Here is my first post from the old blog (originally published Sun, 27 Apr 2008 21:10:13 GMT):
So finally I got some blogging software installed. It had to be, of course, a Lisp based system. Cyrus Harmon’s nuclblog seems to be good. And it was easy to install. So let the blogging begin.
This weekend I wanted to update my FreeBSD box and saw that the ZFS tools fundamentally changed. The UPDATING file has a corresponding note:
20090520:
Update ZFS to version 13. ZFS users will need to re-build
and install both kernel and world simultaneously in order
for the ZFS tools to work. Existing pools will continue to work
without upgrade. If a pool is upgraded it will no longer be
usable by older kernel revs. ZFS send / recv between
pool version 6 and pool version 13 is not supported.
Installing kernel and world simultaneously is scary. If something breaks you are left with a broken system, no way to roll back, and uncertain repair possibilities. Sure, you should have backups, but restoring a whole system is still something that takes a lot of time.
But is this not exactly one of ZFS’ strength? ZFS provides easy snapshots and clones. When I search for “FreeBSD ZFS update” I stumbled of this Mail: Solaris live upgrade like FreeBSD ZFS-rootfs update, which gave me the idea to snapshot my root and usr file system. This should give me a reasonably easy rollback path.
Here is what I did (with some minor tweaks). It might or might not work for you. You might loose all your data following my steps. I cannot be held reliable for that. I also probably made a few mistakes while writing this up. Back up your system!
I have my FreeBSD with encrypted ZFS set-up like I described in my blog post FreeBSD: Encrypted ZFS Root with Geli. It should also work with a different encryption mechanism or without encryption.
Let us assume that the ZFS pool is called tank and that the root file system is under tank/root and the usr file system is under tank/usr.
First, let us create a snapshot of the usr file system and of the root file system:
$ sudo zfs snapshot tank/usr@pre_update $ sudo zfs snapshot tank/root@pre_update
A snapshot is not writable. So let us create clones:
$ sudo zfs clone tank/usr@pre_update tank/usr_pre_update $ sudo zfs clone tank/root@pre_update tank/root_pre_update
We need two versions of the root file system. The normal one (tank/usr) where usr points to the normal tank/usr and the pre update one (tank/root_pre_update) that points to the pre update snapshot tank/usr_pre_update. We will use a symlink for this.
Unfortunately, you cannot change the mountpoint of your usr file system while the system is running. So let us first finalize and boot into the pre update set-up to see that it is working.
$ echo Temporary mounting the pre update root $ sudo mdkir -p /pre_update/root $ sudo zfs set mountpoint=/pre_update/root tank/root_pre_update $ echo Setting the mountpoint of the pre update usr file system $ sudo zfs set mountpoint=/usr_pre_update tank/usr_pre_update $ echo Symlinking usr to the pre update usr file system $ cd /pre_update/root $ sudo rmdir usr $ sudo ln -s usr_pre_update usr $ echo Setting the pre update root mountpoint to legacy $ sudo zfs set mountpoint=legacy tank/root_pre_update
We need a similar set-up in the normal root file system where the symlink points to usr_post_update. In order to set the mountpoint of tank/usr to /usr_post/update we need to reboot into single user mode with ZFS disabled. We go to the boot loader prompt (normally by choosing 6 in the boot menu).
# unset vfs.root.mountfrom # disable-module zfs # set boot_single # boot
Once we are in single user mode, we can import zpool and set the mountpoint.
# zpool import -d /boot/zfs # zpool import -f -R /tank tank # cd /tank/root # rmdir usr # ln -s usr_post_update usr # zfs set mountpoint=/usr_post_update tank/usr_post_update
Now we reboot the system into the pre update environment. We go to the boot loader prompt and change the root file system to the pre update root file system.
# set vfs.root.mountfrom=zfs:tank/root_pre_update # boot
If everything goes well, the machine boots up into the cloned environment. We can verify this by looking that /usr points to /usr_pre_update.
Let us reboot again into the normal environment (by doing nothing special during boot-up). Now /usr should point to /usr_post_update. With this system we can build and install the world and the kernel as mentioned in the Handbook - Rebuilding “world”, without the reboot into single user mode (which will not work, anyway).
If anything goes wrong during the update, we can boot our pre update environment by going to the boot loader prompt, selecting the pre update root file system and booting the old kernel.
# set vfs.root.mountfrom=zfs:tank/root_pre_update # boot kernel.old
Here we can copy over the backup kernel to correct destination and roll back to the snapshot with the comfort of the system in a fully functional state that we know.
Should the update work out, we can destroy the pre update environment in order to safe disk space. Or we can leave it around to mitigate problems of the update we are not aware of just yet.
The other day, I read the article How to Write an Equality Method in Java. Which reminded me that I wanted to write an entry about equality in programming. I cannot hope to cover this topic adequately in a blog post. My goal is merely to show that equality is very hard and that we often have a sever misconception about it.
Most programming languages give us several ways to check equality. Java has the operator == to test equality on primitives and references and the method boolean Object.equals(Object o) to test semantic equality of objects (we will discuss later what that means). Common Lisp has a lot more (eq, eql, equal, equalp, =, string=, …). And did you know about the === operator in Javascript?
What does it mean that two Objects are the same? In the OO world a naive approach is to say, that two objects are the same when they are instances of the same class and all members are the same. This is a nice recursive approach, and it is also not to hard to create a terminating implementation, if you take a bit care of circular references.
However, let us consider the numerical tower. Is the integer 2 the same as the float 2.0? If you interpret class in a Java sense, then they are not the same according to the above definition, because they are not instances of the same class. You might argue that integers can be seen as a super-class of floats (meaning that integers are a subset of floats, i.e., every integer is a float). So you could let the more specialized class decide the equality.
This approach does not help us with a different problem: Two strings, according to our definition, are equal if both strings consist of the same characters (in the same order). So the strings “FooBar” and “foobar” are not equal. But what if we do not care fore case sensitivity, e.g. because our file system does not care for it? Building up a set with all file names is a reasonable use-case. A standard Java HashSet<String> would not be very helpful here.
As a matter of fact, the Java String acknowledges this problem and provides equalsIgnoreCase(), but I cannot tell HashSet to use this method instead of equals() (without extending String).
Object relational mappers (ORM) have a similar, but slightly different, problem. They have to make sure that objects in the data base and in the program are matched up correctly. Here the instances of the class Person with name Maria Smith and Maria Jones might be the same object, just at a different point of time. This is usually mitigated by surrogate keys. So these objects are the same with respect to their ID (the surrogate key), but not with respect to the data they hold. A different distinction for the ORM system when it needs to synchronize objects with the persistency context.
The equals() methods should of course implement an equivalence relation. However, very often they do not. An equivalence relation is reflexive (a = a), symmetrical (if a = b, then b = a), and transitive (if a = b and b = c, then a = c). It is quite easy to break the symmetry the class structure is not considered correctly. Let us assume you have a class A and a class B that is a subclass of A (i.e., it extends A in Java).
class A {
int x,y;
boolean equals(Object o) {
if (o instanceof A) {
A that = (A)o;
return that.x == this.x && that.y == this.y;
}
return false;
}
}
class B {
int z;
boolean equals(Object o) {
if (o instanceof B) {
B that = (B)o;
return that.z == this.z && super.equals(o);
}
}
If we have an instance a of A with x=1 and y=2 and an instance b of B with x=1, y=2, and z=3, then a.equals(b) will be true, but b.equals(a) will not. Just saying, only instances of the same class may be equal, is not always a viable solution, as explained above.
The real problem is, that there is more than one equivalence relation. Java (and many other OO languages) forces us to come up with one (more or less) canonical equivalence relation, rather than being able to define the real equivalence relation one currently needs.
A map implementation where you can define the equivalence relation would help. A Java interface for such a definition could look like this:
public interface Equality {
boolean equals(Object o1, Object o2);
int hash(Object o);
}
The problem is, that this equality definition needs to know about all classes whose instances might end up in the map. This problem can be solved by open classes and multiple dispatch (Java supports neither), but also by other techniques.
The article I mentioned in the first paragraph mentions defining equality on mutable fields as a pitfall. IMHO the final keyword on variables and fields is underutilized in Java (and final should be the default) but this is beside the point. However, it is not so important that the fields used in the equals method are immutable, but that they do not change as long as the object is in the map. The way a lot of libraries and frameworks (e.g., Spring and Hibernate) work, you must give them mutable fields, but these fields might not change after the initialization is done (which is more than just calling the constructor).
The other day, my wife and I made linguine with meatballs and tomato sauce with an improvised recipe. We were quite pleased with the result. This recipe serves three to four people.
My wife made the tomato sauce. Fry the chopped onions and the chopped garlic in olive oil. Add the tomato paste and toast it. Add the tomato puree and cook it. Season it with the Italien herbs, smoked paprika, salt, and pepper. For the finishing touch add some more olive oil and a big sip of Shiraz.
While my wife prepared the sauce, I made the meatballs. Put the minced beef in a bowl and add the meat rub, smoked paprika, salt, and pepper We were considering to also add a chopped or thinly sliced chili. Next time we’ll definitely do that. and mingle it. Form the meat to small meatballs with a diameter between one and two centimeters. Strew some salt and pepper over the meatballs.
Heat up a frying pan with (sun flower) oil and fry the meatballs for two minutes on one side, another two minutes from the other side (until they are crunchy and dark brown) then fry them a little more while tossing them around every now and then. Put the meatballs into the tomato sauce.
Boil water and cook the linguine. Serve the linguine topped with the meatball tomato sauce.
I just released version 0.2 of duckprxy. It now comes with a duck proxy implementation that uses Javassist: JavassistDuckPrxy.
Proxies created with JavassistDuckPrxy are not using runtime reflection, but dynamically created code that is compiled to byte code by Javassist. This should reduce the overhead to that of normal delegation, hence, very little. I did not do any performance tests yet, though.
ZFS is supposed to support encryption, but it does not yet on FreeBSD. In a previous post I wrote about setting up ZFS on FreeBSD where the root file system uses UFS and the rest goes to logical ZFS volumes.
This time I use geli to encrypt a disk partition and use ZFS for the root file system. I encountered a few problems which I’d like to document here.
The FreeBSD Handbook section on encrypted disks show how to set up encrypted disks with geom or geli. However, you cannot use the geli rc-scripts to ask for the passphrase and attach the partition, because we need the partition ready for ZFS which is started by the loader. It needs to be started by the loader, because it shall provide the root file system.
Fortunately, geli can auto-detect encrypted partitions when it is started by the loader. I think it is not possible to use a key-file in this case. At least I could not see how.
I will now show how I set up a new FreeBSD installation with ZFS as the root file system. Following this instruction can and will destroy your data. It can also destroy your data you will put on the newly installed system in the future. I can take no liability.
Firstly, start installing FreeBSD 7.x. Partition the hard disk to have twro partitions. I used 3GB for the first partition which consists of two slices, the first is 1GB for the initial root file system (512MB should be sufficient) and 2GB for swap. The second partition will contain the encrpyted ZFS pool. Continue to install a minimal system.
After the installation is finished initialise the encrypted partition (assuming we want to encrypt ad0s2) and attach it:
# geli init -s 4096 -b /dev/ad0s2 # geli attach /dev/ad0s2
The -b flag will cause geli to ask for the passphrase at start-up.
Now the encrypted partition is ready for ZFS, but I encountered a problem (a bug?). During start-up the the passphrase was not accepted. The problem is that not all charactes I typed where recognized. We will take care of this later (we need a new kernel). For now we use a work-around. Add the following line to /boot/loader.conf
kern.geom.eli.visible_passphrase="1"
ATTENTION: This will show your passphrase when you type it during start-up. This way you will get feed-back what characters got lost.
Create the ZFS pool on the encrypted partition:
# zpool create tank /dev/ad0s2.eli
Edit /etc/rc.conf to enable ZFS:
zfs_enable="YES"
Reboot the system to see if everything works as expected. You should be asked for your passphrase. You will see the passphrase while typing it. After logging in as root you should be able to see that your encrypted partition (ad0s2.eli) is in the ZFS pool by using the following command:
# zpool status
Now you can follow the rest of the instructions for How to install FreeBSD 7.0 under ZFS.
After that is done you probably want to get rid of the passphrase work-around. If you did not have any problems typing in the passphrase, you can simply remove the setting that makes it visible in the /boot/loader.conf Now that we have the space for it, download the FreeBSD sources and create your own kernel configuration with the device dcons removed, which seems to cause the trouble (at least for me). I don’t know why. I just found a discussion where someone mentioned that this might be the problem.
Every now and then I need to implement some interface, but only one (or very few) methods of it. Usually I need something like this in test code, but it is useful elsewhere, too. The mocking libraries I have seen so far offer something similar, but not quite in the way I want to have it.
I had the idea to use something like duck typing for it for a while now. Yesterday I finally found some time to implement it. So I present to you duckprxy. You give it an interface to implement and an object to delegate to you and duckprxy will call the delegate according to the name of the method.
The service interface for this service is defined by DuckProxy. Currently there are two different implementations, but more on that later. Let us assume that the field duckProxy holds such an implementation of DuckProxy (e.g., via injection). Then we can do the following:
interface MyInterface {
void foo();
int bar(int x, int y);
}
public class Delegate {
public int bar() { return 42; }
}
...
MyInterface prxy = duckProxy.makeProxy(MyInterface.class, new Delegate());
prxy.bar(2, 3); // Will return 42.
The standard behavior if an unimplemented method is called is to throw an exception. However, the delegate class can specify a fallback method to call.
public class Delegate {
...
@DuckMethod(fallback = true)
public int defaultMethod(
@DuckArg(DuckArgType.NAME) String name) {
log.info("called method " + name);
}
}
Now every time an unspecified method is called, it will simply be logged.
Other features:
Right now you need to define a class to use the extra-features with annotations. But I am thinking about also allowing to annotate the delegate variable instead of the delegate class.
As mentioned before, there are currently two implementations of DuckProxy. The first is DuckPrxyImpl. This one basically looks what method to call when the call actually happens. The second is DuckPrxyPreCompImpl. This one pre-computes the mapping of the methods in the given interface to the delegate.
I am thinking about using something like javassist for a third implementation that creates a class instead of a proxy. This way the performance impact should be negligible.
I did not really need persistence in Klatschbase, because I never restarted the Lisp image Klatschbase was running in since version 0.2. The only reason I used a different Lisp image in 0.2 than in 0.1 was due to the fact that I moved Klatschbase to another server. Being able to deploy a new version without stopping your application is really a nice feature of Common Lisp.
However, an unexpected reboot of my virtual root server made it clear, that persistence would be a nice thing to have. The machine was rebooted again a few days later. Hence, I decided to do it. It was on my list, anyway.
So I reevaluated the options. I keep on looking at CouchDB. There are two Common Lisp libraries for it. I also played on connecting to it with simple-http directly. This works good.
I also had a brief look at Elepahnt. There was not a lot to see the last time I checked it out, but now it seems to have some good documentation. It is definitely worth a shot if you have to persist something.
But in the end, I decided that my persistence needs are quite basic. I just want to store the clients and their options and the rooms that the server has. I ended up also storing the current messages in the system. Lisp already has a fairly simpl way to store such things: writing s-expressions with write and reading them with read.
I have to admit that the approach that I used does not scale to well, because the whole state is written every n seconds (or minutes). But this is OK in the given context. The plus side is that it gave me persistence with under 100 lines of code.
Another thing I fixed in this version are the parameter names of the generic functions. For some reason I always thought that the names of the generic function parameters must be the respective class. Which means I ended up with generic function definitions like these:
(defgeneric poll-msgs-wait (msg-store t t))
This, of course, is not the case. While in sbcl the above code compiles, other Common Lisp implementations choke on them. The following, more readable version, should work fine:
(defgeneric poll-msgs-wait (msg-store start-key timeout))
