I'm happy to announce new lc-tools release: 0.7.1.0!
It doesn't have new features, but was updated to work with libcloud 0.7.x (earlier libcloud versions are no longer supported!). All the kudos go to Pavel Vozdvizhenskiy for noticing problems and testing.
You can install the new version using either easy_install:
easy_install lctools
or by downloading and installing it by hands:
lctools-0.7.1.0.tar.gz
SHA256 (lctools-0.7.1.0.tar.gz) = 7f65c29cda06d256c02f6544b4957ca984f34f1cfed5b24558c3b4338f87dd9c
Tuesday, January 3, 2012
Wednesday, December 21, 2011
Bug Prediction
Recently I've read a very interesting post about Bug Prediction on Google Engineering Tools blog and decided to inspect FreeBSD source tree in the same manner.
There results appeared not very interesting though. This kind of matches security advisories, but given that there are not very much of them (luckily!), it's not very representative.
I also tried to run the script with "[\t ]+[Bb]ug" regular expression instead of "Security:", and got the following result:
This is somewhat more interesting result. It shows that most 'dangerous' sits in SCTP implementation, NIC drivers (bge, msk, mxge) and also some bits in zfs and bind. I wonder how accurate is that?
The script is available here. It's designed to use git repository. I used freebsd git mirror on github: https://github.com/freebsd/freebsd-head.
The question is how to determine if the given commit is a bug fix. I decided to start with a security fixes first and filtered commit messages by "Security:" tag in them. The result is the following:
| Filename | Score |
|---|---|
| crypto/openssl/ssl/s3_srvr.c | 1.12807360699 |
| crypto/openssl/ssl/s3_clnt.c | 1.12546730725 |
| crypto/openssl/openssl.spec | 0.914805209201 |
| crypto/openssl/NEWS | 0.914805209201 |
| crypto/openssl/CHANGES | 0.914805209201 |
| crypto/openssl/README | 0.914805209201 |
| crypto/openssl/crypto/opensslv.h | 0.914805209201 |
| crypto/openssl/Makefile | 0.914805209201 |
| crypto/openssl/FAQ | 0.914805209201 |
| crypto/openssl/apps/speed.c | 0.707204148366 |
| crypto/openssl/ssl/t1_lib.c | 0.696806056187 |
| crypto/openssl/ssl/s3_lib.c | 0.663083892748 |
| crypto/openssl/crypto/md32_common.h | 0.578509821664 |
| crypto/openssl/crypto/ocsp/ocsp_prn.c | 0.578509821664 |
| crypto/openssl/engines/e_chil.c | 0.578509821664 |
| crypto/openssl/ssl/d1_pkt.c | 0.51304099663 |
| crypto/openssl/crypto/pqueue/pqueue.c | 0.51304099663 |
| crypto/openssl/ssl/d1_both.c | 0.51304099663 |
| contrib/ntp/ntpd/ntp_crypto.c | 0.505424173077 |
| crypto/openssl/crypto/asn1/tasn_dec.c | 0.482876843456 |
| sys/compat/linux/linux_socket.c | 0.477720395098 |
| usr.bin/compress/zopen.c | 0.462407652214 |
| usr.bin/gzip/zuncompress.c | 0.462407652214 |
| sys/kern/uipc_usrreq.c | 0.462407652214 |
| crypto/openssl/ssl/ssltest.c | 0.460574251046 |
| crypto/openssl/ssl/s2_srvr.c | 0.460574251046 |
| crypto/openssl/apps/x509.c | 0.460574251046 |
| crypto/openssl/ssl/s3_pkt.c | 0.456096100304 |
| sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c | 0.44243544619 |
| sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c | 0.44243544619 |
There results appeared not very interesting though. This kind of matches security advisories, but given that there are not very much of them (luckily!), it's not very representative.
I also tried to run the script with "[\t ]+[Bb]ug" regular expression instead of "Security:", and got the following result:
| Filename | Score |
|---|---|
| sys/netinet/sctp_output.c | 7.66251938944 |
| sys/netinet/sctp_pcb.c | 7.2065885975 |
| sys/netinet/sctputil.c | 6.80461378901 |
| sys/dev/bge/if_bge.c | 6.50257671046 |
| sys/netinet/sctp_input.c | 5.62524237411 |
| sys/netinet/sctp_indata.c | 5.25336437345 |
| sys/netinet/sctp_usrreq.c | 5.11155404498 |
| sys/netinet/sctp_timer.c | 3.67543247047 |
| sys/netinet/sctp_structs.h | 2.99517512907 |
| sys/conf/files | 2.89041302648 |
| sys/netinet/sctp_asconf.c | 2.79978400331 |
| sys/netinet/sctp_constants.h | 2.79452724525 |
| sys/dev/bge/if_bgereg.h | 2.62852430236 |
| sys/dev/msk/if_msk.c | 2.31771522145 |
| sys/netinet/sctputil.h | 2.19989952769 |
| contrib/bind9/CHANGES | 2.15204576684 |
| contrib/bind9/version | 2.15204576684 |
| sys/kern/kern_fork.c | 2.12687393752 |
| sys/netinet/in.c | 2.09501338006 |
| sys/netinet/sctp_sysctl.c | 2.06560623664 |
| sys/netinet6/sctp6_usrreq.c | 1.92839801197 |
| sys/netinet/sctp_var.h | 1.91990472478 |
| sys/conf/NOTES | 1.79166008565 |
| sys/dev/mxge/if_mxge.c | 1.74628407608 |
| contrib/bind9/bin/named/query.c | 1.74336559317 |
| contrib/bind9/lib/dns/rbtdb.c | 1.74336559317 |
| sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c | 1.7287195258 |
| sys/i386/xen/pmap.c | 1.68724079063 |
| sys/powerpc/aim/mmu_oea64.c | 1.68644933665 |
| contrib/bind9/lib/dns/validator.c | 1.6676964412 |
This is somewhat more interesting result. It shows that most 'dangerous' sits in SCTP implementation, NIC drivers (bge, msk, mxge) and also some bits in zfs and bind. I wonder how accurate is that?
The script is available here. It's designed to use git repository. I used freebsd git mirror on github: https://github.com/freebsd/freebsd-head.
Sunday, October 23, 2011
Apple keyboard on FreeBSD
Recently I bought an iBook G4 just for fun and of course installed FreeBSD on it. Installation goes pretty smooth starting with 9.0: I've started with 9.0-BETA3 and later updated to 10-CURRENT. I used it as a toy mostly but on Thursday my main laptop (which also runs FreeBSD, but amd64/8-STABLE) died and I had to use iBook as my main system and some problems became noticable.
There are two annoying problems with Apple keyboards:
1. Function keys (F1, F2, ...) work as they supposed to work only when pressing them with FN key. I rely on virtual desktops a lot and I use Ctrl-Fx to switch between them, so you imagine how annoying is that to use Ctrl-Fn-Fx to switch a desktop.
2. Apple keyboard misses 'Insert' key completely, so you cannot use Ctrl-Insert for pasting, so it's extremely annoying also.
Fortunately, both problems could be easily solved.
Several days ago Nathan Whitehorn ⟨nwhitehorn@) committed a support for dev.akbd.%d.fn_keys_function_as_primary which allows you to configure F-keys behavior. In order to make the keys look as F1,F2,... without pressing FN, execute:
The Insert key problem could be solved on X level: I've used xmodmap (x11/xmodmap) to remap F11 to Insert, which is a good solution for me because I don't use F11 key and it's located quite close to a place where you'd expect to have 'Insert' key on a typical keyboard.
So I've created ~/.Xmodmap file:
And added execution of "xmodmap ~/.Xmodmap" on X session start.
I'm not sure if F11 has keycode 95 on every Apple keyboard, it's easy to figure out keycode using xev (x11/xev).
There are two annoying problems with Apple keyboards:
1. Function keys (F1, F2, ...) work as they supposed to work only when pressing them with FN key. I rely on virtual desktops a lot and I use Ctrl-Fx to switch between them, so you imagine how annoying is that to use Ctrl-Fn-Fx to switch a desktop.
2. Apple keyboard misses 'Insert' key completely, so you cannot use Ctrl-Insert for pasting, so it's extremely annoying also.
Fortunately, both problems could be easily solved.
Several days ago Nathan Whitehorn ⟨nwhitehorn@) committed a support for dev.akbd.%d.fn_keys_function_as_primary which allows you to configure F-keys behavior. In order to make the keys look as F1,F2,... without pressing FN, execute:
sysctl dev.akbd.0.fn_keys_function_as_primary=1The Insert key problem could be solved on X level: I've used xmodmap (x11/xmodmap) to remap F11 to Insert, which is a good solution for me because I don't use F11 key and it's located quite close to a place where you'd expect to have 'Insert' key on a typical keyboard.
So I've created ~/.Xmodmap file:
keycode 95 = InsertAnd added execution of "xmodmap ~/.Xmodmap" on X session start.
I'm not sure if F11 has keycode 95 on every Apple keyboard, it's easy to figure out keycode using xev (x11/xev).
Thursday, July 28, 2011
the bike
frame: blkmrkt three57
fork: marzocchi dj2 '09 100mm
handlebar: blkmrkt badaboom
stem: ns quark pro '11
seat: blkMrkt brass knuckles (--> leaf cycles symboll pivotal soon-ish)
seat post: blkmrkt riot stick (--> code pivotal 135mm)
sprocket: federal street 25t
pedals: ns aerial '11
cranks: crmo 3pc, 48 splined
chain: kmc
front hub: ns rotary 20 '11
rear hub: ns rotary single '11 10t
tires: kenda 26x2.10"
spokes: mach
photos are so-so and bike config is not perfect I guess, but it's good starting point.
Tuesday, July 12, 2011
fusefs-cloudstorage: accessing cloud storage using libcloud and fusefs
Few weeks after cloud storage support appeared in libcloud I started playing with it by creating a fusefs filesystem based on it. Finally the first version is almost ready, so it's time to write a few words about it.
So, for example, you have a cloud files account and want to mount it as it were a local filesystem. You do it this way:
where foo is your access id and this hash is the secret key. Filesystem will be mounted to test sub-directory of the current directory.
A list of directories at top level of this filesystem represents a list of containers on your account and looks like this:
Listing each of these directories will (as you might have guessed already) show a list of objects that belong to a corresponding container. For example:
You can create containers using mkdir command, copy files in using cp, remove them with rm and so on. You might want to check a shell script test that tests basic features for more examples: https://github.com/novel/fusefs-cloudstorage/blob/master/test.sh.
However, it seems that it's not that easy and quick tasks to implement such a filesystem.
My implementation is pretty dumb -- it just does things in a straight-forward way that doesn't work well in this case because filesystem usage patters doesn't match cloud storage usage patterns well.
A couple of examples: cloud storage doesn't support doing writes or reads with offset, so if you want to append a line to a text file, you have to remove the old one and create a new one with the new content. This gets especially slow when we're modifying large files.
Other thing that it looks like vfs design generally assumes it's easy to obtain meta-information about the files, but with cloud storage every single simple operation like checking if file exists, involves issuing an API call, which is slow.
It looks like we can speedup things dramatically if we introduce caching for both meta-data and file contents. But the hard thing with cache is that we don't have an exclusive access to the API. I.e. we can cache things in our filesystem driver and some user at the same time might open up Web Dashboard and remove or upload new files and we will never know about it unless re-request all the data.
At this point I consider this project more like "just for fun" type of thing than something really useful, but probably I will implement caching to make it more or less fast if I decide it will be useful.
So, for example, you have a cloud files account and want to mount it as it were a local filesystem. You do it this way:
novel@ritual:~/code/fusefs-cloudstorage/test %> cloudstorage.py -o driver=CLOUDFILES_US -o access_id=foo -o secret=acbd18db4cc2f85cedef654fccc4a4d8 ./test
where foo is your access id and this hash is the secret key. Filesystem will be mounted to test sub-directory of the current directory.
A list of directories at top level of this filesystem represents a list of containers on your account and looks like this:
novel@ritual:~/code/fusefs-cloudstorage/test %> ls -1
backups
cloudservers
event_logs
mir_package
novel@ritual:~/code/fusefs-cloudstorage/test %>
Listing each of these directories will (as you might have guessed already) show a list of objects that belong to a corresponding container. For example:
novel@ritual:~/code/fusefs-cloudstorage/test %> ls -1 backups
1345b96a-ae25-11e0-94a9-40401629a6e1.sql.gz
135dbf74-ae25-11e0-94a9-40401629a6e1.sql.gz
27ed5e4e-7faf-11e0-bbcc-40401629a6e1.sql.gz
4fbb5e06-ae14-11e0-aace-40401629a6e1.sql.gz
4fd6274a-ae14-11e0-aace-40401629a6e1.sql.gz
645e7cde-7f9e-11e0-8a43-40401629a6e1.sql.gz
b181ef6c-ae1c-11e0-a6b9-40401629a6e1.sql.gz
b19dcbba-ae1c-11e0-a6b9-40401629a6e1.sql.gz
c626402a-7fa6-11e0-bb13-40401629a6e1.sql.gz
f460c934-9b2f-11e0-89ce-40401629a6e1.sql.gz
novel@ritual:~/code/fusefs-cloudstorage/test %>
You can create containers using mkdir command, copy files in using cp, remove them with rm and so on. You might want to check a shell script test that tests basic features for more examples: https://github.com/novel/fusefs-cloudstorage/blob/master/test.sh.
However, it seems that it's not that easy and quick tasks to implement such a filesystem.
Why is it hard to make such kind of filesystem fast?
My implementation is pretty dumb -- it just does things in a straight-forward way that doesn't work well in this case because filesystem usage patters doesn't match cloud storage usage patterns well.
A couple of examples: cloud storage doesn't support doing writes or reads with offset, so if you want to append a line to a text file, you have to remove the old one and create a new one with the new content. This gets especially slow when we're modifying large files.
Other thing that it looks like vfs design generally assumes it's easy to obtain meta-information about the files, but with cloud storage every single simple operation like checking if file exists, involves issuing an API call, which is slow.
It looks like we can speedup things dramatically if we introduce caching for both meta-data and file contents. But the hard thing with cache is that we don't have an exclusive access to the API. I.e. we can cache things in our filesystem driver and some user at the same time might open up Web Dashboard and remove or upload new files and we will never know about it unless re-request all the data.
At this point I consider this project more like "just for fun" type of thing than something really useful, but probably I will implement caching to make it more or less fast if I decide it will be useful.
Links
Monday, May 30, 2011
lc-tools 0.5.0.0 released!
A lot of time passed since the latest lc-tools release, and now it's almost summer, so it's time for updates!
The first news is that libcloud is now an Apache Top Level Project (TLP) and new version 0.5.0 has been released with some new features, bug fixes and new support for working with storage and load balancers. And it's a perfect time for new lc-tools release!
There are not so many new features in this lc-tools release, most of the changes were to chase internal libcloud changes. Additionally, I've changed versioning scheme a little and now it looks like $libcloud_version$lctools_version, where $libcloud_version is the version of libcloud current release was targeted and tested with.
The only new major change is the addition of lb-* scripts to manage load balancers. You can get more information on them from Documnetation.
You can install the new version using either easy_install:
easy_install lctools
or by downloading and installing it by hands:
lctools-0.5.0.0.tar.gz
SHA256 (lctools-0.5.0.0.tar.gz) = f09629664a5209be687c56cfe07e123939e28b18070b719436307a74b823f10b
Happy hacking on the cloud stuff!
The first news is that libcloud is now an Apache Top Level Project (TLP) and new version 0.5.0 has been released with some new features, bug fixes and new support for working with storage and load balancers. And it's a perfect time for new lc-tools release!
There are not so many new features in this lc-tools release, most of the changes were to chase internal libcloud changes. Additionally, I've changed versioning scheme a little and now it looks like $libcloud_version$lctools_version, where $libcloud_version is the version of libcloud current release was targeted and tested with.
The only new major change is the addition of lb-* scripts to manage load balancers. You can get more information on them from Documnetation.
You can install the new version using either easy_install:
easy_install lctools
or by downloading and installing it by hands:
lctools-0.5.0.0.tar.gz
SHA256 (lctools-0.5.0.0.tar.gz) = f09629664a5209be687c56cfe07e123939e28b18070b719436307a74b823f10b
Happy hacking on the cloud stuff!
Wednesday, May 18, 2011
Dealing with pkg-config detection of FreeBSD's base OpenSSL
Recently I spotted a software that uses pkg-config to find OpenSSL on the system. This way doesn't work on FreeBSD: it does have OpenSSL in base system, but it doesn't supply '*.pc' file, unfortunately. So, when configure script tries to execute pkg-config openssl it fails, obviously.
There were a number of solutions suggested for this problem on #bsdports like replacing 'pkg-config openssl' with 'pkg-config pkg-config', depending on openssl from ports etc, but the cleanest solution is to define libssl_LIBS and libssl_CFLAGS environment variables, as was suggested by bapt@. In port's Makefile it would look something like this:
Yes, configure will not call pkg-config if we define these 'libssl_*' variables. While it looks quite simple, this issue took surprisingly a lot time to figure everything out. :(
There were a number of solutions suggested for this problem on #bsdports like replacing 'pkg-config openssl' with 'pkg-config pkg-config', depending on openssl from ports etc, but the cleanest solution is to define libssl_LIBS and libssl_CFLAGS environment variables, as was suggested by bapt@. In port's Makefile it would look something like this:
USE_OPENSSL=yes
CONFIGURE_ENV+=libssl_CFLAGS="-I${OPENSSLINC}" libssl_LIBS="-L${OPENSSLLIB} -lssl"Yes, configure will not call pkg-config if we define these 'libssl_*' variables. While it looks quite simple, this issue took surprisingly a lot time to figure everything out. :(
Subscribe to:
Posts (Atom)