Saturday, 18 November 2017

Installing VNC on Ubuntu 16.04

I wanted to use VNC to allow multiple desktops on a server.
In Ubuntu 16.04 (or 14.04, 17.10 or Linux Mint Cinnamon) I found that neither the GNOME desktop nor the XFCE desktop would work with VNC - I tried vnc4server, realvnc, and tigervnc. They all crashed in slightly different ways, but each was unuseable.

The solution, which so far I've tested in Mint and 16.04 with tigervnc, is just to make sure you uninstall the default vino desktop sharing tool before installing tigervnc. Simple!

Thursday, 18 August 2016

Weird matlab behaviour for empty matrices.

I've noticed that (at least in Matlab 2015b) a weird behaviour with Matlab matrices.

Here are two ways of making different empty matrices:
>> a=[]
>> size(a)
ans=
    0  0
>> isempty(a)
ans=
    1


>> b=find(1==[0 0])
>> size(b)
ans=
    1  0
>> isempty(b)
ans=
    1


They both look empty, but have different sizes. But critically for some code, they behave very differently in a for loop
>> for ind=a
>>    ...this never gets executed!
>> end;

>> for ind=b
>>    ...this gets executed once, with ind=[];
>> end;

This is a pernicious problem. Is there any reason for it to be like this?

Monday, 21 April 2014

Illustrator revalation

I often use Adobe Illustrator to draw illustrations or finesse graphics. While I'd prefer a cheaper package, there doesn't currently seem to be anything that comes close to the sophistication of this tool. Like Photoshop (or Flash) it isn't the most intuitive thing, though. You really need expertise. But first, you need to know what the feature is that you need. As soon as you know this, Google is your friend.

So, today, I thought "Wouldn't it be cool if there were a way to align multiple objects not just to each other, but to the fixed position of a single object?"

It turns out this is dead simple - you shift-click to select all the objects you wish to align, and then do an extra click on one of these selected items. This becomes the reference to which everything else is aligned.

I was so excited I had to share.

Thursday, 3 April 2014

Sunday, 27 October 2013

meld

Both for maintaining github repositories, and assorted code merging, I've been enjoying using the tool "meld". For adhoc merging, if you have two different versions of a file, do

meld [filename1] [filename2]

You'll then see the two files in parallel, with MELD highlighting new, changed or deleted code in each file. You can then click the arrows to transfer the change in one direction or the other.


You may also merge more than two versions.

If you use git, you may set this to use meld by default:

git config --global merge.tool meld

Then type

git mergetool

will resolve all of the conflicts that have arisen with an update.


Saturday, 5 October 2013

Benchmarking ZFS against LVM

ZFS is a filing system that offers deduplication (i.e., two copies of the same file in different parts of the file system are only stored once) and compression with two algorithms (lzjb or gzip). A trial dataset of typical neuroimaging data from our system (59GB ) gave a 1.5 deduplication ratio and compression of 1.25 with lzjb and 1.32 with gzip. These are tests of ZFS backed by EBS on an Amazon EC2 cc2.8xlarge server.

Following a clear of the memory cache with:
sync; echo 3 > /proc/sys/vm/drop_caches

I then benchmarked a directory listing (~1000 nifti files) and read/write (10 random EPI files per subject). Access times (ms)  with statistics over 10 subjects of 4 sessions:

dir zfs-nocompress 58.8 +/- 42.0 zfs-lzjb 38.5 +/- 16.2 zfs-gzip 53.9 +/- 47.6 lvm 146.8 +/- 480.6
read zfs-nocompress 576.3 +/- 277.0 zfs-lzjb 629.7 +/- 301.7 zfs-gzip 523.5 +/- 82.1 lvm 499.8 +/- 377.1
write zfs-nocompress 207.1 +/- 13.1 zfs-lzjb 207.0 +/- 12.6 zfs-gzip 205.5 +/- 6.5 lvm 188.3 +/- 1.3

 On a second run immediately succeeding the one above (to maximize caching)

dir zfs-nocompress 6.0 +/- 0.3 zfs-lzjb 5.8 +/- 0.4 zfs-gzip 5.9 +/- 0.3 lvm 4.8 +/- 0.2
read zfs-nocompress 549.8 +/- 150.6 zfs-lzjb 601.4 +/- 292.7 zfs-gzip 544.2 +/- 210.8 lvm 461.2 +/- 200.0
write zfs-nocompress 207.8 +/- 17.3 zfs-lzjb 206.9 +/- 16.4 zfs-gzip 203.7 +/- 1.9 lvm 189.8 +/- 3.1

Directory listing has got faster, and reads a bit faster.

ZFS seems slower for reading (1-2.5 times, depending on conditions), partly due to decompression latencies. It also uses more CPU, but from watching the system monitor, this was barely noticeable here as the disk access to EBS is still the bottleneck.

ZFS has other benefits - it supports caching (on SSD drives for cc2.8xlarge ephemerals); and snapshots for backup, which don't use memory except to store changes in files.

In summary, for our data deduplication and compression look valuable, but it does come with a performance cost. But, this can be made up for by more parallelisation of jobs. gzip looks the better option for our data.


Update: After copying in ~3TB of data, in lots of small files, ZFS appears to be slowing down for both directory listing and reading. The files had probably fallen out of the cache (unless ZFS had kept them in because of their frequency of access).
dir zfs-nocompress 35.8 +/- 33.6 zfs-lzjb 86.2 +/- 51.6 zfs-gzip 127.7 +/- 179.2 lvm 5.1 +/- 0.2
read zfs-nocompress 792.1 +/- 415.7 zfs-lzjb 822.9 +/- 372.7 zfs-gzip 887.2 +/- 475.5 lvm 390.0 +/- 36.3
write zfs-nocompress 209.5 +/- 2.6 zfs-lzjb 208.5 +/- 2.1 zfs-gzip 208.1 +/- 3.7 lvm 192.5 +/- 1.4

And a repeat, where the files will have been in the cache:dir zfs-nocompress 19.2 +/- 8.2 zfs-lzjb 23.3 +/- 14.2 zfs-gzip 22.0 +/- 19.5 lvm 5.0 +/- 0.2
read zfs-nocompress 781.9 +/- 429.4 zfs-lzjb 698.2 +/- 179.7 zfs-gzip 788.0 +/- 320.0 lvm 409.9 +/- 135.7

write zfs-nocompress 207.8 +/- 3.3 zfs-lzjb 206.4 +/- 2.2 zfs-gzip 207.5 +/- 5.0 lvm 193.3 +/- 2.5
ZFS is doing even worse. This seems worrying... does anyone have a solution?

Update: Under reasonable load (here 5 simultaneous copies from lvm to zfs) things really seem to fall apart with ZFS:
dir zfs-nocompress 4438.2 +/- 20030.3 zfs-lzjb 453.5 +/- 554.9 zfs-gzip 732.9 +/- 530.3 lvm 14.3 +/- 10.5
read zfs-nocompress 8622.1 +/- 5483.3 zfs-lzjb 5494.3 +/- 4014.2 zfs-gzip 1485.2 +/- 824.7 lvm 445.9 +/- 198.3
write zfs-nocompress 4400.4 +/- 4677.5 zfs-lzjb 1946.9 +/- 3082.8 zfs-gzip 2445.3 +/- 2301.6 lvm 201.5 +/- 13.5
I've checked the L2ARC cache and it is nowhere near full - 8 GB out of 32 GB max. What's the bottleneck, and why does it degrade so dramatically?

Update: Part of the bottleneck is de-duplication, as described in this post:
http://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSDedupMemoryProblem?showcomments#comments
Switching this off speeds things up, but not enough under reasonable load - ZFS is still unacceptably slow.

Conclusion: Abandoned as a solution for now, until I can identify the cause of the slowing.


Monday, 5 August 2013

For UWO employees... accessing journal or book subscriptions from outside the office

If you've gone through pubmed or Scholar and arrived at a paper you'd like to obtain, but you've been rejected because the site doesn't recognize your subscription, then just put "proxy2.lib.uwo.ca" onto the end of the site name, between the URL and its parameters. For example...
http://www.sciencedirect.com/science/article/pii/S1053811910010062
becomes...
http://www.sciencedirect.com.proxy2.lib.uwo.ca/science/article/pii/S1053811910010062

You'll then need your Western ID and password, of course.

Magic, huh?

Tuesday, 30 July 2013

Setting up your .bashrc for Mturk on cercimage

Add these two lines on the end:

export PATH=/home/rcusack/software/aws-mturk-clt-1.3.0/bin:$PATH
export MTURK_CMD_HOME=/home/rcusack/software/aws-mturk-clt-1.3.0

Sunday, 28 July 2013

Thursday, 25 July 2013

Poisson disc in Matlab

Poisson disc sampling is a nice way of getting random yet even coverage.

Following the helpful page from devmag here are 2d and 3d implementations of Poisson disc (sphere?) sampling in Matlab. Also an implementation using SPM's grey mask to parcellate grey matter to an arbitrary degree.

Wednesday, 20 February 2013

Get rid of Unity under RealVNC

Go to the directory .vnc in your home directory.
Backup the file xstartup, make a new file, and replace it with:


#!/bin/sh

# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc

#[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
#xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
touch /home/rhodri/.vnc/haverunxstarup

gnome-session --session=gnome-classic &

On restarting your vncserver, you should get an old style desktop.

Thursday, 3 January 2013

Ubuntu reanimates netbook

I had an essentially dead Acer Aspire One D257 netbook. With Windows 7 Starter, which it arrived with out of the box (along with lots of bloatware) it had become unbearably slow. There were two options: dustbin, or reanimation with Linux. After a failed diversion with Joli OS (which is very pretty but the Wifi resolutely refused to work), I installed Ubuntu 12.10. This works very immediately, and has made the machine a pleasure to use. I also found the preview version of Spotify for Linux, which seems to work well too.

Tuesday, 23 October 2012

Matlab/FSL + Ubuntu 12.04 library problem

Since upgrading to Ubuntu Precise 12.04, we've been getting an error on starting matlab of 

/usr/local/MATLAB/R2011b/bin/util/oscheck.sh: /lib64/libc.so.6: not found

This same problem also seemed to break bet from FSL.

Now in your matlab path, for us
/usr/local/MATLAB/R2011b/sys/os/glnxa64
you'll find
libstdc++.so.6
symbolically linked to
libstdc++.so.6.0.1.13

Ubuntu ships with a newer version of this library in
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16

So, I symbolically linked the Matlab version through to the one shipped with Ubuntu

$ cd /usr/local/MATLAB/R2011b/sys/os/glnxa64
$ mv libstdc++.so.6.0.1.13 libstdc++.so.6.0.1.13.old
$ ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16 libstdc++.so.6.0.1.13

This seems to fix it.

Friday, 5 October 2012

Ubuntu Precise 12.04 & VNC

When we upgraded to 12.04, the RealVNC desktop would come up blank. I found the solution here.

Tuesday, 21 August 2012

Installed iBeat

I've installed the iBeat infant brain image processing software on ssc-cercimage.

To use it, add to your .bashrc:


 export IBEAT_HOME=/home/rcusack/software/ibeat
 source /home/rcusack/software/ibeat/ibeatEnv.bash

Sunday, 5 August 2012

Using Ubuntu 12.04 by VNC

I find the Unity desktop that ships with Ubuntu is clumsy to use over VNC. When changing workspaces, for example, it shows a full screen overview of what is on each desktop, which takes a long time to load over a slower connection. I'm sure it is very beautiful when sitting in front of the machine, but it is grating when used over a remote connection.

Also, there seem to be some exotic keystrokes (e.g., ctrl+alt+arrow to switch workspaces), or lots of middle and right button work on the taskbar, which often don't translate properly down a VNC.

To switch desktops, first install gnome with
apt-get install gnome-session-fallback


Then, to configure VNC to launch into this, change the file .vnc/xstartup so that it has the line:
exec gnome-session --session=gnome-classic


Tuesday, 10 July 2012

ec2 & fstab

If you're setting up an ec2 instance in the Amazon cloud and are setting devices (EBS volumes) to mount on reboot by adding them to /etc/fstab, make sure you put "nobootwait" in the options, like this:

/dev/xvdh /export/home ext3 auto,rw,nobootwait 0 0

If you don't, then if the drive isn't present, the boot process will hang, either with a blank console output, or with:


The disk drive for xxx is not ready yet or not present
Continue to wait; or Press S to skip mounting or M for manual recovery


and as the keyboard is miles away, you can't press that key.

Tuesday, 27 March 2012

Monday, 26 March 2012

Configure remote access to git without erasing your existing keys

This is a solution if you'd like to connect to github using ssh (typical for read-write access to a repository) but you don't want to delete an existing ssh key (e.g., id_rsa) as requested in the git instructions. On ubuntu, I did it like this:

$ cd .ssh
$ ssh-keygen -t rsa -C "myemail@place.com"
Enter file in which to save the key (/home/username/.ssh/id_rsa): github_rsa
Enter a passphrase: your secret phrase

Copy the public key file ~/.ssh/github_rsa.pub to your github.com ssh settings.

Edit the file .ssh/config, and add the following entry (or create a new file if it doesn't currently exist)

Host github
  Hostname github.com
  User git
  IdentityFile ~/.ssh/github_rsa

Then, I had to do
$ ssh-add ~/.ssh/github_rsa
to get rid of an "Agent can't sign error"

Then, if you do
$ ssh -T github

you should see
Hi rhodricusack! You've successfully authenticated, but GitHub does not provide shell access.

Then you can do
git clone github:/rhodricusack/automaticanalysis

Sunday, 18 March 2012

Using Univa grid engine for FSL

GridEngine is installed on cercimage at the BMI. Elsewhere, you'll need to install the Univa GridEngine. Then, add the following line to the end of the .bashrc file in your home directory
. /gridware/univa/default/common/settings.sh
A number of FSL functions (including POSSUM & MELODIC) will then execute in parallel. You may type qstat or "watch qstat" to see the jobs being queued up.