I'm Alex Kearney, I'm Science at the University of Alberta. I focus on Artificial Intelligence and Epistemology.
I'm Alex Kearney, I'm Science at the University of Alberta. I focus on Artificial Intelligence and Epistemology.
Trump's unexpected popularity shows the American election cycle is less a political enterprise and more a reality T.V. bloodsport
I've been running some scripts on west-grid recently, so I've collated the information I've gathered from reading through the guides.
Because I've been running parameter sweeps with multiple algorithms over a nice data-set, I've had to change the way I run experiments. I just don't have the capacity to run experiments in a sane amount of time on my machine. To get around this, I've setup my code to run experiments on west grid, a research computing system.
In the process, I've written up a little guide for running experiments politely. This is mostly a refrence for future-me.
Note: This isn't particularly exhaustive: it's just an introduction; make sure you read west-grid's guides.
Jasper meets my needs, so I'll use it for the overview. The technical specifications are:
Jasper is an SGI Altix XE cluster with an aggregate 400 nodes, 4160 cores and 8320 GB of memory. 240 nodes have Xeon X5675 processors, 12 cores (2 x 6) and 24 GB of memory. Of these, 32 have additional memory for a total of 48 GB. 160 nodes, formerly part of the Checkers cluster, have Xeon L5420 processors, 8 cores (2 x 4) and 16 GB of memory.
| Resource | Limit | | ---------------------------- | -------------- | | Maximum Walltime | 72 hours | | Maximum Running Jobs | 2880 | | Maximum Jobs Submitted | 2880| | Maximum Jobs in Queue | 5|
What walltime means is that the longest you can run a job for is 72 hours. After 72 hours, the job will be killed. You can get around this, by writing a script that performs your job in chunks, meaning that even if it terminates, you can pick-up where you left off. In fact, this is advisable. This way, if you set your wall time too low and your job is aborted part way through, you can pick-up where you left off without the hassle of re-running everything.
The maximum jobs you can have running or submitted at any one time is 2880. That means that the number of blocked jobs that area waiting to be executed may not exceed 2880, and the numer of running jobs may not exceed 2880.
When a job is submitted via qsub it is put through a scheduling system. The scheduler balances fairness with utilization in a number of ways. The actual algorithm is publicly available, but I've just found it's easier to submit a large number of short jobs, rather than a
#!/bin/sh
<a href="/t/PBS">#PBS</a> -S /bin/sh
<a href="/t/PBS">#PBS</a> -j oe
<a href="/t/PBS">#PBS</a> -r n
<a href="/t/PBS">#PBS</a> -o logs/filename.$PBS_JOBID.log
<a href="/t/PBS">#PBS</a> -l nodes=1:ppn=1,walltime=0:20:00,mem=1gb
cd $PBS_O_WORKDIR
echo "Current working directory is `pwd`"
echo "Starting run at: `date`"
python experiment.py --horizon=50
echo "Completed run with exit code $? at: `date`"
#!/bin/bash
for s in s1
do
for a in a1 na1
do
for alg in autotd td tdr totd
do
echo '#!/bin/bash
<a href="/t/PBS">#PBS</a> -S /bin/bash
<a href="/t/PBS">#PBS</a> -M kearney@ualberta.ca
<a href="/t/PBS">#PBS</a> -m bea
<a href="/t/PBS">#PBS</a> -l walltime=01:00:00
<a href="/t/PBS">#PBS</a>
cd $PBS_O_WORKDIR
echo "Current working directory is `pwd`"
module load application/python/2.7.3
time python ./pysrc/experiments/prosthetic-experiment.py 1000 '$runseed' ~/usage-td-experiments2/usage-td-experiments/results/rndmdp-experiments/state-100-ftype-binary/ '$alg' > '$alg'-'$runseed'.txt' > $alg-$runseed.pbs
qsub $alg-$runseed.pbs
done
done
done
list available space on your account
lfs quota -u kearney /lustre
shows all the jobs associated with Kearney
showq -u Kearney
delete jobs
qdel jobid
To immediately kill all your current jobs:
qdel $(showq -u yourname | awk {'print $1'})
This takes the first parameter from showq---the process id---and tells the scheduler to delete it.
For more on running jobs, look here
A discussion of my first nine weeks in the indieweb community.
So, this site is now almost 9 weeks old. I've been loosely following the guide on indiewebcamp, plodding through the recommendation and trying to figure out what the best way to set things up is.
A map of the way I started building things is basically:
Week 1:
I spent an awful lot of time during my first week trying to figure out what the best way to implement things was. Since so much of the indie-web specification is open-ended, there's a lot of decisions right from the get-go that you have to make.
How do you store your photos and your text? What if the text has a related photo? Should the text be stored as mark-down and then converted into HTML? Maybe you should just store it in HTML... The list goes on, and I really wanted to figure out what the most sensible design choice was, because making major changes to the way I store files and the way I format files would be a pain. After all, what sold me on the indieweb was a sense of permanence and structure in the way I store my photos and my text.
Week 2
This was going to be the week dedicated to POSSE: the idea of posting on your own site first, then syndicating elsewhere. I was going to kind-of fake this by using Quill for text posts and Own Your Gram for images. OYG takes images posted on instagram, and sends them to your site. Technically this is PESOS, but I was willing to temporarily compromise, since you can post to facebook, tumblr, and twitter directly from instagram.
This was the week that everything I posted got collected onto my site.
Week 3
This was going to be the week I was going to actually work on POSSE. There's a couple of reasons this is a good thing to do. Chief among them is formatting: when I publish on my site, I can format the text. When I get text back from instagram posts, it's mangled with handles and hashtags. Ew.
Additionally, by posting on my own site first, I can add links back to my site. However, in my attempts to free myself from the instagram app I came face-to-face with the fact that you can't post to instagram from their API
At this time, uploading via the API is not possible. We made a conscious choice not to add this for the following reasons:
- Instagram is about your life on the go – we hope to encourage photos from within the app.
- We want to fight spam & low quality photos. Once we allow uploading from other sources, it's harder to control what comes into the Instagram ecosystem. All this being said, we're working on ways to ensure users have a consistent and high-quality experience on our platform.
So I dumped instagram.
I may come back to it in the future, but I don't care enough about it right now to go ahead and deal with the droll work of removing hashtags and handles. This was the motivating factor to make my own posting client. With it, I can post images (albeit, clumsily) and post to twitter. Through twitter, my posts go to face book.
Boom. Everything is POSSE.
Week 4
This was a week for formatting. Whomever last edited the indieweb site suggests you work on your design first and foremost.
UX and design is more important than protocols, formats, data models, schema etc. We focus on UX first, and then as we figure that out we build/develop/subset the absolutely simplest, easiest, and most minimal protocols & formats sufficient to support that UX, and nothing more. AKA UX before plumbing.
Boo.
Well, I disagree with this primarily because the whole point of this exercise is to take the stuff that I typically leave to rot on social media sites and store it somewhere safe and secure. Safe, secure, and in a nice format that will last in the long-term. For this reason I chose to prioritize the fetching posts from sites I already use and the mechanics of syndication. So, up until this point, the site was largely unusable, but was a kind of storage facility. I promise it's not because I'm an engineer
So, eventually things need . I added a form of navigation that lets you find, for instance, all of the posts from a specific month and year. I started putting effort into a semi-respectable UI. That being said, I'm not known for being a good UX designer
Week 5
Week 6
I took a break and built a little site for rating informatics courses at my university.
Week 7
Because things were usable, I slowed down a bit. I started to work on less interesting things, continuing to focus on the usability, while doing some testing and refactoring the hacked-together mess that was my blog.
Week 8
Skyrim belongs to the Nords.
Week 9
I now feel like I actually have something semi-respectable. Sure, it's spaghetti code that's tangled up, but it does what it's supposed to, and I think I made some reasonable choices that will ensure stability in the long-run. That being said, I'm still a lone island in a vast sea. I don't really spend much time talking to indieweb people, and I've not implemented any of the functionality that allows me to interact with indieweb sites.
Going forward, I'd like to implement webmentions and functionality for event-based posts. Beyond that, I guess spending more time getting integrated in the community would help my project grow in the long-term.
There seems to be some tentative interest in Edinburgh University Hoppers to run some web-development workshops next year. If there's enough interest, I'm sure we could get a few people to start a hombrew club and build our own community.
Speaking of which... I should probably update the ancient hoppers page.
Here's an example of a collection of images, or an album. So I've started this as a means of collating a group of related items.
I wanted the extension to be as minimal as possible. As a result, I treat a collection post the same way as any other post: there's no additional information or details associated with a collection. I manage this by making albums that are simply 'responses' to other posts on the site with the in-reply-to field. This lets me give all the individual images additional information, by letting them be their own posts. It's a bit of a hack and I'm kind-of using in-reply-to fields for something other than what they were originally intended for.
It's just a mock up, really, but it does what I need it to do. I still need to figure out what the most sensible way to display all this is. :/
Found this gem when going through photo's yesterday. Portrait of the Fargeys circa 1900.
A test to see if I've fixed my tweeter!
