Downloading files from usenet

One of the most common questions i get asked is how to get files off of usenet and into a useable form. These files can be any number of things, but the purpose of this entry you can think of them as legit. :)

A little background first. Usenet predates the web, and allows anyone to post information to one of many newsgroups. These are distributed all over the net, and are synchronized automatically. Everyone has access to usenet via google groups. It is easiest to think of them as large forums or mailing lists. All usenet groups have size limits on the files posted up to them. For text groups this isn't a problem, but before the web and downloading there was no easy way to transfer files to other people (this was well before people ran there own servers or had "always on connections"), so someone came up with the idea of converting a file into a number of smaller files, converting those (in a reproducable manner) into text and posting them onto the newsgroup. Anyone interested could then get all these bits of file, re-combine them, and turn them back into the original file. This was at a time where if you were connected you knew how to do this. Geek would be the word you're thinking of!

Fast forward to now, and usenet still exists. It deals with huge amounts of traffic every day, and it is still a repositry for files. The difference is that normal people now use the internet, and they haven't a clue what usenet is, never mind how to use it! That's where this comes in.
There are a number of ways and means of doing this, but i'll cover the ones that work for me. First, you have to find the files. Each newsgroup is made up of millions of messages, and each message has a header. Think of this header as the to, from, and subject part of an e-mail. If you want to know what's in a newsgroup you have to get these headers. As you may need to go into a number of newsgroups, and each one has millions of headers, you don't want to do this. You want an index. Searchable, simple ways of getting exactly what you want. Luckily one exists. Go to newzbin. Here comes shock one. You have to pay (it won't be the first time) if you want to make this useful. You can search newzbin within any of the categories, but that will only get you the existence of the file and the newsgroup it's in. If you're cheap, you can go to the newsgroup and get all the headers, then select the ones you want, and download them, but that is a lot of hassle. It is better to pay newzbin (about 20p a day if i remember correctly) to become a member, you can then get an option to download a newzbin file (extension .nzb) which contains exactly what you need to get the file.

Once you have this file you need to use it, for that you need a program that can read and understand it. I recommend one called binary newsreaper. Once this is installed you need newsgroup access. You probably have this through your internet provider, but that probably doesn't have binary newsgroups and will have terrible retention (how long you can get a file for, newsgroups are purged after a period of time) and completion (there is no way to know if something is there until you try to download it. Bad completion means lots of wasted effort). Once again, it's time to part with some cash. I use newshosting which offers unlimited downloading for $14.95 a month. Pay up, get the access details, and follow the newsreaper instructions to get connected.

Once you're up and running, newsreaper has an "import nzb" option. Choose it, select your nzb file, a destination, click ok and then watch it go. It may take a while depending on the size of the target file.

When it's finished downloading you can look in your destination folder. You probably have several files ending in rar, par2, or r and a number (r01, r02 etc). Not very helpful. Step one is to confirm all the files are complete. That is what all the par2 files are for. Download a free program called quickpar, install, and you should then be able to double click a par2 file and have it automatically check (and repair if necessary) all your files. Once that has completed, get a copy of winrar and the rar files can be clicked. Always choose the .rar file, or the part01.rar file. The program can deal with it if you pick another file, but it's best to make it simple.
The unrar process will leave you with you file(s). All the par2, rar etc. files can be deleted now.

(K)ubuntu update

It's been a lot longer since my last post than I had anticipated, but a new job, and a child on the way conspire against my free time!

I said I would update once I had established myself with kubuntu, so here I am. The experience has been interesting. Initial installation was a dream, one click, 6 questions, easy enough partitioning (managed to keep my home and data areas whilst wiping everything else), and away it goes. First impressions on first boot, it's not as pretty us SuSE. The boot loader for kubuntu (or ubuntu for that matter) is a basic text with DOS style letters. SuSE has a nicer box and a more graphical feel. Once booting, SuSE gives you a nice splash screen, and gives you the option of pressing escape for more information (which appears bordered by the SuSE splash screen headers and is colour coded, uses a nice font and is exceptionally well laid out). Kubuntu gives you a splash screen that looks like it was knocked up in 30 seconds, some default information as things start (but no information as to how they got on), and then half way through the boot, the splash screen disappears and a text screen appears using a 640x480 resolution which can't fit the information on to the screen,. The information is just the default linux messages, but with no formatting, no sizing. Basically, it's a mess.

Once booted and logged in (default KDMs are as nice as each other and irrelevant as I change it to my choice anyway), I was presented with my desktop. Thankfully this was the same one I had under SuSE, so the settings had transferred nicely. A few things were missing, but as I had added them to SuSE I wasn't surprised with the need to add them to kubuntu. And there in lies the only major difference between the systems.

Package management (and when all else fails, manual package installation, or, heaven forbid, source compilation). SuSE uses RPM, kubuntu uses DEB. This is irrelevant. kubuntu uses adept, SuSE uses YAST. This is the difference. There is much debate on the web over the brilliance that is the .deb package management system. It is, without a doubt, superior to .rpm, but you don't need to know this with default system packages. Everything should have been chosen and tested to work together, so all that counts is the ease of choosing applications. The first thing to realise is that non of the distributions come with packages to do everything you want. The most glaring deficiency is the lack of support for most media, meaning you have to add new repositries. Within SuSE this is fairly graphical and easy enough. Within kubuntu you are editing a text file. It may try to pretty things up, but essentially you are editing sources.list. I don't mind this (in fact I edit the file manually at the command line), but it isn't very user friendly.

Side note: I have now found an application called easyubuntu which modifies your sources for you and installs the applications you probably want. It's still not as easy as it could be.

With your sources modified you have to choose what you want to install. With both distributions, there really is a need to know what you want. To me though, SuSE's search options were more intuitive the kubuntu's. Once selected, things install and generally work.

What happens if you can't find an app in the repositry? You need to install it manually. Within SuSE and kubuntu if you find an rpm or a .deb, then things are easy. Double click the file and away you go (not strictly accurate, kubuntu defaults to unpacking the file in ark instead of installing it, but it is negotiable). If you only have source to go on here comes the fun part! Within YAST there is a helpful option on the menu that allows you to install all development header files for all the libraries you have installed. One option, one click, that's it. Kubuntu needs you to know what they are all called, and then you select them manually. When you are compiling something complicated (for example hugin an open source panoramic photo editor) there are dozens of library files needed, and no one tells you up front what you need. You have to run ./configure again and again letting it fail slightly further each time as you install the dependancy it's missing. Boy is that fun!

And that is it. For my own use, the distributions are as similar as they need to be. Some things are in different places, but it's easy enough to work out. Both distributions are capable of running everything I need. There is slightly more support for (k)ubuntu the SuSE, but not enough to worry about.

There is a 'but'. Isn't there always! I have an AMD 64 bit chip as my main machine's processor. On SuSE and kubuntu there is equal support for the main OS. However, application providers don't have the same support. Within SuSE they get round this by installing the 32 bit libraries alongside the 64 bit ones. If an app can only run on 32 bit, there isn't a problem. Kubuntu doesn't have this option. I ended up wiping the OS and installing the 32 bit version. It hasn't made a difference to performance, but it doesn't feel right. It 'should' be easy enough (I don't know I'm not a developer) to install the 32 bit libraries, why can't kubuntu do it. I did find instructions to setup a '32 bit chroot', and tried this, but it broke after an update, and it was too much hassle to keep up to date.

For all it's failings, kubuntu does seem to be a good OS. It fixed the problems I was having with SuSE (but so have SuSE apparently), and I won't be switching again soon. Stay tuned to know how updating to the next version of kubuntu goes later this month.