< ?php $a = base64_decode(YSBsb25nIHN0cmluZyBvZiBiYXNlNjQgdGV4dAo=.......);

right at the top of a script. base64_decode is rarely used by the Good Guys outside of mailers and doing tricks with images, but it's almost never found right at the top of a script. I did write a really convoluted script that found calls to base64_decode and exec and guessed whether they were nefarious (generally, for example, base64_decode is called with a variable (base4_decode($mailBody)), not just a string (base64_decode(dGV4dAo=)) but that just ate all my I/O and didn't really work.

So I came up with a much cruder way of doing it. Have a script called ~/bin/base64_in_head

#! /bin/bash
file=$1
head $file | grep base64 || exit 1;
echo $file
exit 0;

And then run it like this:

$ ionice -c3 find /home/user/public_html/ -name \*.php -exec ~/bin/base64_in_head {} \;

I've not yet had a situation where that's missed a file that later manual greps has found.

It's really easy:

#!/bin/bash
if [ -z $1 ]; then
        echo "Usage"; echo "  $0 "; echo ""; exit
fi
wget http://software.virtualmin.com/gpl/scripts/install.sh
export VIRTUALMIN_NONINTERACTIVE=""
bash install.sh -f -host $1
rm install.sh

And then you call it like so:

./virtualmin.sh virtualmin.vm.avi.co

That was almost a disappointment.

MediaWiki has this nifty feature where you can split the logging for particular extensions out into individual files by doing things like this:

 
$wgDebugLogGroups = array(
        'SomeExtension'     => '../logs/wiki_SomeExtension.log',
);
 

What's not made overly clear (well, with hindsight, it is implied by the manual) is that the keys of the hash don't necessarily have anything to do with the name of the extension. I assumed that, in debugging SimpleCaptcha, what I wanted was

 
$wgDebugLogGroups = array(
        'SimpleCaptcha'     => '../logs/wiki_SimpleCaptcha.log',
);
 

But not so! What I actually wanted was

 
$wgDebugLogGroups = array(
        'captcha'     => '../logs/wiki_confirmedit.log',
);
 

And, as far as I can find, this isn't documented *anywhere*. For other extensions lacking in documentation so, you can find this out by poking around in the code, and looking for where the extension does this sort of thing:

 
function log( $message ) {
      wfDebugLog( 'captcha', 'ConfirmEdit: ' . $message . '; ' .  $this->trigger );
}
 

That first argument to wfDebugLog is what you want as the key in the hash. Why it can't just use the name of the class invoking it, which is the name used to configure the rest of the extension, I've no idea - it seems, from the perspective of a non-developer who doesn't do OOP, like the obvious way to do it.

So I've removed the code it uses to check.

Here's a pair of diffs if you'd also like to do this. These are on version 1.17.0 but I suspect it's not changed very much.

This just comments out the two blocks of code in UploadBase.php which check whether files are considered safe and warn if they're not - it prevents the checking and the warning:

wiki:/home/wiki/public_html# diff includes/upload/UploadBase.php includes/upload/UploadBase.php.bak
447,455c447,454
< // ## Avi Commented this out so that we can upload whatever we like to our server. That was nice of him
< //            // Check whether the file extension is on the unwanted list
< //            global $wgCheckFileExtensions, $wgFileExtensions;
< //            if ( $wgCheckFileExtensions ) {
< //                    if ( !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) ) {
< //                            $warnings['filetype-unwanted-type'] = $this->mFinalExtension;
< //                    }
< //            }
< //
---
>               // Check whether the file extension is on the unwanted list
>               global $wgCheckFileExtensions, $wgFileExtensions;
>               if ( $wgCheckFileExtensions ) {
>                       if ( !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) ) {
>                               $warnings['filetype-unwanted-type'] = $this->mFinalExtension;
>                       }
>               }
>
557,570c556,569
< // ## Avi Commented this out so that we can upload whatever we like to our server. That was nice of him
< //            /* Don't allow users to override the blacklist (check file extension) */
< //            global $wgCheckFileExtensions, $wgStrictFileExtensions;
< //            global $wgFileExtensions, $wgFileBlacklist;
< //            if ( $this->mFinalExtension == '' ) {
< //                    $this->mTitleError = self::FILETYPE_MISSING;
< //                    return $this->mTitle = null;
< //            } elseif ( $this->checkFileExtensionList( $ext, $wgFileBlacklist ) ||
< //                            ( $wgCheckFileExtensions && $wgStrictFileExtensions &&
< //                                    !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) ) ) {
< //                    $this->mTitleError = self::FILETYPE_BADTYPE;
< //                    return $this->mTitle = null;
< //            }
< //
---
>
>               /* Don't allow users to override the blacklist (check file extension) */
>               global $wgCheckFileExtensions, $wgStrictFileExtensions;
>               global $wgFileExtensions, $wgFileBlacklist;
>               if ( $this->mFinalExtension == '' ) {
>                       $this->mTitleError = self::FILETYPE_MISSING;
>                       return $this->mTitle = null;
>               } elseif ( $this->checkFileExtensionList( $ext, $wgFileBlacklist ) ||
>                               ( $wgCheckFileExtensions && $wgStrictFileExtensions &&
>                                       !$this->checkFileExtension( $this->mFinalExtension, $wgFileExtensions ) ) ) {
>                       $this->mTitleError = self::FILETYPE_BADTYPE;
>                       return $this->mTitle = null;
>               }
> 

And this just stops Setup.php making-safe the $wgFileExtensions array by removing whatever's in $wgFileBlacklist from it, which I think wouldn't complain had I not already done Bad Things to those two variables, but it's late and it can't hurt to turn this off, too:

wiki:/home/wiki/public_html# diff includes/Setup.php includes/Setup.php.bak
296,298c296,297
< // ## Avi Commented this out so we can upload whatever we like to our server. That was nice of him
< //# Blacklisted file extensions shouldn't appear on the "allowed" list
< //$wgFileExtensions = array_diff ( $wgFileExtensions, $wgFileBlacklist );
---
> # Blacklisted file extensions shouldn't appear on the "allowed" list
> $wgFileExtensions = array_diff ( $wgFileExtensions, $wgFileBlacklist );
 

I know that one of the wonderful features of these photo managers is that you can tag photos, and obviously a photo can be in more than one tag rather more easily than it can be in several directories. That said, all photo managers seem to have decided that an easy, fast way to tag photos isn't what's needed.

Additionally, shotwell's got this weird thing for hiding the fact that there's a filesystem from you, and I can't find any way to tag files by directory. So I've poked around the database and written a script to do it, which is below and here and pasted below in case I change my mind about file hierarchies later.

The oddest bit is the way the filenames are linked to the tags. The TagTable table has a field `photo_id_list` which contains a list of photo IDs in a format that I've not found anywhere else in the (admittedly not very extensive) db.

They're created by taking the id of the image (its value in the `id` field of the PhotoTable table), converting it to a hex value, padding it out to 16 characters with leading zeroes, and then concatenating it onto the string 'thumb':

my $hexPhotoId = sprintf("%x", $photoId);
my $thumbString = "thumb".sprintf('%016s', $hexPhotoId);

Anyway, the script's a bit simple because bash is quite good at handling loads of files; usage is like this to tag the contents of ~/Pictures/2011-france/ with the tag 'morzine':

avi@brilliant:~$ find ~/Pictures/2011-france/ -type f -exec ./shotwell-tag {} morzine \;
Creating tag morzine
tagged /home/avi/Pictures/2011-france/R0012810.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012850.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012911.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012931.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012921.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012794.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012883.JPG with morzine
tagged /home/avi/Pictures/2011-france/R0012881.JPG with morzine

I've no idea if it breaks anything - I wrote it about an hour ago, have tagged ~500 photos with it since, and Shotwell doesn't seem to be annoyed. YMMV. Here's the script:

 
#! /usr/bin/perl
 
# shotwell-tag
#
# Tags files specified by filename in shotwell. Handy for
# getting round shotwell's attempts at hiding the filesystem.
#
# Avi 2011
 
use strict;
use DBI;
 
my $file = shift;
my $tag = shift;
 
if ($tag !~ /.+/){
        print "Usage:\n\n\tshotwell-tag [file] [tag]\n\n";
        print "Tags [file] with [tag] in shotwell's db\n";
        exit 1;
}
 
my $dbfile = $ENV{'HOME'}."/.shotwell/data/photo.db";
my $dbh = DBI->connect("dbi:SQLite:dbname=$dbfile","","");
 
# Each tag has a string of photo 'ids'. These are generated
# by taking the ID of the photo from PhotoTable, representing
# it in hex, padding that out to 16 characters with leading
# zeroes and then appending it to the string 'thumb'
my $sth = $dbh->prepare("select id from PhotoTable where filename='$file'");
$sth->execute();
my $row = $sth->fetch;
my $photoId = $row->[0];
unless($photoId =~ /\d+/){print "$file is not in shotwell library\n"; exit 0;}
my $hexPhotoId = sprintf("%x", $photoId);
my $thumbString = "thumb".sprintf('%016s', $hexPhotoId);
 
$sth = $dbh->prepare("select id from TagTable where name='$tag'");
$sth->execute();
$row = $sth->fetch;
my $tagId = $row->[0];
unless($tagId =~ /\d+/){
	print "Creating tag $tag\n";
	my $sth = $dbh->prepare("insert into TagTable (name) values('$tag')");
	$sth->execute;
}
 
$sth = $dbh->prepare("Select photo_id_list from TagTable where name='$tag'");
$sth->execute();
$row = $sth->fetch;
my $photoList = $row->[0];
if($photoList !~ /,$/ && $photoList =~ /._/){
	$photoList.=',';
}
if($photoList =~ /$thumbString/){
	print "$file is already tagged with $tag\n";
	exit 0;
}else{
	$photoList.=$thumbString.',';
	$sth = $dbh->prepare("update TagTable set photo_id_list = '$photoList' where name='$tag'");
	$sth->execute;
	print "tagged $file with $tag\n";
	exit 0;
}
 

So, I've now got a postfixadmin system that stores cleartext passwords, and in case you want to do it, too, I've put a patch up. It gives you an extra couple of options in the config.inc.php file, which I hope are well enough explained by the comments:

 
// cleartext
// Do you want to store cleartext passwords for email accounts?
// true = store cleartext passwords (need to have a password_clear column in the mailbox table)
// false = don't store cleartext passwords
$CONF['cleartext'] = false;
// and the same for admins:
$CONF['cleartext_admin'] = false;
 

If you do want this (and are aware of the problems with storing cleartext passwords) it's quite easy to do. First, add a couple of columns to the MySQL db:

 
ALTER TABLE mailbox ADD `password_clear` varchar(255);
ALTER TABLE admin ADD `password_clear` varchar(255);
 

Next, apply my patch:

avi@amazing:/var/www/postfixadmin$ wget -q http://avi.co/stuff/postfixadmin_plaintext-passwords.txt
avi@amazing:/var/www/postfixadmin$ patch < postfixadmin_plaintext-passwords.txt

Lastly, configure it; my patch sets both the config variables to 'false' because I like safeguards like that :)

It's worth noting that if you're using cleartext passwords, and then turn it off, the cleartext columns wont be affected - you'll need to update them with nulls or something if you want to get rid of the data in them. ]]> http://b.avi.co/postfixadmin-with-clear-text-passwords/feed/ 0 http://b.avi.co/fail2ban-and-date-formats/ http://b.avi.co/fail2ban-and-date-formats/#comments Tue, 04 Jan 2011 16:53:56 +0000 avi http://b.avi.co/?p=735 [...]]]> Fail2Ban is utterly daft in at least one respect. Here's me testing a regex on a date format it doesn't recognise:

# fail2ban-regex '2010-12-14 15:12:31 - 80.87.131.48' ' - <HOST>$'
Found a match but no valid date/time found for 2010-12-14 15:12:31 - 80.87.131.48. Please contact the author in order to get support for this format
Sorry, no match

And on one that it does:

fail2ban-regex '2010/12/14 15:12:31 - 80.87.131.48' ' - <HOST>$'

Success, the following data were found:
Date: Tue Dec 14 15:12:31 2010
IP  : 80.87.131.48

Date template hits:
0 hit: Month Day Hour:Minute:Second
0 hit: Weekday Month Day Hour:Minute:Second Year
1 hit: Year/Month/Day Hour:Minute:Second
0 hit: Day/Month/Year:Hour:Minute:Second
0 hit: TAI64N
0 hit: Epoch

Benchmark. Executing 1000...
Performance
Avg: 0.10257935523986816 ms
Max: 0.125885009765625 ms (Run 8)
Min: 0.10085105895996094 ms (Run 780)

Ignoring for the moment the fact that it doesn't recognise 2010-12-14 15:12:31 (Seriously?)¹ , the only way to get that list of date formats is by happening to pick a correct one. As soon as you no longer need a list of date formats you may use, it presents you with one.

What?

So, as an attempted fix for this situation, see above for a list of compatible date formats.

1. It's worth noting, too, that the author is of the opinion that specifying your own date format is too much like hard work, so if you want support for any date format other than those already supported, you've to patch it yourself. Which is obviously way easier than just having a date regex in the config file

Also in favour is the fact that Wikileaks is being a bit of a pain to a few institutions (well, governments) that annoy the crap out of me; I'd not mind being part of that. In addition, the huge majority of the released cables appear to be of no interest whatsoever, and the large governmental opposition to them has only served to increase the perception of their importance. I'd like as many people as possible to be able to read them such that they can judge for themselves how interesting they are. The point appears to be less what's been found out and more that anything has at all.

But I have concernes, too. Firstly, these cables were all sent on the basis they were confidential, so they naturally contain the sort of information that neither end wants made public. I'm already livid at the apparent acceptance of just anybody being able to subject me to surveillance, and I don't see why embassy staff should necessarily be treated differently. The argument that they work for the government is moot - millions of privately-employed people do work for the government, and they also should have a right to an expectation of privacy. I honestly have no problem at all with governments talking to each other in privacy, it seems to be quite a natural way of working and is not at all contrary to the idea of an open government.

Second, and of more concern than that, is the sort of things these people are likely to be sticking down encrypted tunnels. I don't want to inadvertently find myself hosting a document that results in an informant being tortured or killed. I don't really want to be party to releasing information that only serves to embarras or otherwise compromise someone. I don't want *anyone* to do that, but I've only got control over my servers.

That's all well and good, you say. Wikileaks are sifting through these and specifically redacting anything they deem not fit for release. That's some hubris right there.

And here's the difference. I trust the Debian project, and Canonical, the Perl foundation, Zend and the like, to not put things I disagree with on my server. I do not trust Wikileaks in this respect at all.

The whole 'Collateral Murder' release is a great example of Wikileaks not releasing information for the sake of it being free, but releasing specifically compromising information, with a decidedly skewed context, in order to further some particular viewpoint. That video, or perhaps its commentary, removed the bulk of my respect for Wikileaks. Why on earth would I assume they're not going to similarly skew the releases here also? Wikileaks does have a stated aim they're pursuing with all the leaking; it's not just because they feel information should be free.

So, it's not that I've got some opposition to the leaking, or feel that it shouldn't be mirrored. It's just that I don't feel I can trust Wikileaks to only publish what I think should be published, and picking-and-choosing which bits to host is not how a mirror works.

I can't find a reboot-and-hold-down-X-key method of rebooting into recovery mode (the boot menu) that works, and it seems to be different for each variant of this device in any case. Using adb, the Android debugger, does work, and contrary to several scare stories doesn't require proprietary Samsung drivers.

adb's really easy to make work. First, install a jdk. You might do this differently if you're not running Debian:

root@debian:~# apt-get install sun-java6-jdk

While we're here, if you want to suggest a more interesting hostname, go for it. I'm having a bit of an imagination failure in that department.

Next, you need to grab the tarball of the Android SDK, extract it somewhere and make its tools subdir part of your $PATH. You probably should do most of this as some user that isn't root, but I was rather excited at the time:

root@debian:~# mkdir adb && cd adb
root@debian:~/#wget -q http://dl.google.com/android/android-sdk_r07-linux_x86.tgz
root@debian:~/# tar -xzf android-sdk_r07-linux_x86.tgz
root@debian:~/adb# ls android-sdk-linux_x86
add-ons  platforms  SDK Readme.txt  tools
root@debian:~/adb# export PATH=${PATH}:/root/android-sdk-linux_x86/tools

Now (or perhaps while you're waiting for the tarball to arrive), enable USB debugging on your phone. It's under Settings -> Applications -> Development for some reason. Check the box next to "USB debugging". You'll need to have the USB cable unplugged. Plug it back in again, then, returning to your shell with adb in its path, check for the presence of your device:

root@debian:~/adb# adb devices
List of devices attached
90006e8ba84e    device

If yours doesn't show up, I'm not really sure what to do. Google?

Now, you need to have the update.zip somewhere. I've uploaded the one I've used to here but this'll work for any of them.

root@debian~/adb# wget -q http://aviswebsite.co.uk/stuff/galaxys_root/update.zip

If UMS works on yours, copy update.zip with your favourite file copying method. Mine didn't, so I used adb. There are two sdcards in the Galaxy, an internal one and an external (removable) one. The internal one is mounted at /sdcard, the traditional location of removable ones, and the external one at /sdcard/sd. You want to put update.zip in /sdcard, not /sdcard/sd.

root@debian:~/adb# adb push update.zip /sdcard/update.zip

You then use adb to reboot into the recovery menu:

root@debian:~/adb# adb reboot recovery

Select "Apply sdcard:update.zip" and wait while it installs it, then "reboot system now". You now have root. The quickest way I can think to test it is to download and install the 'superuser' application from the market, then test it with adb:

root@debian:~/adb# adb shell
$ su
#

You'll get prompted (on the phone) to allow an unknown application root access, and then you'll have root. Congratulations, your phone is now yours. :)

Now, I'm off to follow the rest of How to make the vibrant software not suck, 'cause it's shocking out of the box.

But MS have decided for some reason that the best way to do this is is render it as a web page in IE8. So before making an unbiased decision with no leading questions, you've already had to configure IE8.

I was going to post about the crapness of IE8's multiple configuration dialogues, but this is more amusing.