The problem is that the “old archive storage” also has limited disk space and I got fed up managing the archive by hand. The solution I came up is to scan the hierarchy of  log files in the storage (logs are stored hierarchically according to origin and type) and delete old files until I have enough room to move some newer files in. That way the “old archive storage” is always kept full and keeps as much back-log as possible and does this automatically.

The piece of code that determines which files we want to delete works like this:

1. Use find to list all the files in the directory structure
2. Pipe it to perl and collect all the file names in a list
3. Use perl’s sort operator to compare the modification times of each file in the list and show them in the order (i.e. oldest first)
4. Use head to get just the first file

So it looks like this:

find /mnt/httpd_back/ -type f | perl -nle 'next unless -f; push @files, $_; END { foreach $file (sort { @a=stat($a); @b=stat($b); $a[9] <=> $b[9] } @files) { print $file; }}' | head -n1


Note: normally we use head to get some initial output and terminate the process early before it does more costly work – when head has enough data it terminates the pipe sending SIGPIPE to the upstream process and that usually terminates the process that generates the data. In this case – and in all other cases involving sort – the upstream process buffers all the data in its own memory before outputting anything, so it can sort everything, and using head here is just a filter to get what I want and does not actually save me from doing all the work. I could have easily done the same thing inside the perl script itself by replacing the block of  print $file; with print $file; last; – this has the same effect as using head, because head will send SIGPIPE to perl after getting the first print and will terminate it. Deciding which way you want to go is probably more about readability of the code and I prefer my original version because its easier to read to non-perl specialists.

I can then just remove that file, see if I have enough room to move in the newer log file and if no – repeat the process.

This would work well, I believe, but it may be inefficient if I find a bunch of small files and I want to copy in a large file. So what I did next is to take advantage of the fact that all the log files I have are named using the following simple format:

<service>-<type>_log-<year><month><day>.gz

and that allows me to easily find all the log files that record the same day and eliminate them at the same time. Subsequent moving of additional files will likely succeed because I cleared out all the log files of an entire day. If not, I can always go and clear up another day’s worth of logs.

It was obviously way to geeky for the task at hand, so here it is now for you to answer – or better yet: invent your a worst geeky personality question:

If you’d have been a design pattern (or an anti-pattern), which design pattern you’d be and why?

I, for laughs, I’d say I would have been a Duff’s device – complicated and impossible to debug

This is in Internet Explorer 8 when set to IE7 mode, but this is very faithful to the original as I’ve tested it on a real IE7 and it behaves the same. What happens here is that when you scroll the page down, all absolutely positioned elements (the two “combo boxes” which are a custom UI widget and the text “Dimensions:… at the top right corner”) get pulled down a few pixels. When you scroll up, they get put back in the correct place.

I would (maybe) understand if that would have happened only for the custom UI widgets (as there is a lot of Javascript code in place to support these, and I would understand if it behaves funny on broken browsers), but it affects the absolutely positioned text exactly the same, and there is nothing funky with that – its simply a block of text that is absolutely positioned.

On a slightly different subject, here is a useful tip for doing using Javascript to manipulate elements in IE7: if you have some Javascript that acts immediately after the page loads and put more content on the page and moves some elements around, if any of these elements are absolutely positioned then IE7 will ignore any changes to their layout, such as setting margins (especially negative margins) and such. This happens if you both change the element’s layout implictly (by adding content, for example) and explicitly (by setting style attributes) in the same Javascript call – the style attributes are ignored.

What you should do is to change the content in one Javascript call, then let the layout settle by exiting back to the rendering engine and then coming back to Javascript to change the layout. I do it like this:


$('block').innerHTML = 'Some text that should be updated';
window.setTimeout(function() {
$('block').style.marginTop = '-20px';
},0);


The window.setTimeout() with a zero miliseconds timeout will cause the code to stop executing without doing the style manipulation, the Javascript machine hands the execution back to the browser which then lets the rendering engine repaint the screen with the updated content – the problem with IE7 is that the rendering engine fails to take into account style changes that were made after content changes were done in the same Javascript execution, so we don’t do that. As soon as the rendering engine is done – less then one milisecond later – the timeout kicks in and adjusts the style, and finally handing the execution back to the rendering engine to adjust that as well.

You can find Kohana RPM for the current stable release 3.0.5 here, and the source RPM is available here in case you want to rebuild it yourself (and you might, details follow). New releases to correspond with new releases from Kohana will be updated there as needed.

This package is built on a CentOS 5.4 machine, with pretty much default settings.

Please note that Kohana is not tested with the PHP version 5.1.6 which is what RHEL/CentOS 5 uses by default, so there could be some problems with using it on such a default installation of the operating system. There was one place where Kohana uses a function in a way that is not supported on PHP 5.1.6, and I had to patch that with a workaround.

This package includes additional patches that I felt where necessary, some to work around Kohana issues and some to add functionality that I felt is missing or or lacking. If there is interest I can expand on the work there. I do intend to submit all patches upstream, of course – once I’m more familiar with the community process of Kohana.

In the mean time, you can try it out and let me know if it works for you or not.

Related articles by Zemanta

• Top 5 Free Linux Distributions for Servers in 2010 (taragana.com)

But I actually found out about Go from a friend that asked about it, didn’t mention Google, and gave me a link to the home page at GoLang.org where Google is not mentioned. So my review below was not influenced by any Google hype, and on the face of it, after reading through about half of the tutorial and a couple of other documents, I don’t like it.

It is basically a C variant with garbage collection, some support for objects, a different type system (possibly better, definitely simpler then C++’s)  and internal support for threading and asynchronous IPC. I don’t like the fact that it has “native types” and “reference types” but requires the developer to know the difference and care about it (use new() vs. make(), Slices vs. Arrays, etc.) unlike Java which has similar issues but cares much less. Go is obviously built to compile fast and run fast with minimal considerations for coherent programming, and also to play into a few select people limited sense of “easy to program” – and the contrived syntax does not help.

Go also tries to adopt some useful features that were introduced in dynamic languages such as tuples, functions as 1st class values and closures but in this C variant this only makes the syntax more contrived and less understandable, possibly due to the designer’s aversion from using parenthesis to group stuff.

I can see how it could be used to write small services instead of writing them in C or C++, and the benefits of that, but I would be very careful writing large software projects in this language, or small system administration tools.

The problems Go presents for large software development are in code reuse: lacking any sort of inheritance (not even Ruby-style mixins, though those would be obvious here) providing multiple implementations of an interface is going to be a pain. It looks like they prefer to implement polymorphism using services (as APIs or communication protocols) that reduce code duplication by importing functionality from packages – hardly what I’d consider sane for a large software project. Using flat packages (a-la C) for managing code is doable but allows developers to easily make a lot of mess and if I was designing a programming language with support for better code management I would have done it differently.

The problems Go presents for small system administration tools is that its overhead – while slightly lower then C or C++ – is still much higher then Perl and other dynamic languages (Ruby, Javascript, python) and possibly even higher then Java.

Its only benefits I see of Go are built-in multi-processing semantics with asynchronous communications, and possibly the small footprint which I guess would only interest people that have limits on the hardware they can deploy (small embedded systems, mobile devices, etc.). If you are planning to need these things, then by all means – go ahead and learn it, otherwise I suggest putting time into a more mature language with better semantics – like Ruby. It may be slower performing but it will allow you to deploy software faster and maintain it better.

After I’ve seen the Google blog post, I’d also like to comment that this new programming language came from several rather well known people:

• Rob Pike is one of the fathers of UNIX, author of programming books and well known hater of object oriented design.
• Ken Thompson is also one of the fathers of UNIX and responsible to a large degree for C (and many of its horrors).
• Russ Cox is like the previous two one of the people responsible for Plan 9, the failed UNIX sequel from Bell Labs.
• And a few other folks who with a quick Googling I did not find any interesting information about them other then the fact that they work for Google .

All in all, a respected group that I feel will allow the Go programming language develop into a mature programming language on its own. But at the moment, I suggest staying away from it.

Actually, I wasn’t surprised to find about a dozen different questions in the same gist, but here is the one that I really liked, especially some of the more interesting answers: How many lines should a method typically have?.

And to quote one of the answers that I really liked:

It should fit in my head.

That is to say, if I put my head against the monitor, the method should not be seen as extending past the boundaries of my head.

So what do you think should a method’s size be?

The way I see it – in the simplest terms – a function or method should be only as long to do one logical operation and no more. That being said, if your one logical operation is to large to fit on the screen, then something is very wrong with your logic .

Say you want to check which JAR files (or ZIP files for that matter, as Java ARchive files are just ZIP files with a different extension) contain files that contain some text. grep is the obvious answer, but how to grep files in JARs?

A simple loop can go over the archives you are interested in, then unzip -c can extract the content of files to the standard output so that grep can be used on them:

pattern="smtp"
for jar in /usr/share/java/*.jar; do
    for file in $(unzip -l $jar | perl -nle 'split /\s+/ and print $_[-1];'); do
        unzip -c $jar $file | grep -qi "$pattern" && (
            rpm -qf $jar 2>&1 >/dev/null && \
                rpm -qf $jar || \
                echo "orphan jar: $jar"
        );
    done
done | sort -u

This could also be written as single line in the terminal, obviously.

My version of the loop’s internal code looks at matching files and prints out the RPM they belong to or proclaim them to be orphaned files.

An alternative loop code might simply print the matching output and the file that contained it, a-la grep -H:

output="$(unzip -c $jar $file | grep -i "$pattern")"
[ -n "$output" ] && echo "$jar, $file: $output"

Although there is an obvious bug in that last code example – see if you can figure it out.

For the past couple of week I’ve been trying to build a new version for the Jetty package based on the current Jetty6 package from JPackage, and in the process combating its hellish dependency tree and the way open source Java projects build opon each other in a complicated, confusing and often circular manner.

Examples for circular dependencies are varied and not that interesting – In order to build Jetty you need some third party libraries, which in order to build them you need a provider of Servlet API 2.4, of which there are several candidates including the current version of Jetty (you need to build Jetty6 in order to be able to build Jetty6), the previous version of Jetty (you need to build Jetty5 in order to build Jetty6 – a bit less silly but still silly) or Tomcat (you need to build the competitor to Jetty in order to build Jetty – quite silly even if its open source we are talking about).

This is not the worst circular dependency – they were others far worst last time I tackled something like this (a couple of years back), like you need to have Jakarta’s commons-httpclient installed in order to be able to build Jakarta’s commons-httpclient. Luckily it appears they got that fixed.

Update:

Continuing to work on this, here is another problem I encountered: In order to build Jetty 6 you need wadi 2, which in order to build you need Maven 2 and specifically the Mojo Maven 2 plugin for javacc, which in order to build you need Xfire, which in order to build you need Jetty 6. Lather, Rinse, Repeat.

But what prompted this rant was another very common problem with Java open source project – feature creep in the worst kind of way: I’m trying to build the JAXB package, which is a simple interface to parse and build XML. It depends on args4j which is described as “a small Java class library that makes it easy to parse command line arguments”. First I’ve heard of it but oh well, lets try to build it. Well, apparently in order to build it it requires Saxon which is an XSLT processor. why ?!?

Noticed the “small” part that I highlighted earlier? In my dictionary “small” includes many properties, one of which is “Does not have large or complex external dependencies”.

And you get this all the time when trying to build Java libraries and applications – up until now I had to install 2 versions of groovy, 2 versions of saxon, 3 versions of the Java development kit itself and I had to build and install Xalan twice (once for bootstrapping and a second time to actually get the xsltc processer I needed).

A few years back I ranted about this in an email to JPackage maintainer Ralph Apel at which point he replied that this is the best they can do to fight the insanity in the upstream.

Anyway, apparently searching the web for “Linux remote shutdown” yields no useful results (or so I’ve been told), but frankly – when you have standard UN*X tools at your fingertips, a remote shutdown tool is simply typing ssh root@server shutdown -r now at your local console. But still, for people who want a “tool” – read on.

If you want to control multiple servers, it does gets a bit more tricky – you don’t really want to start sshing to server1, start the shutdown, ssh to server2, start the shutdown, lather, rinse, repeat.

So lets suppose you have a file with a list of all your servers. I’m assuming its a text file that looks something like this:

server1    1.2.3.4
server2    1.2.3.5

So here is a little something that scans such a list, calls each server to shut it down and waits 30 seconds for the server shutdown to complete. The assumption here is that you have exchange keys with each server so you can login as root to each server without the need to type in a password. If this is not the case, this script will attempt to publish your DSA key to the remote server so it will work better next time .

Here is the script, with some inline documentation. Just copy and past it into a text file and make it executable. The server file is expected to be in a file named lan-servers.txt but its easy to change this.


#!/bin/bash

# where the server list sits
LAN_LIST="$HOME/lan-servers.txt"


# Function to check if the server responds to ping
function testServer() {
    if ping -c1 $ip>/dev/null 2>&1; then
	return 0
    else
	return 1
    fi
}

# get a list of IPs for the servers
function getServers() {
    cat $LAN_LIST  | egrep -v '^(#|\s*$)'  | perl -ple 's/\s+/ /g;' | cut -d' ' -f 2
}

# go over all servers
for ip in $(getServers); do
    # check if the server is alive
    if ! testServer $ip; then # if its not then its already down
	echo "Shutting down $ip: already down"
	continue; # to next server
    fi

    # check if I can log in to the server with no password
    if ! ssh -o "NumberOfPasswordPrompts 0" root@$ip '/bin/true' >/dev/null 2>&1; then
	# if I can't, try to do a key exchange (this will require an interactive password prompt)
	cat $HOME/.ssh/id_dsa.pub | ssh root@$ip 'mkdir -p .ssh; cat >> .ssh/authorized_keys'
	# check if I can now log in to the server with no password
	if ! ssh -o "NumberOfPasswordPrompts 0" root@$ip '/bin/true' >/dev/null 2>&1; then
	    # if I still can't then something is broken
	    echo "Failed to complete key exchange with $ip - skipping this server"
	    continue;
	fi
    fi

    # start to shutdown the server
    echo -n "Shutting down $(ssh root@$ip hostname)"

    # halt the remote server
    ssh root@$ip halt
    count=0
    # wait until its down or 30 seconds have passed
    while testServer $ip; do
        sleep 1; echo -n "."
        count=$(( $count + 1 )) # count up to 30
        if [ "$count" -gt 30 ]; then # if the server did not shutdown in 30 seconds, something is broken
    	    echo -n "Server did not shutdown properly!"
	    break;
	fi
    done
    echo ""
done

And thats it. Of course if you really want an application we can complicate it a lot more if we want – add a GUI, some command line switches, a web interface – what not. But the basic functionality is something that a sysadmin should be able to hack in a few minutes.

This story highlights one issue I have with MS-Windows: you need special apps to handle all kinds of simple administration tasks – for example my Brother uses a special program for batch renaming of files. This functionality is completely unneeded on Linux because your simple shell is so much more powerful and an advanced user can write much more complex renaming commands then what a GUI can offer, using simple bash commands.

The point is that some computer administration tasks are simple enough that you don’t need special tools – but what is considered “simple” is subjective also depending on your OS of choice. With MS-Windows simple stops at about the process of moving an entire directory tree. With Linux simple can be much much more interesting.


if ( (isset($tid)) && (!isset($pid))) {
// do something
} elseif ( ($mainfile) xor ( ($pid==0) AND (!isset($pid)))) {
// do something else
} else {
if(!isset($pid)) $pid=0;
// do a third thing
}


כל הקוד הזה מטופש, אבל הכי משך את עיני החלק האחרון של ה-if השני: “אם pid שווה לאפס וגם אינו מוגדר” .

אני רואה את ה-Script Kiddie שכותב את הקוד הזה – מוסיף תנאי ועוד אחד, וזה עדיין לא עובד כמו שצריך, אז נוסיף עוד תנאי פה ועוד תנאי כאן ואולי קצת סוגריים, עד שזה יעבוד.

זה לא שחסרה למרבית מתכנתי ה-PHP השכלה בסיסית במדעי המחשב – חסרה להם יכולת בסיסית לקרוא מדריכים, להבין אותם ולהסיק מסקנות, או חמור מזה – יכולת להסתכל על קוד ולהגיד “איכס! אפשר לכתוב את זה יותר טוב”.