Both sets of problems had the same cause, which I only found once I turned error_reporting up to E_ALL:  iconv() or related iconv extension functions failing to return any output, and throwing a Notice-level error usually saying something helpful like “Notice: iconv(): Wrong charset, conversion from `’ to `UTF-8′ is not allowed” where UTF-8 is the charset used by the Pdf or Lucene module.

Normally these would be caught during development, where our environments are configured to display all errors, even Notices.  But this only happens on my production servers so it was not caught until functionality started failing in production.  The moral of that sub-story:  something about Solaris maybe, I’m not sure if there even is a moral there.

At any rate, on the problematic Solaris servers, calling iconv() with an empty first parameter $in_charset causes the above Notice, and the function returns False.  The parameter is empty because I did not specify a value for the optional charset parameter to methods like Zend_Pdf_Page::drawText() and Zend_Search_Lucene_Field::Text().

Personally I think it’s a bit of lazy-ish coding in ZendFramework which allows that parameter to be empty in the first place; behavior for iconv() is not defined in the documentation for an empty input charset.  But I can’t fault them in the end since this behavior seems to be so highly isolated and isn’t even replicatable among our development and QA servers.  So far my only remedy has been to find every single place that an iconv function is used in the ZF codebase, and make sure that the variable for the input encoding, if empty, gets replaced with the value of the iconv.input_encoding config variable.

We’re currently using the latest “gold” version of ZF as of about 10/1/2010, and I have patched the source to eliminate the problem from both the Pdf and Search modules.  If you would like a cvs patch or gnu patch to apply those changes feel free to contact me via email.

Apache’s custom log format shows how to log variables from the Apache environment.  If there’s an environment variable called FOOBAR, you log it with the format string %{FOOBAR}e.  This is great because PHP can inject data into the Apache environment, meaning we can get more or less any data we like into an Apache custom log.

The first Apache configuration step was to create a custom log format that we can then apply to a logfile.  We create the log format in httpd.conf.  Find the log_config_module section and add a LogFormat directive there:

<IfModule log_config_module>
LogFormat "%t %U %D %a %{MY_UID}e" mystats
</IfModule>

In this example, we add a log format named “mystats” which logs these datapoints:  date/time the request was received, URL path requested, time taken to serve the request, remote client IP address, and the environment variable MY_UID.  For our purposes, we are logging these data because our aim is to keep a 7-day rolling log of webserver performance – how long it took to process each request – to monitor our servers more effectively.

Then we set up a custom log for our site’s vhost in httpd-vhosts.conf:

<VirtualHhost 192.168.100.1:80>
...
CustomLog logs/mysite_custom_log mystats
...
</VirtualHost>

This makes Apache start logging every request to that host, in the format defined by our LogFormat directive named “mystats”.

The first step in PHP code, then, is to simply inject the MY_UID environment variable containing the user ID of whoever is logged in.  In our master controller there’s a check to see if the client has an authenticated session, and if they do we just add a quick call to set the environment var:

apache_setenv('MY_UID', $the_current_user_id);

Once this in all in place, you should be able to tail the logfile mysite_custom_log and see that requests are being logged with the requesting client’s user ID, or “-” if no user is authenticated.

We rotate these logs daily.  Our final step is a cron’d PHP script that runs in the wee hours, taking yesterday’s log file, iterating each line, inserting the data into a table… and we’re done!  We logged each request with how long our server processed it, and who was signed in for that request, and we did it without adding any overhead to our application.

Thanks, Habitat for Humanity of Oregon, for the opportunity to lend a hand!

Out of the box, though, its column sorting feature does not work unless your grid is hooked up to a server-side data source.  Here is a svn patch that adds this functionality to the current head branch of flexigrid.js:

flexigrid-sort-in-place.diff

This patch is built on the current trunk revision of flexigrid.js, which is r3.  I am currently unable to create a patch for the current release version (“1.0b3″), which is slightly different; but to apply these changes to that version is pretty simple, since I only made 3 changes:

1. add “sortorder: ‘asc’” to the default properties struct up at the top
2. add the “else if (!p.url) this.inPlaceSort();” clause to the end of the changeSort method
3. copy in the inPlaceSort() function

Nick: <typing carefully>

Nick:  <clicks button>

[some time passes]

Internet: “Oh, sorry, no trailing period? I’ll just assume you mean to send this
email to freakin Nepal. MORON.”

So you are hypothetically deploying your ant-built tar’d up software release. Let’s further say for the sake of argument that it contains some business-critical scripts that must keep their executable mode to be run from cron. Oops, those files are no longer executable! You will soon be disappointed at the failure of your cron job to do anything at all, and you may experience some increased email volume accompanied by mild irritation.  If this condition persists, call your doctor, or go back and RTFM again because as the docs imply, this shortcoming can be easily overcome.

Your job is to identify all the files that need to keep their mode, username, or groupname, and specify said property in a special fileset called tarfileset.  In my ant build, let’s say the build root is ${builddir} and contains 2 components called src and lib.  I’ve got one specific file src/foo/exec.sh, and a whole tree of stuff at lib/bar, that have to be executable.  I will need to call the Tar task with two tarfilesets: one to include the executables and set their mode, and a second to include everything else while excluding the executables.

Before:

<tar destfile="${buildfile.name}"
     basedir="${builddir}"
     longfile="gnu"
     includes="src/**, lib/**"/>

After:

<tar destfile="${buildfile.name}" longfile="gnu">
     <tarfileset dir="${builddir}" filemode="755">
         <include name="src/foo/exec.sh"/>
         <include name="lib/bar/**"/>
     </tarfileset>
     <tarfileset dir="${builddir}">
         <include name="src/**"/>
         <include name="lib/**"/>
         <exclude name="src/foo/exec.sh"/>
         <exclude name="lib/bar/**"/>
     </tarfileset>
</tar>

Works for me!

The other day, someone at Site5 decided they had to respond to a vague and specious rumor circulating about a vulnerability in OpenSSH.  Their response?  To disable all customer ssh access.  Wheeeee!

Yesterday they finally turned back on SSH access.  But without any notice, they disabled ssh1 access on my server.  I don’t dispute that I should probably have been using ssh2 keys for accessing my svn repo over svn+ssh, but I wasn’t.  And since I am sort of a doofus sometimes when it comes to sysadmin type stuff, I got to spend half a day looking for some problem with my svn client before I thought of the obvious fact that while Site5 claimed all ssh access was restored, that claim may not have been entirely true.  Thanks Site5, you totally owe me a beer.

Update – SANS said yesterday it was indeed a hoax.

The launch was nearly 100% smooth, the only exception being when I broke Habitat’s email for a half hour or so.  Turns out that if you move DNS for a domain, you move it for, you know, the whole domain, and stuff.  Who knew you could be a professional web developer while having such a crappy grasp of DNS?  Well, I’m living proof that it’s true.  Or false, depending on your definition of “professional”.

Update:  DNS caching is a giant PITA.

• PageSpot
• Media2Layout

So far today I have compulsively checked the download statistics and somehow 60+ people downloaded PageSpot within about 6 hours of it being published.  I know there are a lot of WordPress blogs and users out there but there must be even more than I assumed.