∑ Back to DHTML Tutorial.

HTML Correctness and Validators

Xah Lee, 2008-12-28

Some notes about html correctness and html validator.

Condition Of Website Correctness

My website “xahlee.org” has close to 4000 html files. All are valid html files. “Valid” here means passing the w3c's validator at http://validator.w3.org/. Being a programing and correctness nerd, correct html is important to me. (correct markup has important, practical, benefits, such as machine parsing and transformation, as picked up by the XML movement. Ultimately, it is a foundation of semantic web.)

In programing lang communities, the programer tech geekers are fanatical about their fav lang's superiority, and in the case of functional langs, they are often proud of their correctness features. However, a look at their official docs or websites, they are ALL invalid html, with errors in just about every 10 lines of html source code. It is fucking ridiculous.

In the web development geeker communities, you can see how they are tight-assed about correct use of HTML/CSS, etc, where there are often frequent and heated debates about propriety of semantic markup, while totally, absolutely, ignore any practical issues as if real world doesn't exist, and they don't hesitate to ridicule Microsoft Internet Explorer browser (which is the first browser to drag Netscape out of proprietary tags back in ~1996), and sneer at the average HTML coders. However, a look at the html they produced, also almost none are valid.

The bad html also happens in vast majority of docs produced by organization of standards, such as the Unicode Consortium, IETF. For example, if you run w3c's validator on IETF's home page, there are 32 errors, including “no doctype found”, and if you validate unicode's http://www.unicode.org/faq/utf_bom.html, there are 2 errors. (few years ago, they are much worse. I don't think even “w3.org”'s pages are valid back then.)

In about 2006, i spent few hours research on what major websites produce valid html. I found only one major site that produces valid html, and that is Wikipedia. This is fantastic. Wikipedia is produced by MediaWiki engine, written in PHP. Many other wiki sites also run MediaWiki, so they undoubtfully are also valid. As far as i know, few other wiki or forum software also produces valid html, though they are more the exceptions than norm. (did try to check 7 random pages from “w3.org”, looks like they are all valid today.)

Note

I love Wikipedia, and i found PHP to be practically one of the best language. For essays related to Wikipedia, see:

For essays related to PHP, and python, and html correctness, see:

Personal Need For Validator

My personal need is to validate typically hundreds of files on my local drive. Every month or so, i do systematic regex find-replace operation on a dir. This often results over a hundred changed files. Every now and then, i improve my css or html markup semantics site wide, so the find-replace is on all 4000 files. Usually the regex is carefully crafted to contain extra patterns to ensure correctness of intended match, or done in emacs interactively, so possible regex screwup is minimal, but still i wish to validate by batch after the operation.

Batch validation is useful because, if i screwed up in my regex, often it will ends up with badly formed html, so html validation can catch the error.

In 2008, as a experiment, i converted a few of my projects on my site from html 4 transitional to html 4 strict. The process is quite a manual pain, even though the files i start with are valid.

Here are some examples. In html4strict:

Lets look at the image tag example. You might think it is trivial to transform because you can simply use regex to wrap a “<div>” to image tags. However, it's not that simple. Because, for example, often i have this form:

<img src="pretty.jpg" alt="pretty girl" width="565" height="809">
<p>above: A pretty girl.</p>

The “p” tag immediately below a “img” tag, functions as the image's caption. I have css setup so that this caption has no gap to the image above it, like this:

img + p {margin-top:0px;width:100%} /* img caption */

I have the “width:100%” because i have “p” set to a limited width “width:80ex” for reading. (my website is dominated in a essay format)

Now, if i simply wrap a “div” tag to all my “img” tags, i will end up with this form:

<div><img src="pretty.jpg" alt="pretty girl" width="565" height="809"></div>
<p>above: A pretty girl.</p>

Now this screws up with my caption css, and it is not possible for css selector to match “<p>” that comes after a “<div><img ...></div>”.

Also, sometimes i have a sequence of images. They are rendered side by side from left to right. Wrapping each with a “div” would put them vertically.

This is just a simplified example. In short, converting from html4transitional to html4strict while hoping to retain appearance or markup semantics in practical ways is pretty much a manual pain. (the ultimate reason is because html4transitional is far from being a good semantic markup. (html4strict is a bit better))

Validators

In my work i need a batch validator. What i want is a command line utility, that can batch validate all files in a dir. Here are some solutions related to html validation.

I heavily relie on the above 2 FireFox tools. Every time i create or edit a page, then view in browser, the validity icon will tell me if the file is not valid. However, the FireFox tools do not let me do batch validation. (which is needed when i do massive regex operations on all files) Over the years i've searched for batch validation tools. Here's some list:

One semi solution for batch validation i found is: “Validator S.A.C.”, at http://habilis.net/validator-sac/. It is basically w3c's validator compiled for OS X with a GUI interface. However, this is not designed for batch operation. If you want to do batch, i run it like this: /Applications/Validator-SAC.app/Contents/Resources/weblet ‹html file path›. However, it output a whole report in html on the validation result (same as the page you see in w3c validation). This is not what i want. What i want is simply for it to tell me if a file is valid or not. So, in order to use “Validator SAC” to do batch job, i wrap a perl script, which takes a dir and simply print any file's name if it is invalid.

Here's the perl script:

# perl

# 2008-06-20
# validates a given dir's html files recursively
# requires the mac os x app Validator-SAC.app
# at http://habilis.net/validator-sac/
# as of 2008-06

use strict;
use File::Find;

my $dirPath = q(/Users/xah/web/emacs);
my $validator = q(/Applications/Validator-SAC.app/Contents/Resources/weblet);

sub wanted {
  if ($_ =~ m{\.html$} && not -d $File::Find::name) {

    my $output = qx{$validator "$File::Find::name" | head -n 11 | grep 'X-W3C-Validator-Status:'};

    if ($output ne qq(X-W3C-Validator-Status: Valid\n)) {
      print q(Problem: ), $File::Find::name, "\n";
    } else {
      print qq(Good: $_) ,"\n";
    }

  }
}

find(\&wanted, $dirPath);

print q(Done.)

However, for some reason, “Validator S.A.C.” took nearly 2 seconds to check each file, in contrast, the FireFox html validator add-on took a fraction of a second while also render the whole page completely. For example, suppose i have 20 files in a dir i need to validate. It's much faster, if i just open all of them in FireFox and eyeball the validity indicator, than running the “Validator SAC” on them.

I wrote to its author Chuck Houpt about this. It seems that the validator uses Perl and loads about 20 heavy duty web related perl modules to do its job, and over all is wrapped as a Common Gateway Interface. Perhaps there is a way to avoid these wrappers and call the parser or validator directly.

I'm still looking for a fast, batch, html validation tool.

∑ Back to DHTML Tutorial.
2008-12
© 2008 by Xah Lee.