=begin sdf H1: Copyright Copyright 2000 by Robb Canfield. Released under the same terms as Perl Robb Canfield, Canfield Research Group, {{EMAIL robbc@canfield.com}} =cut package Exten::Table; # use Data::Dumper; # DEBUG use Carp; use HTML::Filter; @ISA = qw/HTML::Filter/; use strict; =begin sdf H1: parse_table Parses the table data embedded in the HTML string passed. The string passed should include all the HTML data B the table tags but B including the table tags! Uses C to extract headers. A row that starts with C ends the header extraction. Optionally pass a header count and that many data rows are converted to headers without regard to the column type. If no headers are found or the header count is 0 then no labels header is defined and all rows are considered data. Returns an array of each row of data. Each element contains the data for that column. Each possible column is always set. If a cell spanned one or more cells then spanned cells are set to undef(). This is different from empty cells which are assigned a blank string C<"">. The B element is special and contains the deduced type for the row of data. If any C elements were found then the row is labeled C
otherwise it is labled C. Note that multiple header rows can appear anywhere in the table. Example\: > [ > [HEADER, undef, Tempurature, undef, undef, Precipitation, undef, undef], > [HEADER, Place, Today, Tonight, Saturday, Today, Tonight, Saturday], > [DATA, Seattle, 51, 42, 49, 60, 100, 80], > [DATA, Everett, 50, 41, 48, 60, 100, 80], > ] Note that the first header row used colspan 3 which are represented by the two undefs following C and C. H2: Extensions The following data elements are added to the standard Filter object. These are all internal and should B be used by outside methods. All are placed under the key C: !block table Element Description table_active True if currently processing a table data Array for parsed table, retrieve via L<"table_data"> row_id Current row within the C element col_id Current col within the C element type Type (HEADER, DATA) of row skip_eof Skips all further parsing, used to prevent going beyond end of table column_count Set to the number of columns wide the table is, not counting internal columns jagged Set to TRUE if the table has uneven rows, normally automatically fixed by L<"_clean_up_table"> and changed to FALSE force_blank Another intanal variable, used to handle setting cells to undefine during C processing no_out Prevent output if TRUE, used to avoid output while inside a table but outside a cell pre_on True if the C
 HTML tage is active, used to know if spaces/newlines should be trimmed
!end_block

The following are global for the process
!block table
Element         Description
NEW_TEXT        New processed text, see L<"get_output">
TABLE_STACK     Contains pending table data (all data in above table) when processing nested tables
POST_PROCESS    Name/reference to method for table post processing - applies to all tables found during parsing
!end_block


H2: How to use

The simplist way to use this module is:
!block example
    use Exten::Table;

    my $filter = Exten::Table->new();
    $filter->parse_file($ARGV[0]);          # or $filter->parse(some-string)

    # Retrieve the formatted text
    $print $filter->get_result();
!end block

Since this module acts as a filter it has to have an input and an output. The input comes via
L<"parse"> or L<"parse_file"> and the output is placed within the object itself and retrieved
via L<"get_result">. If you want to control the table filtering more carefuly then overload
the L<"table_post_processing"> method with your own. Typically this is done to gain fine control
over the HTML page being scanned. Combined with the L<"parse"> method it becomes trivial to scan
the HTML page looking for a particular table, extract it, process it with the filter, post-process
it as desired, and replace it if needed.

Note: The L<"set_table_post_process"> can also be used to change the post-processing method, this
is often easier for simple code such as in the example below.


!block example
    # An example of a custom table post processing. This example scans the input
    # file for a table containing 'Precipitation' (as an example) and then
    # custom formats that table.

    use Exten::Table;

    # Define a simple post-process method
    sub my_post_process {
        my $self =  shift;

        # Do a header join and then dump the table using table_out. This
        # has a very different look than the hierarchial output.
        $self->join_headers();
        $self->drop_blank_columns;      # usually a good idea

        # User method returns a STRING that is then forwarded to
        # the L<"output"> method *internal processing). The L<"output"> can
        # be overloaded to accept and process the result of this method
        # as desired. See also L<"end"> the 'table' section.
        "
" . $self->table_out() . "
";
    }

    # Read a file into a string for ease of debugging.
    open(T, $ARGV[0])
        or die "Unable to open $ARGV[0], $!";


    my $tmp = join("\n", );

    # Massage string to make it easier to look for table. This is a trick I often
    # use to make the REGX faster, easier and more reliable.
    $tmp =~ s/()/$1\xFE/gi;

    # Scan each table looking for Precip table
    $tmp =~ s/\xFF([^\xFF\xFE]+)\xFE/
        $data = $1;                         # Remember it in case I need to use another REGX

        # If proper table is found, process it and use the output as the
        # replacement text.
        if($data =~ m'Precipitation') {     # There, I needed to use a REGX
            # this is the table I want, create a new object for it
            # and use my own post processing, easier to use this way than create another module
            # for overloading the default post processor.
            $parser = Exten::Table->new();
            $parser->set_table_post_process(\&my_post_process);
            $parser->parse($data);
            $parser->get_result();
        } else {
            $data;
        }
    /ge;

    $tmp =~ s/\xFF|\xFE//g;

    print $tmp;

!end_block


If you want to process the HTML at the tag level instead of filter then:
    * Process the HTML per above then re-process using your own module that
    uses the HTML::Parser or HTML::Filter module directory (Recomened)
    * OR overload the this module to add the functionaility you need. This is not
    normally the best way since the code can quickly become convoluted, but it
    is handy for quick and dirty fixes. Your choice.


=cut


=begin sdf

H2: Globals

Variables defined globably. These should probably be defined as part of the object
but they are here for either testing, experimenatation of laziness.

=cut

=begin sdf

H3: $Column_Seperator

HTML text that is used to seperate columns in a table. This is only used when there
are no headings for the table and the columns are two wide to place side by side.

=cut
#my $Column_Seperator = '
';
my $Column_Seperator = '
';

=begin sdf

H3: $Row_Seperator

HTML text that is used to seperate columns in a table. This is only used when there
are no headings for the table and the columns are two wide to place side by side.

=cut
#my $Row_Seperator = '
';
my $Row_Seperator = '
';

=begin sdf

H3: $Table_Start

HTML text that is used to mark the start of a table. If the output format can
handle tables then use C<
> otherwise set as desired. =cut #my $Table_Start = '
'; my $Table_Start = "
"; =begin sdf H3: $Table_End HTML text that is used to mark the end of a table. If the output format can handle tables then use C<
> otherwise set as desired. =cut #my $Table_End = '
'; # A little larger than Start, allows nested tables to be noticed (mostly for debugging) my $Table_End = ''; =begin sdf H2: Methods =cut =begin sdf H3: new Stub for creating a new object =cut sub new { my $class = shift; my $p = HTML::Filter->new(@_); $p = bless $p, $class; $p->{TABLE}{no_out} = 0; # output can occur by default $p->{TABLE}{data} = []; # Initialize to empty array $p->{NEW_TEXT} = ''; # Default output text to blank $p->{TABLE_STACK} = []; # Default is no tables on stack $p->set_table_post_process('_default_post_process'); # Allows for OO style calling $p; } =begin sdf H3: parse Overloads the parent's C method for future expansion. =cut sub parse { my $self = shift; $self->SUPER::parse(@_); $self; } =begin sdf H3: parse_file Overloads the parent's C method for future expansion. =cut sub parse_file { my $self = shift; $self->SUPER::parse(@_); $self; } =begin sdf H3: set_table_post_process Sets the table's post processing to the method name or reference passed. If a string is passed it is used as the method name for a standard object oriented call. The post porcess method method is called with the filter's object. Note that each new table encountered purges the previous so archive the data if you need to merge or work with more than one table at a time. Returns the previoust post process method. If undef is passed NO changes are done to the current post process (used for reading current). =cut sub set_table_post_process { my $self = shift; my( $method, ) = @_; my $old_method = $self->{POST_PROCESS}; if(defined $method) { $self->{POST_PROCESS} = $method; } $old_method; } =begin sdf H3: table_out Formats the table using by guesing its structure: * If there are no columns in the table then a blank string is returned * If there are no rows in the table then a blank string is returned * If the first line does not contain any headers then ** If one column, display as
** If only 2 columns, combine columns using C<' - '> and display each row on its own line using
** If more than 3 columns then display first column then indent using
    ,
  • for each subsequent column * If first line is a header: ** Combine all headers, see L<"join_headers">. ** Display first header name followed by a C<':
    '> then loop through all rows and display first column of data with each subsequent column is indented via C
      and C
    • with column name and data using C<': '> to concatenate the strings. The defaults can be altered via paramaters to this method. For more complex operations process the data structure directly. Returns the string with the new HTML code. undef() indicates an error occured. A blank string indicates an empty table. =cut sub table_out { my $self = shift; my $result = ''; # Result string my $text; # Return text from user call my $column_count = $self->column_count(); # No table, no output return '' unless $column_count; return '' unless $self->row_count(); # Check for any headers if($self->row_type() ne 'HEADER') { if($column_count == 1) { $result = $self->process_table( sub { my ($self, $type, $row) = @_; return '' if $type ne 'DATA'; $row->[0] = '' if ! defined $row->[0]; "$row->[0]
      \n"; }, ); } elsif($column_count == 2) { $result = $self->process_table( sub { my ($self, $type, $row) = @_; return '' if $type ne 'DATA'; foreach(@$row) { $_ ||= ''; } # Guess a nice length where the '-' doesn't make sense. # Most often triggered on wide tables, especially tables containing # other tables. if(length($self->text_only($row->[0] . $row->[1])) > 40) { "$row->[0]
      $row->[1]$Row_Seperator\n"; } else { "$row->[0] - $row->[1]
      \n"; } }, ); } else { $result = $self->process_table( sub { my ($self, $type, $row) = @_; return '' if $type ne 'DATA'; foreach(@$row) { $_ = '' if ! defined $_; } # Handle properly if nested within another table if($self->{TABLE}{table_active}) { join($Column_Seperator, @$row) . $Row_Seperator; } else { "$row->[0]
      • " . join("
      • ", @{$row}[1 .. ($column_count-1)]) . "
      \n"; } }, ); } } else { if($column_count == 1) { $result = $self->process_table( sub { my ($self, $type, $row) = @_; return '' if $type ne 'DATA'; $row->[0] = '' if ! defined $row->[0]; "$row->[0]
      \n"; }, 1, # Skip header column ); # Add column header $result = $self->get_row(0, 1)->[0] . ":
      \n$result"; } elsif($column_count == 2) { $result = $self->process_table( sub { my ($self, $type, $row) = @_; return '' if $type ne 'DATA'; foreach(@$row) { $_ = '' if ! defined $_; } # Guess a nice length where the '-' doesn't make sense. if(length($self->text_only($row->[0] . $row->[1])) > 40) { "$row->[0]
      $row->[1]
      "; } else { "$row->[0] - $row->[1]
      \n"; } }, 1, # Skip header column ); # prefix column headers $result = join(" - ", $self->get_row(0, 1)) . "
      \n$result"; } else { # Complex multi-column table # Build hierarchial column headers - takes less width than joining headers into a single string my $user_data = $self->headers_to_tree(); # now process remaining rows $result .= $self->process_table( sub { my ($self, $type, $row, $err, $format) = @_; return '' if $type ne 'DATA'; # Format the row using the user-data header, convert undefined into blanks # to avoid Perl's warnings. foreach(@$row) { $_ = '' if ! defined $_; } sprintf($format, @$row); }, 1, # Skip header row undef, # go to end of table $user_data, # This will be passed to the user method defined above ); } } $result; } =begin sdf H3: headers_to_tree Designed to convert a table into a n HTML list (C<
        >, <
      • >). In essence this trades the table width for length making it fit better on shallow screesn such as the Palm Pilot. Originally designed for use in SiteScooper files for output to iSilo. Builds an array of headers for each column of the table. The actual table data is NOT modified. This method is handy when a table is using column groups: > | Tempurature | Precipitation > Place | Today | Tonight | Saturday | Today | Tonight | Saturday Note that if the grouping must be done using C attribute otherwise there is no reliable way to know which group applies to which column. If C was not used then the grouping will appear incorrect. The following rules were determined to provide the most flexible and accurate output. Sample output contains only the HTML tags (header text is appended). The first rule that matches is used and all others are skipped. * Undefined columns are skipped * Blank columns are skipped * If there is text in the same column of the previous row then it is the first column in the conitionation of a multi-column heading: C<
        • > * If there is no previous column then treat as a blank (see next rule) * If the previous column of the current row is undefined then it is another multi-column header: C<
        • > * If the previous column of the current row is blank ("") then it is a new level: C<
          • > * If the previous column of the current row contains text (only option left) then it is a normal column: C<
          • > Note that these rules were determined trhough experimentation and appear to work on most tables. Tables that do not use header tags C<> properly or strangly formatted tables will probably cause weird header output. But in general even that is better than simply scrapping all table formatting! The first column is indentented like all others for consitancy and to avoid strange gaps in spacing that iSilo uses between before the first C<
              > tag. Returns a header format string designed for sprintf style formatting. Typically each row can be outputed by: C[$column_count], @row_data)>. Note: Alignment is ignored since all data is displayed in a veritcal list. All other format tags are utilized, such as C<>, C<>, C<>, etc... Note: It is recomended that L<"drop_blank_columns"> be called before this method. It will allow for more compressed table displays especially in cases where a column of images was nulled out. It is not auticamatically called since this method is not allowed to modify the table data. Note: Normally headers should be in bold, but that makes the output too hard to read since headers are repeated over and over. Instead use a bold ']' after the header text and before the column text. This seems to work well on the desktop and the Palm screen. =cut sub headers_to_tree { my $self = shift; my( $offset, ) = @_; $offset ||= 0; # Handle from end-of-table offsets if($offset < 0) { $offset = $self->row_count() - $offset; $offset = 0 if $offset < 0; } # Used to seperate the column headings from the cell contents my $seperator = "] "; # Starting at the row specified scan for headers building tree. my( @header, # Header format $column_text, # Contents of current column $column_loop, # Current column being processed $max_depth, # Maximum header depth (offset) $depth, # Current depth in header $open_tags, # How many open tags exist for the current column $tmp, # whatever... $column_is_group, # False until any row in the column is determined to be part of a group. Used to help adjust the table for better readability ); my($type, $data, $max); $max = $self->row_count(); $max_depth = 0; $open_tags = 0; for($column_loop = 0; $column_loop < $self->column_count(); $column_loop++) { $column_is_group = 0; # Loops until a non-header row is found for($depth=0;;$depth++) { ($type, $data) = $self->get_row($offset + $depth); if($type ne 'HEADER') { last; } $max_depth ||= $depth; # Remember maximum depth of header, only bother to set once $column_text = $data->[$column_loop]; # Easier to work with PROCESS: { # If current column is undefined then skip it (! defined $column_text) && do { $column_is_group = 1; # Column is part of a group last; }; # If current column is blank then append to header (used for side effect of defining header if it is currently undefined) ($column_text eq '') && do { # If on first row of header close out all open tags # This normally happens between two group columns or on the first column (where $open_tags is 0 anyway) if($depth == 0) { $header[$column_loop] = "
            " x $open_tags; $open_tags = 0; } else { $header[$column_loop] .= $column_text; } last; }; # *** At this point it is known that $column_text contains some kind of data # If there is data in the column header then a there must be data in one of the previous rows # for this column and therfor this header is probably another level of previous rows header if(defined $header[$column_loop]) { $header[$column_loop] .= "
            • "; $open_tags++; $header[$column_loop] .= $column_text; } else { # No data in previous row for this header, so see how far to back out of the # list tags. # See how many closeing tags must be generated, one for each level deep into the # header block. Note that 0 results in no text being appending. $tmp = $max_depth - $depth; if($tmp > $open_tags) { $tmp = $open_tags; } $header[$column_loop] .= ("
            " x $tmp); $open_tags -= $tmp; # If there are no open tags then add one, usually only occurs when at top of # first column if(! $open_tags) { $header[$column_loop] .= "
              "; $open_tags++; } # Now add another list element for this level of header $header[$column_loop] .= "
            • $column_text"; } } # End of PROCESS # See if ANY rows in this column belong to a group that was started in this column. if($column_loop != $self->column_count()-1 && ! defined $data->[$column_loop+1]) { $column_is_group = 1; } } # Loop for rows # If on the first column and it is NOT part of a group then adjust paramaters tp # bring this column seperate from the others. Often the first column is a key or # label column. if($column_loop == 0 && ! $column_is_group) { # This causes the next column to NOT close this open tag $open_tags = 0; } } # Loop for columns # Append the '%s' to each header to display the actual data column text # Normally headers should be in bold, but that makes the output too hard to read # since headers are repeated over and over. Instead use a ']' and make it bold. # This seems to work well. foreach(@header) { $_ .= "$seperator%s\n"; } # Combine header array into a single string my $header = join('', @header); # Append appropiate number of
            tags to each header by counting the number of open tags versus closing # tags. # This longer algorthm was chosen over simply looking at the open tag count because the open tag # count is NOT reliable for knowing the total number of open tags (funny that). The reason is that # the open tag count can be adjusted to highlight column 1 by placing it at its own list depth. # This algorithm also recovers from any errors above (there shouldn't be any, but...) my($open, $close) = (0, 0); $header =~ s/(
              )/$open++; $1/gie; $header =~ s/(<\/UL>)/$close++; $1/gie; if($close >= $open) { die "Internal error: The number of close ULs ($close) created exceeds or equals the number of open ULs ($open)"; } $header . ("
            " x ($open - $close)); } =begin sdf H3: row_type Returns type of row it is. A row offset (first row is 0) may be provided, defaults to the first row (0). Rows may be specified from end of table by using a negative offset, -1 = last row in table. =cut sub row_type { my $self = shift; my( $offset, ) = @_; $offset ||= 0; # Handle from end-of-table offsets if($offset < 0) { $offset = $self->row_count() - $offset; $offset = 0 if $offset < 0; } $self->{TABLE}{data}[$offset][0]; } =begin sdf H3: row_count Returns the number of rows in the table. =cut sub row_count { my $self = shift; scalar(@{$self->{TABLE}{data}}); } =begin sdf H3: column_count Returns the number of user columns in the table. =cut sub column_count { my $self = shift; scalar($self->{TABLE}{column_count}); } =begin sdf H3: get_row Returns the row type and a reference to the data. The data is adjusted to remove all internal data elements. If a scalar was requested then only the data is returned. Returns undef for data if the table is empty or a request is made beyond the table range. If $normalize is TRUE then all undefined cells become blanks. Useful for quick processing and Perl's warnings are on. Note: The L<"process_table"> is a much more effecient way to process multiple rows. This method was created to allow easy retrieval of one or two rows such as headers. Note: Use negative values to retrive data from the end of the table, where -1 = last row in table. =cut sub get_row { my $self = shift; my( $offset, $normalize, # If true causes all undefined cells to become blanks ) = @_; $offset ||= 0; # Handle from end-of-table offsets if($offset < 0) { $offset = $self->row_count() - $offset; } if($offset < 0 || $offset >= $self->row_count()) { if(wantarray) { (undef, undef); } else { undef; } } else { my @data = @{$self->{TABLE}{data}[$offset]}; my $type = shift @data; if($normalize) { foreach(@data) { $_ = '' if ! defined $_; } } if(wantarray) { ($type, \@data); } else { \@data; } } } =begin sdf H3: process_table Allows all the table data to be processed. Loops through each row of the table calling a user defined method with the following: !block table Paramater Description $self Reference to underlying table object $row_type Type of row, C
            or C \$data Reference to data elements, the first data element is in [0]. Note that this data is isolated from the rest of the table and can be directly modified without effecting the underlying table. Also note that internal elements (such as data type) are removed. \$err Optional error code. The user method can set this to 0 for ALL ok (default) or to a negative value to inidcate an error. Negative error codes abort table processing. The user error is returned by this method depending on how it was called, see return. $user_data Any type of data the caller wishes to be passed to the user method. Typically this is a refernce to an array or hash that allows variables that are needed from row to row. Undefined is passed if no $user_data was received. !end_block The user method can be defined as follows, no matter how passed the paramaters it receives are identical. * Recomended: As a string, in this case the call is done via the object allowing full object oriented interface. * As a standard refernece to a method. The user method should return: * Text to use for this row. All text is combined into a single string on return by this method without any addition characters added. * Optionally adjust the \$err passed. If set to a negative value that value is returned to the original caller of this method. Postive return codes are reserved for future use. Returns\: * If an array was requested then returns: ($output_text, $rows_processed, $err) where $err is < 0 as returned by user defined method. Note that on error $output_text is probably invalid or at least incomplete. * If a scalar was requested then returns: $output_text, or undef on error. =cut sub process_table { my $self = shift; my( $user_method, # Reference to user's method or a string for use with OO style $start_row, # Row to start on, default is 0 (first row) $row_count, # Number of rows to process, default is entire table $user_data, # Special user data ) = @_; my $out; # Output o return my @data; # Used to isolate data from underlying table and mask internal elements my $type; # Holds row type my $err; # Error from user method my $user_rtn; # Data returned by user method my $offset; # Current location into table array $start_row ||= 0; $start_row = 0 if $start_row < 0; $row_count = undef if(defined $row_count && $row_count < 0); $row_count = $self->row_count() if ! defined $row_count; for($offset = $start_row; $offset < $row_count; $offset++) { @data = @{$self->{TABLE}{data}[$offset]}; $type = shift @data; $err = 0; if(! ref $user_method) { $user_rtn = $self->$user_method($type, \@data, \$err, $user_data); } else { $user_rtn = &$user_method($self, $type, \@data, \$err, $user_data); } last if $err; $out .= $user_rtn; } if($err > 0) { croak("Unexpected return code of '$err' from user method '$user_method'. Postive return codes are reserved for use by this module "); } # Determin what to return if(wantarray) { ($offset, $out, $err); } else { if($err) { undef; } else { $out; } } } =begin sdf H3: force_headers Forces the first x rows to be header rows, all other rows are forced to data. Use this then a table fails to utilize C to indicate a header or uses C when C should have been used. Returns the parser object. =cut sub force_headers { my $self = shift; my( $header_count, ) = @_; my $data; foreach $data (@{$self->{TABLE}{data}}) { if($header_count--) { $data->[0] = 'HEADER'; } else { $data->[0] = 'DATA'; } } $self; } =begin sdf H3: drop_column Causes the column number specified to be dropped (default 0, first column). The optional $count (default 1) can be provided to drop more than one column. Pass a count of -1 to drop columns to end of table. Returns the parser object passed. =cut sub drop_column { my $self = shift; my( $start, $count, ) = @_; $count = 1 if !defined $count; if($start < 0) { croak "Starting paramater ($start) cannot be less than 0"; } if($count < -1) { croak "Count paramater ($count) cannot be less than -1"; } # handle end of table paramater if($count == -1) { $count = $self->column_count() - $start; } $start++; # Skip past the internal data column if($count > 0) { if($count + $start > $self->column_count()+1 ) { croak "Start paramater ($start) plus count paramater ($count) cannot exceed table width (" . $self->{TABLE}{column_count} . " columns)"; } foreach(@{$self->{TABLE}{data}}) { splice(@$_, $start, $count); } # Adjust table width tracker $self->{TABLE}{column_count} -= $count; } $self; } =begin sdf H3: drop_blank_columns Drops all columns that have every row blank (either undefined or an empty string or all white space). Normally done to clean-up a table where columns were post processed, such as removing images. =cut sub drop_blank_columns { my $self = shift; my $offset = 0; # current row offset my @data_found = (); # Tracks if data was found for each column my $data; # Contains referecne to table data, becomes undefined at end of table my $column; # current column offset # Loop through all rows looking for blank columns # Stop scan if all columns are determined to be non-blank while($data = $self->get_row($offset++)) { $column = 0; foreach(@$data) { if(defined $_ && length($_)) { $data_found[$column] = 1; } $column++; } # Exit unless at least one column is still blank if(@data_found) { last unless(grep $_, @data_found); } } # Work backwords to avoid having to recalculate offsets - skip column 0 (internal column type indicator) for($offset = scalar(@data_found)-1; $offset >= 0; $offset--) { next if $data_found[$offset]; $self->drop_column($offset); } $self; } =begin sdf H3: drop_row Drops the row number specified from the data. Normally this is used to remove a header that is not needed or wanted. If no value is passed a value of zero is assumed and the first row is dropped (the most common application). If $count is specified then that number of rows is droped, default is one. If 0 is passed no action will occur. Returns the parser object. Note: Currently values less than 0 for $count are treated as 0, but the use of a negative count is reserved for future expansion. Note: Remember that rows start at 0 B 1! =cut sub drop_row { my $self = shift; my( $start, $count, ) = @_; $count = 1 if !defined $count; if($start < 0) { croak "Starting paramater ($start) cannot be less than 0"; } if($count < -1) { croak "Count paramater ($count) cannot be less than -1"; } # handle end of table paramater if($count == -1) { $count = $self->row_count() - $start; } if($count > 0) { $start ||= 0; splice(@{$self->{TABLE}{data}}, $start, $count); } $self; } =begin sdf H3: join_headers Merges multiple consecitive header rows into a single row using the connector string passed. Normally used when a table ues group headers: > | Tempurature | Precipitation > Place | Today | Tonight | Saturday | Today | Tonight | Saturday becomes (if C<: > is used as the connector string) > Place | Tempurature: Today | Tempurature: Tonight | etc... Note that the C column from the second header line was used as is. This allows for cases where a table begins grouping columns after column 1, a rather common occurance. This also effects single header rows that used the C attribute. The extra undefined cells created during parsing are filled with the primary cell name. Headings defined Returns the parser object passed. Note: Scans all the rows and merges ANY multiple header rows found. Note: Use this method with caution, the standard min_width_text() method does a pretty good job of organizing multiple heading outputs. Note: The connector is not encoded into HTML allow it to contan HTML tags (such as , , etc) if desired. =cut sub join_headers { my $self = shift; my( $connector, ) = @_; my @new_data = (); # new table data is placed here, still related to $self for all data rows. Actual memory requirements for this variable are minimal. my $last; # Reference to last row addedif it was a header row my $row; # Current row being processed my $tmp; # whatever... $connector = ':' if ! defined $connector; foreach $row (@{$self->{TABLE}{data}}) { # fill any undef elements if this is a header row if($row->[0] eq 'HEADER') { # No need to skip first element since it is never undef and # second element can never be undef, although it could be blank. # See end() TD/TH processing if unsure why... foreach(@$row) { if(defined($_)) { $tmp = $_; } else { $_ = $tmp; } } } if(! @new_data) { # If no data then seed the new data push(@new_data, $row); } else { # See if this row is a header, if so merge with previous # if it also was also header if($row->[0] eq 'HEADER' && $last) { # Merge the rows # Increase size of $last if necessary while(scalar(@$last) < scalar(@$row)) { push(@$last, $last->[-1]); } # Increase size of $row if necessary while(scalar(@$last) > scalar(@$row)) { push(@$row, $row->[-1]); } # Now merge current row with last row, using connector, skip row type [0] element # If no text is in last row then do NOT perform conactenation, handles # cases were grouped columns start after column 1 (rather common). for($tmp = 1; $tmp < scalar(@$last); $tmp++) { if($last->[$tmp]) { $last->[$tmp] .= $connector . $row->[$tmp]; } else { $last->[$tmp] = $row->[$tmp]; } } } else { # Not a multiple header sequence so add row normally push @new_data, $row; } } # Remember if current row is a header if($row->[0] eq 'HEADER') { $last = $new_data[$#new_data]; } else { $last = undef; } } # Transfer modified array back to object $self->{TABLE}{data} = \@new_data; $self; } =begin sdf H3: join_columns Synopsis\: join_columns($connector, \@join_list1 {,\@join_list2, ...}, $smart_join, \@forced_keep) Joins two or more columns into a single column using the connector string supplied. Note that this joins ALL columns in the table including headers. Multiple column joins are allowed during the same call. For multiple joins each works from the original rows offsets. If $smart_join is turnd on then when two heading columns are joined a check is done to see if the column names are the same, if they are the text is left unchanged. This only applies to heading rows, data rows are always joined. For example the following: > Today | Tonight | Saturday | Today | Tonight | Saturday when merged on 0,3 then 1,3 then 2,3 using connector ':' > $parser->join_columns(':', [0, 3]); > $parser->join_columns(':', [1, 3]); > $parser->join_columns(':', [2, 3]); without $smart_join > Today:Today | Tonight:Tonight | Saturday:Saturday with $smart_join (for example: C<$parser->join_columns(':', [0, 3], 1)> ) > Today | Tonight | Saturday Or use the single pass option to do all with one call. Note that the offsets are based on the original row. This format is also a bit faster. > $parser->join_columns(':', [1, 3], [2, 4], [3, 5], 1); See also L<"join_same_columns"> This is normally used to compress a table that used group columns into an easier to read format. Returns the parser object passed. Note: Columns must be passed by reference! Note: Columns are removed as they are joined so that when multiple joins take place, as in the example above where 3 seperate column pairs were joined, you must adjust the offsets properly. You are guarenteed that the offsets do not change during a single call, so a list of 2,3,4 will join columns 2,3,4 into a single column 2. On the next call remember that columns 3,4 are now something different. Note: Column 0 is the first column of the row (even through internally [0] is used for type, this translation is handled transparently to the caller). Note: Column numbers may be in any order and will be merged in the order provided with all text appearing in the B column in the list. The remaining columns will be removed when the join is done. Note: Column values less than 0 cause a fatel exception, column numbers beyond the length of the table also cause a fatal exception. Note: Columns must be unique within the list. Note: The connector is not encoded into HTML allow it to contan HTML tags (such as , , etc) if desired. Note: Errors are indicated by column number and group number, where the first group in the list is 1. Note: On occasion it is necessary to keep a column that would otherwise be deleted. This can be done via the optional \@forced_keep paramater. For example if 2,3,4 are merged (into 2) normally 3 and 4 are deleted. But if \@forced_keep is [3,4] then 3 and 4 will remain. Forced keep only applies to this call and a column need only be mentioned once even it was merged several times. H4: To do This method got way out of hand and does far more than it was originally designed to. I have tested it as well as I can given time. A re-thining of the code may be in order and some test code. =cut sub join_columns { # # Extract paramaters, a bit weird but handy # my $self = shift; my $connector = shift; # Scan remaining paramaters until a non-array reference is found my @column_list; while(ref($_[0]) =~ /ARRAY/) { # Create a unique array to avoid mucking with passed data - never muck with user data without good reason push @column_list, [@{$_[0]}]; shift; } # Get remaining paramaters my( $smart_join, $forced_keep, # refernce to a list of columns that should never be deleted ) = @_; # now define some variables my $column; # Refernce to current column being processed my $row; # current row being processed my $target; # target column my $group; # tracks which column group is being worked on my %delete_list; # Contains columns that will be deleted (1) or forced saved (-1) my $tmp; # whatever my %tmp; # whatever my @tmp; # whatever my $offset; # # Make sure column counts are valid and adjust to skip past internal column (type) # $tmp = ''; %tmp = (); $group = 1; foreach $column (@column_list) { $offset = 0; %tmp = (); foreach(@$column) { $_ += 0; # Force to be a number if($_ < 0) { $tmp .= "Negative column #$_ in group $group is not allowed. "; } elsif($_ > $self->{TABLE}{column_count}) { $tmp .= "Column #$_ in group $group exceeds maximum table width of $self->{TABLE}{column_count}. "; } # Adjust to skip past internal type column $_++; # Do NOT allow duplicates $tmp{$_}++; # Add to delete list all target columns (2nd+ in a group) if($offset++) { $delete_list{$_} = 1; } } if(@$column < 2) { $tmp .= "At least two columns must be provided (group $group). "; } foreach(sort {$a <=> $b} %tmp) { if($tmp{$_} > 1) { $tmp .= sprintf("Column #%d is repeated %d times in group $group, must be unqiue within the group. ", $_-1, $group); } } $group++; } if($tmp) { croak("Errors with column IDs passed: $tmp"); } # # Clean up the delete list, remove forced keep columns # foreach(@$forced_keep) { delete $delete_list{$_}; } # # Clean up removed groups (forced keep) # @column_list = grep(defined $_, @column_list); # # Now join the columns # foreach $row (@{$self->{TABLE}{data}}) { foreach $column (@column_list) { @tmp = @$column; # Isolate so it doesn't get changed $target = shift @tmp; # Get starting column foreach(@tmp) { if(! $smart_join || $row->[0] ne 'HEADER' || $row->[$target] ne $row->[$_]) { $row->[$target] .= $connector . $row->[$_]; } } } # Now delete the columns that were joined, delete from end to start # to avoid having to recalculate offsets foreach(sort {$b <=> $a} keys %delete_list) { splice(@$row, $_, 1); } } $self; } =begin sdf H3: join_same_columns Uses the column names from the header row specified (defaults to 0) and checks to see if any columns have the same names. If so the columns are marked for joining. All columns in the table are joined using this this list. This was originall written to handle common cases where a table is grouped into multiple sections such as: !block example 0 : | Tempurature | Precipiation 1 : Place | Today | Tonight | Saturday | Today | Tonight | Saturday 2 : Seattle| 51 | 42 | 49 | 60 | 100 | 80 !end_block In this example the first row (0) would probably be deleted, see L<"delete_row">, and then columns merged using this method. If the connector string was C<"/"> then the result would be: !block example 0 : Place | Today | Tonight | Saturday 1 : Seattle| 51/60 | 42/100 | 49/80 !end_block This can compress the table data considerable. See also L<"join_columns"> Note: This method uses L<"join_columns"> with the $samrt_join enabled. Note: The column names must match exactly including whitespace and HTML tags =cut sub join_same_columns { my $self = shift; my( $connector, $header_row, ) = @_; $header_row ||= 0; if($header_row < 0) { croak "Header row paramater ($header_row) cannot be less than 0"; } if($self->{TABLE}{data}{$header_row}[0] ne 'HEADER') { croak "Header row paramater ($header_row) references a row that is not a header."; } my %lookup; # Used to track same name columns my $offset; for($offset = 1; $offset <= $self->{TABLE}{column_count}; $offset++) { # Use column name to track groups - adjust offset to be from user's # perspective (not counting internal type column) push @{$lookup{$self->{TABLE}{data}[$header_row][$offset]}}, $offset-1; } # Build group list for all joins where more than one column is present my @join_list = (); foreach(values %lookup) { next if @$_ < 2; push @join_list, $_; } # Join the columns if(@join_list) { $self->join_columns($connector, @join_list, 1); } $self; } =begin sdf H3: start Intercepts the following table tags: * table * tr * th * td * /pre - B skiped, but prevents newline/space conversion, see ouput() These tags are removed and output offsets are initialized (on C). See L for incrementing offsets and setting of row data type. Output is turned on when the start of a CELL is found. =cut sub start { my $self = shift; # Parser object my( $tag, # tag name in lower case $attr, # hash of all attributes, attribute names in lower case $attrseq, # array containing attributes in order found $origtext, # original full HTML text ) = @_; return if $self->{TABLE}{skip_eof}; PROCESS:{ # Strip the table tags from the output $tag eq 'table' && do { # Save any pending table definition on stack if nesting if($self->{TABLE}{table_active}) { push (@{$self->{TABLE_STACK}}, $self->{TABLE}); } # Reset Table stuff delete $self->{TABLE}; # Remove all current data $self->{TABLE}{row_id} = 0; $self->{TABLE}{jagged} = -1; # Tracks if the table was determined to be jagged, an increment of 1 is unavoidable if speed is to be maintained (see end()) so set to -1 to compensate. $self->{TABLE}{column_count} = 0; $self->{TABLE}{row_id} = 0; $self->{TABLE}{col_id} = 0; $self->{TABLE}{type} = undef; $self->{TABLE}{data} = []; $self->{TABLE}{table_active} = 1; $self->{TABLE}{no_out} = 1; # No more ouput until a cell is encountered $self->{TABLE}{force_blank} = 0; last; }; $tag eq 'tr' && do { $self->{TABLE}{col_id} = 1; # [0] is reserved for row data type $self->{TABLE}{type} = 'DATA'; # Assume data until otherwise determined # Remeber callspan attribute for processing by end() if(exists $attr->{colspan} && $attr->{colspan} > 1) { $self->{TABLE}{force_blank} = $attr->{colspan} - 1; } last; }; $tag eq 'th' && do { $self->{TABLE}{type} = 'HEADER'; $self->{TABLE}{no_out} = 0; # Output is now ok, inside a cell # Remeber callspan attribute for processing by end() if(exists $attr->{colspan} && $attr->{colspan} > 1) { $self->{TABLE}{force_blank} = $attr->{colspan} - 1; } last; }; $tag eq 'td' && do { $self->{TABLE}{no_out} = 0; # Output is now ok, inside a cell last; }; $tag eq 'pre' && do { # Turn on 
             processing, see output() for details
            $self->{TABLE}{pre_on}++;
            last;
        };

        # all other tags are processed normally
        $self->SUPER::start(@_);
    };
}


=begin sdf

H3: end

Intercepts the following table tags:
    * /table
    * /tr
    * /th
    * /td
    * /pre - B skiped, but prevents newline/space conversion, see ouput()

These tags are removed and output offsets incremented. The C tag also triggers
the setting of the row type for the current row and checking for maximum column
count and jaggedness.

Output is turned off when the end of a CELL is found.

Note: Table data has all leading and trailing spaces removed!

=cut

sub end {
    my $self = shift;       # Parser object
    my(
        $tag,               # tag name in lower case
        $origtext,          # original full HTML text
    ) = @_;

    return if $self->{TABLE}{skip_eof};

    PROCESS:{
        # Strip the table tags from the output
        # As long as the parent start() is not called the tags will vanish from
        # the output.
        $tag eq 'table' && do {
            # Make sure table is cleaned up, otherwise weird and wild things may occur
            $self->_clean_up_table();

            # Handle nested tables properly, this way process can adjust its output
            # if nested tables are being processed.
            if(scalar(@{$self->{TABLE_STACK}}) > 0) {
                $self->{TABLE}{table_active} = 1;
            } else {
                $self->{TABLE}{table_active} = 0;
            }

            $self->{TABLE}{no_out} = 0;        # Restore normal output
            my $tmp = $self->set_table_post_process();
            $tmp = $self->$tmp();               # Remember result

            # If there is a table stack pending then grab result of last table
            # and add it to the appropiate cell for table on stack restoring
            # table on stack as current
            if(scalar(@{$self->{TABLE_STACK}}) > 0) {
                $self->{TABLE} = pop @{$self->{TABLE_STACK}};   # Restore parent table
            }

            $self->output($tmp);                            # save result into appropiate area
            last;
        };

        $tag eq 'tr'    && do {
            # Set type based on encountering any header like columns
            $self->{TABLE}{data}[$self->{TABLE}{row_id}][0] = $self->{TABLE}{type};

            # Check to see if the table is jagged and adjust width count
            # Anytime the number of columns in a row do NOT equal the previous count
            # indicates a jagged table.
            my $tmp;
            $tmp = scalar(@{$self->{TABLE}{data}[$self->{TABLE}{row_id}]}) - 1;       # Ignore the type column
            $self->{TABLE}{jagged}++ if $tmp != $self->{TABLE}{column_count};
            if($tmp > $self->{TABLE}{column_count}) {
                # Only remember row
                $self->{TABLE}{column_count} = $tmp;
            }

            $self->{TABLE}{row_id}++;
            last;
        };


        # TH and TD are the same for end tag processing
        ($tag eq 'th' || 
        $tag eq 'td')    && do {

            # Force a blank string to ouput just in case the CELL has no data defined
            # in it. Differenates from a spanned cell which is undefined.
            # Note that I don't bother to add a check statement first since
            # that just complicates maintenance and has minimal speed impact for
            # this application.
            my $tmp = \$self->{TABLE}{data}[$self->{TABLE}{row_id}][$self->{TABLE}{col_id}];
            $$tmp .= "";
            $$tmp =~ s/^\s+//g;
            $$tmp =~ s/\s+$//g;

# Didn't end up being a good idea, too many bolds mixed up display
# instead try bolding the seperator between the header name and data, see table_out
# keeping code just in case it is later needed and to prevent re-inventing the wheel
#            # TH by default places text in bold, so follow suit, unless cell is blank
#            if($tag eq 'th') {
#                if($$tmp) {
#                    $$tmp = "$$tmp";
#                }
#            }

            # Update offsets to prepare for next cell
            $self->{TABLE}{col_id}++;

            # See if colspan is in effect, if so then create false entries
            if($self->{TABLE}{force_blank} > 0) {
                while($self->{TABLE}{force_blank}--) {
                    $self->output();
                    $self->{TABLE}{col_id}++;
                }
            }

            $self->{TABLE}{no_out} = 1;        # all output outside a cell is ignored

            last;
        };


        $tag eq 'pre'   && do {
            # Turn off 
             processing, see output() for details
            $self->{TABLE}{pre_on}--;
            last;
        };

        # all other tags are processed normally
        $self->SUPER::end(@_);
    };
}




=begin sdf

H3: output

Saves data in the appropiate row/column of the table data. Does not adjust
output offsets. If there is no table currently being processed the data is
appended to a holding buffer. All data can be retrieved via L<"get_result">.

If the C
             tag is not active then convert newlines to spaces and collapse multiple
spaces into one. Leading and trailing spaces remain, these are handled in the L<"end">
method, see the TR/TH processing section.

Note: Pass an empty string to force the cell to become defined but empty. Pass undef() to
create a new cell that is not defined.

=cut


sub output {
    my $self = shift;
    my(
        $out
    ) = @_;

    # See if NO output should be done
    if(! $self->{TABLE}{no_out}) {
        if(! $self->{TABLE}{table_active}) {
            $self->{NEW_TEXT} .= $out;
            return;
        }

        # Handle when undef() is in a cell of a table. It must remain undefined only
        # if cell is already undefined.
        if(! defined $out) {
            if(! defined $self->{TABLE}{data}[$self->{TABLE}{row_id}][$self->{TABLE}{col_id}]) {
                $self->{TABLE}{data}[$self->{TABLE}{row_id}][$self->{TABLE}{col_id}] = undef;
            }

            # otherwise don't bother doing anything.
        } else {
            if(! $self->{TABLE}{pre_on}) {
                # If PRE is NOT on (allows nested) then convert all newlines
                # to spaces and collapse multiple spaces into one.
                $out =~ s/[\n\s]+/ /g;
            }

            $self->{TABLE}{data}[$self->{TABLE}{row_id}][$self->{TABLE}{col_id}] .= $out;
        }
    }
}


=begin sdf

H3: get_result

Returns the text string resulting from parsing.

=cut

sub get_result {
    my $self = shift;

    $self->{NEW_TEXT};
}


=begin sdf

H3: clear_output

Clears all pending output for the object. Does B clear the table data.

Returns the object passed.

=cut

sub clear_output {
    my $self = shift;

    $self->{NEW_TEXT} = '';
}


=begin sdf

H3: text_only

Returns the text with all HTML tags removed. Designed to be very quick and
therfor uses a brute force method of removing tags:
    * Locate and remove all <..> patterns!

Originally written to help calculate the amount of text characters a cell would
require to help better convert table data to non-table data.

=cut

sub text_only {
    my $self = shift;
    my(
        $in,
    ) = shift;

    $in =~ s/<[^>]+>//gs;            # treat input string as a single line of text
    $in;
}



=begin sdf

H2: Private Methods

These methods should not be called by methods outside this module.

=cut


=begin sdf

H3: _clean_up_table

Does the following clean up for the parsed table data:
    * Foces all rows to have the same number of columns. All added columns
    contain a blank string.

    * Sets 'column_count' element to column width of table

Normally called automatically by L<"parse"> and L<"parse_file">, this method is
critical to prevent weird errors during column/row manipulation and display.

Returns parser object passed.

See also L<"parse">, L<"parse_file">

=cut

sub _clean_up_table {
    my $self = shift;

    # Scan adjusting all columns to be maximum width if table was found to be jagged
    if($self->{TABLE}{jagged}) {
        my $max = $self->column_count() + 1;

        foreach(@{$self->{TABLE}{data}}) {
            while(scalar(@$_) < $max) {
                push @$_, '';
            }
        }

        # Table is no longer jagged
        $self->{TABLE}{jagged} = 0;
    }

    $self;
}


=begin sdf

H3: _default_post_process

Default handler for processing tables for this module. The default is to run the table
through the L<"table_out"> method and take the resulting string and add it to the table
output string. All blank columns (see L<"drop_blank_columns">) are first droped to
help simplify the table.

Also place text in a TABLE block. This causes iSilo (and others) to properly handle preceeding

             tags and other things as well as highlighting the table in some manner.

Users can override this default via L<"set_table_post_process"> or by overloading
this method.

The results are returned as a string which are passed to the L<"output"> method,
which the user can overload if desired to accept/process whatever data format is
desired.

Note: The $self->{TABLE}{table_active} variable is guarenteed to be:
    * 0 - Text is not nested within another table
    * 1 - Text is nested within another table

Note: If there is no non-whitespace data for the table then the entire table
is ignored, a blank string is returned.

=cut

sub _default_post_process {
    my $self = shift;

    $self->drop_blank_columns();

    my $tmp = $self->table_out();
    if($tmp =~ /^\s*$/) {
        # If no data then empty table (data was probably stripped by another process)
        # so return nothing.
        '';
    } else {
# DEBUG
open(T, ">>t") && do {
    print T $tmp, "\n---------------\n\n";
};
        $Table_Start . $tmp . $Table_End;
    }
}



1;