FlatFileParser Class Reference

This class parses fields and rows from a field delimited text buffer. More...

#include <FlatFileParser.h>

List of all members.

Public Member Functions
	FlatFileParser (const char fieldDelim, const char rowDelim, const char quote, const char escape, bool doTrim=false)
	Constructs a FlatFileParser.
void	scanRow (const char *buffer, int size, const FlatFileRowDescriptor &columns, FlatFileRowParseResult &result)
	Scans through buffer until the end of a row is reached, and locates columns within the row.
void	scanColumn (const char *buffer, uint size, uint maxLength, FlatFileColumnParseResult &result)
	Scans through buffer to find the length of a column value.
void	scanFixedColumn (const char *buffer, uint size, uint maxLength, FlatFileColumnParseResult &result)
	Scans a fixed format column.
void	stripQuoting (FlatFileRowParseResult &rowResult, bool trim)
	Remove quoting and escape characters from a row result, saving the results into the row result.
uint	stripQuoting (char *buffer, uint size, bool untrimmed)
	Removes quoting and escape characters from a column value.
uint	trim (char *buffer, uint size)
	Trim spaces from beginning and end of text.
Private Member Functions
const char *	scanRowEnd (const char *buffer, int size, bool rowDelim, FlatFileRowParseResult &result)
	Scans through buffer to recover from any row errors.
const char *	scanRowDelim (const char *buffer, int size, bool search)
	Scan through buffer to find a row delimiter, or non row delimiter character.
bool	isRowDelim (char c)
	Determines whether or not character is a row delimiter.
Private Attributes
char	fieldDelim
char	rowDelim
char	quote
char	escape
bool	doTrim
bool	fixed
	Whether to perform fixed mode parsing.

---

Detailed Description

This class parses fields and rows from a field delimited text buffer.

The main entry point is scanRow() which returns pointers into a text buffer, representing columns. The method stripQuoting() may be useful for decoding a quoted value. Other methods are primarily made public for testing purposes.

Author:

John Pham

Version:

Id

//open/dev/fennel/flatfile/FlatFileParser.h#2

Definition at line 389 of file FlatFileParser.h.

---

Constructor & Destructor Documentation

FlatFileParser::FlatFileParser	(	const char	fieldDelim,
		const char	rowDelim,
		const char	quote,
		const char	escape,
		bool	doTrim = false
	)

Constructs a FlatFileParser.

See FlatFileExecStreamParams for more detail on the parameters.

Parameters:

[in]	fieldDelim	character delimiter
[in]	rowDelim	row delimiter
[in]	quote	quote character
[in]	escape	escape character
[in]	doTrim	whether to trim column values before processing them. A column value is trimmed before it is unquoted.

Definition at line 78 of file FlatFileParser.cpp.

References fixed.

{
    this->fieldDelim = fieldDelim;
    this->rowDelim = rowDelim;
    this->quote = quote;
    this->escape = escape;
    this->doTrim = doTrim;

    fixed = (fieldDelim == 0);
    if (fixed) {
        assert(quote == 0);
        assert(escape == 0);
    }
}

---

Member Function Documentation

const char * FlatFileParser::scanRowEnd	(	const char *	buffer,
		int	size,
		bool	rowDelim,
		FlatFileRowParseResult &	result
	)			[private]

Scans through buffer to recover from any row errors.

Scans to row delimiter if one was not found. Then scans past spurious row delimiters. On success, increments row delimiter count.

Parameters:

[in]	buffer	flat file buffer to be parsed
[in]	size	size of buffer
[in]	rowDelim	whether a row delimiter was read from previous row
[in,out]	result	result from scanning for row

Returns:

pointer to the next row, or end of buffer

Definition at line 187 of file FlatFileParser.cpp.

References FlatFileRowParseResult::INCOMPLETE_COLUMN, FlatFileRowParseResult::nRowDelimsRead, FlatFileRowParseResult::ROW_TOO_LARGE, scanRowDelim(), and FlatFileRowParseResult::status.

Referenced by scanRow().

{
    const char *read = buffer;
    const char *end = buffer + size;
    switch (result.status) {
    case FlatFileRowParseResult::INCOMPLETE_COLUMN:
    case FlatFileRowParseResult::ROW_TOO_LARGE:
        assert(read == end);
        return read;
    default:
        break;
    }

    // if a row delimiter was not encountered while scanning the row,
    // search for the next row delimiter character
    if (!rowDelim) {
        read = scanRowDelim(read, end - read, true);
        if (read == end) {
            return read;
        }
    }
    result.nRowDelimsRead++;

    // search for the first non- row delimiter character
    read = scanRowDelim(read, end - read, false);
    return read;
}

const char * FlatFileParser::scanRowDelim	(	const char *	buffer,
		int	size,
		bool	search
	)			[private]

Scan through buffer to find a row delimiter, or non row delimiter character.

Parameters:

[in]	buffer	flat file buffer to be parsed
[in]	size	size of buffer
[in]	search	if true, look for row delimiter, else a non row delim

Returns:

pointer to character found, or end of buffer

Definition at line 219 of file FlatFileParser.cpp.

References isRowDelim().

Referenced by scanRow(), and scanRowEnd().

{
    const char *read = buffer;
    const char *end = buffer + size;
    while (read < end) {
        if (isRowDelim(*read) == search) {
            break;
        } else {
            read++;
        }
    }
    return read;
}

bool FlatFileParser::isRowDelim

(

char

c

)

[private]

Determines whether or not character is a row delimiter.

If the row delimiter is any of the line characters (/r or /n), then it must be encoded as newline (/n) and it matches any other line character.

Definition at line 236 of file FlatFileParser.cpp.

References rowDelim.

Referenced by scanColumn(), scanFixedColumn(), and scanRowDelim().

{
    assert(rowDelim != '\r');
    return (rowDelim == '\n') ? (c == '\r' || c == '\n') : (c == rowDelim);
}

void FlatFileParser::scanRow	(	const char *	buffer,
		int	size,
		const FlatFileRowDescriptor &	columns,
		FlatFileRowParseResult &	result
	)

Scans through buffer until the end of a row is reached, and locates columns within the row.

The main options are a "bounded", "lenient", and "mapped".

If a scan is "bounded", the output must match an expected format as specified by the column descriptions. Rows with the wrong number of columns are treated as bad rows. However, if a "bounded" scan is also "lenient", the parser is forgiving. Missing values are filled in with null, and extra values are discarded. If columns are "mapped" then they default to null, and columns read are assigned to output columns according to the mapping.

If a scan is "unbounded", then the output may have any number of columns. The other options are not applicable to unbounded mode.

Parameters:

[in]	buffer	buffer with text to be parsed
[in]	size	size of buffer, in characters
[in]	columns	description of columns to be parsed
[out]	result	result of parsing row

Definition at line 94 of file FlatFileParser.cpp.

References FlatFileRowParseResult::addColumn(), FlatFileRowParseResult::clear(), FlatFileRowParseResult::current, FlatFileColumnParseResult::FIELD_DELIM, FlatFileRowDescriptor::getMap(), FlatFileRowDescriptor::getMaxColumns(), FlatFileRowDescriptor::getMaxLength(), FlatFileRowParseResult::INCOMPLETE_COLUMN, FlatFileRowDescriptor::isBounded(), FlatFileRowDescriptor::isLenient(), FlatFileRowDescriptor::isMapped(), FlatFileColumnParseResult::MAX_LENGTH, FlatFileRowParseResult::next, FlatFileColumnParseResult::next, FlatFileRowParseResult::NO_COLUMN_DELIM, FlatFileColumnParseResult::NO_DELIM, FlatFileRowParseResult::NO_STATUS, FlatFileRowParseResult::resize(), FlatFileColumnParseResult::ROW_DELIM, rowDelim, scanColumn(), scanRowDelim(), scanRowEnd(), FlatFileRowParseResult::setColumn(), FlatFileRowParseResult::setNull(), FlatFileColumnParseResult::size, FlatFileRowParseResult::status, FlatFileRowParseResult::TOO_FEW_COLUMNS, FlatFileRowParseResult::TOO_MANY_COLUMNS, and FlatFileColumnParseResult::type.

{
    assert(size >= 0);
    const char *row = buffer;
    uint offset = 0;
    FlatFileColumnParseResult columnResult;

    result.status = FlatFileRowParseResult::NO_STATUS;
    bool bounded = columns.isBounded();
    bool lenient = columns.isLenient();
    bool mapped = columns.isMapped();
    bool strict = (bounded && (!lenient));

    uint maxColumns = columns.getMaxColumns();
    uint resultColumns = columns.size();
    if (bounded) {
        result.resize(resultColumns);
        for (uint i = 0; i < resultColumns; i++) {
            result.setNull(i);
        }
    } else {
        result.clear();
    }

    // Scan any initial row delimiters, helps for the case when a row
    // delimiter is multiple characters like \r\n and the delimiter
    // characters are split between two buffers. (The previous row could
    // be complete due to \r, and parsing could begin at \n.
    const char *nonDelim = scanRowDelim(row, size, false);
    offset = nonDelim - row;

    bool done = false;
    bool rowDelim = false;
    for (uint i = 0; i < maxColumns; i++) {
        uint maxLength = columns.getMaxLength(i);
        scanColumn(
            row + offset,
            size - offset,
            maxLength,
            columnResult);
        switch (columnResult.type) {
        case FlatFileColumnParseResult::NO_DELIM:
            result.status = FlatFileRowParseResult::INCOMPLETE_COLUMN;
            done = true;
            break;
        case FlatFileColumnParseResult::ROW_DELIM:
            if (strict && (i+1 != columns.size())) {
                if (i == 0) {
                    result.status = FlatFileRowParseResult::NO_COLUMN_DELIM;
                } else {
                    result.status = FlatFileRowParseResult::TOO_FEW_COLUMNS;
                }
            }
            done = true;
            rowDelim = true;
            break;
        case FlatFileColumnParseResult::MAX_LENGTH:
        case FlatFileColumnParseResult::FIELD_DELIM:
            if (strict && (i+1 == columns.size())) {
                result.status = FlatFileRowParseResult::TOO_MANY_COLUMNS;
                done = true;
            }
            break;
        default:
            permAssert(false);
        }
        if (bounded) {
            int target = mapped ? columns.getMap(i) : i;
            if (target >= 0) {
                assert (target < resultColumns);
                result.setColumn(target, offset, columnResult.size);
            }
        } else {
            result.addColumn(offset, columnResult.size);
        }
        offset = columnResult.next - row;
        if (done) {
            break;
        }
    }
    result.current = const_cast<char *>(row);
    result.next = const_cast<char *>(
        scanRowEnd(
            columnResult.next,
            buffer + size - columnResult.next,
            rowDelim,
            result));
}

void FlatFileParser::scanColumn	(	const char *	buffer,
		uint	size,
		uint	maxLength,
		FlatFileColumnParseResult &	result
	)

Scans through buffer to find the length of a column value.

Keeps going until it reads a delimiter or it completes a fixed column. A column is considered to be quoted if and only if the first character is a quote character.

Parameters:

[in]	buffer	buffer containing text to scan
[in]	size	size of buffer contents, in characters
[in]	maxLength	max length of column, excluding escapes and quotes
[out]	result	result of scanning buffer

Definition at line 242 of file FlatFileParser.cpp.

References doTrim, escape, FlatFileColumnParseResult::FIELD_DELIM, fieldDelim, fixed, isRowDelim(), FlatFileColumnParseResult::NO_DELIM, quote, FlatFileColumnParseResult::ROW_DELIM, scanFixedColumn(), FlatFileColumnParseResult::setResult(), and SPACE_CHAR.

Referenced by FlatFileExecStreamTest::checkColumnScan(), and scanRow().

{
    if (fixed) {
        return scanFixedColumn(buffer, size, maxLength, result);
    }

    assert(buffer != NULL);
    const char *read = buffer;
    const char *end = buffer + size;

    // read past leading spaces before checking for quotes
    if (doTrim) {
        while (read < end && SPACE_CHAR == *read) {
            read++;
        }
    }

    bool quoted = (read < end && *read == quote);
    bool quoteEscape = (quoted && quote == escape);

    FlatFileColumnParseResult::DelimiterType type =
        FlatFileColumnParseResult::NO_DELIM;
    if (quoted) {
        read++;
    }
    while (read < end) {
        if (*read == quote) {
            read++;
            if (quoteEscape) {
                // read next character to determine whether purpose of
                // this character is an escape character or an end quote
                if (read == end) {
                    break;
                }
                if (*read == quote) {
                    // two consecutive quote/escape characters is an
                    // escaped quote
                    read++;
                    continue;
                }
            }
            if (quoted) {
                // otherwise a quote may be a close quote
                quoteEscape = quoted = false;
            }
        } else if (*read == escape) {
            read++;
            // an escape escapes the next character
            if (read == end) {
                break;
            }
            read++;
        } else if (quoted) {
            read++;
        } else if (*read == fieldDelim) {
            type = FlatFileColumnParseResult::FIELD_DELIM;
            break;
        } else if (isRowDelim(*read)) {
            type = FlatFileColumnParseResult::ROW_DELIM;
            break;
        } else {
            read++;
        }
    }

    uint resultSize = read - buffer;
    result.setResult(type, const_cast<char *>(buffer), resultSize);
}

void FlatFileParser::scanFixedColumn	(	const char *	buffer,
		uint	size,
		uint	maxLength,
		FlatFileColumnParseResult &	result
	)

Scans a fixed format column.

In this mode, the quote character and column delimiter are ignored. It is possible to stop parsing because (1) a row delimiter is read (2) the max length is reached or (3) the end of buffer is reached

Definition at line 315 of file FlatFileParser.cpp.

References isRowDelim(), FlatFileColumnParseResult::MAX_LENGTH, FlatFileColumnParseResult::NO_DELIM, FlatFileColumnParseResult::ROW_DELIM, and FlatFileColumnParseResult::setResult().

Referenced by scanColumn().

{
    assert(buffer != NULL);
    const char *read = buffer;
    const char *end = buffer + size;
    uint remaining = maxLength;

    FlatFileColumnParseResult::DelimiterType type =
        FlatFileColumnParseResult::NO_DELIM;
    while (read < end && remaining > 0) {
        if (isRowDelim(*read)) {
            type = FlatFileColumnParseResult::ROW_DELIM;
            break;
        }
        read++;
        remaining--;
    }

    // Resolve delimiter type if another character can be read. This allows
    // us to catch the case where a row delimiter follows a max length field.
    if (type == FlatFileColumnParseResult::NO_DELIM && read < end) {
        if (isRowDelim(*read)) {
            type = FlatFileColumnParseResult::ROW_DELIM;
        } else if (remaining == 0) {
            type = FlatFileColumnParseResult::MAX_LENGTH;
        }
    }

    uint resultSize = read - buffer;
    result.setResult(type, const_cast<char *>(buffer), resultSize);
}

void FlatFileParser::stripQuoting	(	FlatFileRowParseResult &	rowResult,
		bool	trim
	)

Remove quoting and escape characters from a row result, saving the results into the row result.

Parameters:

[in,out]	rowResult	the row result to be stripped
[in]	trim	whether to trim columns before processing them

Definition at line 351 of file FlatFileParser.cpp.

References FlatFileRowParseResult::getColumn(), FlatFileRowParseResult::getRawColumnSize(), FlatFileRowParseResult::getReadCount(), and FlatFileRowParseResult::strippedSizes.

Referenced by FlatFileExecStreamTest::checkStrip().

{
    int nFields = rowResult.getReadCount();

    if (rowResult.strippedSizes.size() < nFields) {
        rowResult.strippedSizes.resize(nFields);
    }

    for (uint i = 0; i < nFields; i++) {
        char *value = rowResult.getColumn(i);
        uint newSize = 0;
        if (value != NULL) {
            uint oldSize = rowResult.getRawColumnSize(i);
            newSize = stripQuoting(value, oldSize, trim);
        }
        rowResult.strippedSizes[i] = newSize;
    }
}

uint FlatFileParser::stripQuoting	(	char *	buffer,
		uint	size,
		bool	untrimmed
	)

Removes quoting and escape characters from a column value.

If untrimmed is set, then the value will be trimmed first. Otherwise, quoted values are expected to begin and end with a quote.

Examples (assuming both quote and escape are double quote):
"a quote" becomes a quote
"""a quote""" becomes "a quote"
"a quote becomes a quote
a quote" becomes a quote"
""aquote" becomes (empty string)

Parameters:

[in,out]	buffer	buffer containing column value text
[in]	size	size of column value text, in characters
[in]	untrimmed	if value is untrimmed, it will be trimmed first

Returns:

size of resulting column text

Definition at line 372 of file FlatFileParser.cpp.

References escape, quote, and trim().

{
    assert(buffer != NULL);
    if (sizeIn == 0) {
        return 0;
    }
    int size = untrimmed ? trim(buffer, sizeIn) : sizeIn;
    bool quoted = false;
    char *read = buffer;
    char *end = buffer + size;
    char *write = buffer;

    if (*buffer == quote) {
        quoted = true;
        read++;
    }
    bool quoteEscape = (quoted && quote == escape);
    while (read < end) {
        if (quoteEscape && *read == quote) {
            read++;
            if ((read < end) && (*read == quote)) {
                // two consecutive quote/escape characters is an escaped quote
                *write++ = *read++;
            } else {
                // single quote/escape is end quote
                break;
            }
        } else if (quoted && *read == quote) {
            break;
        } else if (*read == escape) {
            read++;
            if (read < end) {
                *write++ = *read++;
            }
        } else {
            *write++ = *read++;
        }
    }
    return write - buffer;
}

uint FlatFileParser::trim	(	char *	buffer,
		uint	size
	)

Trim spaces from beginning and end of text.

Parameters:

[in,out]	buffer	buffer containing text
[in]	size	size of text, in characters

Returns:

size of resulting column text

Definition at line 414 of file FlatFileParser.cpp.

Referenced by FlatFileExecStreamTest::checkTrim(), and stripQuoting().

{
    assert(buffer != NULL);
    if (size == 0) {
        return 0;
    }
    char *read = buffer;
    char *write = buffer;
    char *end = buffer + size;

    while (read < end && *read == ' ') {
        read++;
    }
    end--;
    while (end >= read && *end == ' ') {
        end--;
    }
    end++;
    while (read < end) {
        *write++ = *read++;
    }
    return write - buffer;
}

---

Member Data Documentation

char FlatFileParser::fieldDelim [private]

Definition at line 391 of file FlatFileParser.h.

Referenced by scanColumn().

char FlatFileParser::rowDelim [private]

Definition at line 392 of file FlatFileParser.h.

Referenced by isRowDelim(), and scanRow().

char FlatFileParser::quote [private]

Definition at line 393 of file FlatFileParser.h.

Referenced by scanColumn(), and stripQuoting().

char FlatFileParser::escape [private]

Definition at line 394 of file FlatFileParser.h.

Referenced by scanColumn(), and stripQuoting().

bool FlatFileParser::doTrim [private]

Definition at line 395 of file FlatFileParser.h.

Referenced by scanColumn().

bool FlatFileParser::fixed [private]

Whether to perform fixed mode parsing.

Definition at line 400 of file FlatFileParser.h.

Referenced by FlatFileParser(), and scanColumn().

---

The documentation for this class was generated from the following files:

• /home/pub/open/dev/fennel/flatfile/FlatFileParser.h
• /home/pub/open/dev/fennel/flatfile/FlatFileParser.cpp

---