CrcSegOutputStream Class Reference

CrcSegOutputStream extends SegOutputStream by stamping each page with a CRC/PseudoUuid/PageId combination. More...

#include <CrcSegOutputStream.h>

Inheritance diagram for CrcSegOutputStream:

List of all members.

Public Member Functions
PageId	getFirstPageId () const
	Gets the first PageId allocated.
BlockNum	getPageCount () const
	Returns: the number of pages allocated by this stream
void	updatePage ()
	Updates current page header without unlocking; allows a SegInputStream to read the contents from the same thread (but another thread would be locked out).
virtual void	getSegPos (SegStreamPosition &pos)
	Obtains the current stream position.
SharedSegment	getSegment () const
	Returns: segment accessed by this stream
SegmentAccessor const &	getSegmentAccessor () const
	Returns: segment accessor used by this stream
FileSize	getOffset () const
	Returns: current offset from beginning of stream
bool	isClosed () const
	Returns: whether the object has been closed
void	close ()
	Closes this object, releasing any unallocated resources.
void	writeBytes (void const *pData, uint cbRequested)
	Writes bytes to the stream.
PBuffer	getWritePointer (uint cbRequested, uint *pcbActual=NULL)
	Copyless alternative for writing bytes to the stream.
void	consumeWritePointer (uint cbUsed)
	Advances stream position after a call to getWritePointer.
void	hardPageBreak ()
	Marks the current buffer as complete regardless of how much data it contains.
virtual void	setWriteLatency (WriteLatency writeLatency)
	Changes the write latency.
template<class T>
void	writeValue (T const &value)
	Writes a fixed-size type to the stream.
Static Public Member Functions
static SharedSegOutputStream	newCrcSegOutputStream (SegmentAccessor const &segmentAccessor, PseudoUuid onlineUuid)
	Creates a new CrcSegOutputStream.
static SharedSegOutputStream	newSegOutputStream (SegmentAccessor const &segmentAccessor)
	Creates a new SegOutputStream.
Protected Member Functions
uint	getBytesWrittenThisPage () const
	Returns: number of bytes already written to current page
virtual void	flushBuffer (uint cbRequested)
	Must be implemented by derived class to flush buffered data.
virtual void	closeImpl ()
	Must be implemented by derived class to release any resources.
void	setBuffer (PBuffer pBuffer, uint cbBuffer)
	Sets the current buffer to be written.
uint	getBytesAvailable () const
	Returns: number of bytes remaining to be written in current buffer
Protected Attributes
PageId	firstPageId
	First PageId allocated.
PageId	lastPageId
	Last PageId allocated.
uint	cbMaxPageData
	Maximum number of data bytes which can be stored per page (does not include header).
BlockNum	nPagesAllocated
	Counter for getPageCount().
SegmentAccessor	segmentAccessor
	Accessor for segment containing stream data.
SegStreamLock	pageLock
	Lock held on current page, if any.
uint	cbPageHeader
	Number of bytes in page header.
FileSize	cbOffset
	Byte position in stream.
bool	needsClose
WriteLatency	writeLatency
	Current write latency mode.
Private Member Functions
virtual void	writeExtraHeaders (SegStreamNode &node)
	Hook for subclasses; default is to do nothing.
	CrcSegOutputStream (SegmentAccessor const &segmentAccessor, PseudoUuid onlineUuid)
Private Attributes
PseudoUuid	onlineUuid
boost::crc_32_type	crcComputer

---

Detailed Description

CrcSegOutputStream extends SegOutputStream by stamping each page with a CRC/PseudoUuid/PageId combination.

This can be used to reliably detect end-of-stream during recovery.

Definition at line 38 of file CrcSegOutputStream.h.

---

Constructor & Destructor Documentation

CrcSegOutputStream::CrcSegOutputStream	(	SegmentAccessor const &	segmentAccessor,
		PseudoUuid	onlineUuid
	)			[explicit, private]

Definition at line 39 of file CrcSegOutputStream.cpp.

References onlineUuid.

Referenced by newCrcSegOutputStream().

    : SegOutputStream(segmentAccessorInit,sizeof(SegStreamCrc))
{
    onlineUuid = onlineUuidInit;
}

---

Member Function Documentation

void CrcSegOutputStream::writeExtraHeaders

(

SegStreamNode &

node

)

[private, virtual]

Hook for subclasses; default is to do nothing.

Parameters:

node

the node being flushed

Reimplemented from SegOutputStream.

Definition at line 47 of file CrcSegOutputStream.cpp.

References SegStreamNode::cbData, SegStreamCrc::checksum, crcComputer, SegOutputStream::lastPageId, onlineUuid, SegStreamCrc::onlineUuid, and SegStreamCrc::pageId.

{
    SegStreamCrc *pCrc = reinterpret_cast<SegStreamCrc *>(&node+1);
    pCrc->onlineUuid = onlineUuid;
    pCrc->pageId = lastPageId;
    crcComputer.reset();
    crcComputer.process_bytes(pCrc + 1,node.cbData);
    pCrc->checksum = crcComputer.checksum();
}

SharedSegOutputStream CrcSegOutputStream::newCrcSegOutputStream	(	SegmentAccessor const &	segmentAccessor,
		PseudoUuid	onlineUuid
	)			[static]

Creates a new CrcSegOutputStream.

Parameters:

	segmentAccessor	accessor for the segment in which to store the data
	onlineUuid	PseudoUuid with which to stamp each page

Returns:

shared_ptr to new SegOutputStream

Definition at line 29 of file CrcSegOutputStream.cpp.

References CrcSegOutputStream(), onlineUuid, and SegStream::segmentAccessor.

Referenced by LogicalTxnLog::LogicalTxnLog().

{
    return SharedSegOutputStream(
        new CrcSegOutputStream(
            segmentAccessor,onlineUuid),
        ClosableObjectDestructor());
}

uint SegOutputStream::getBytesWrittenThisPage

(

)

const [protected, inherited]

Returns:

number of bytes already written to current page

Definition at line 110 of file SegOutputStream.cpp.

References SegOutputStream::cbMaxPageData, and ByteOutputStream::getBytesAvailable().

Referenced by SegOutputStream::getSegPos(), and SegOutputStream::updatePage().

{
    return cbMaxPageData - getBytesAvailable();
}

void SegOutputStream::flushBuffer

(

uint

cbRequested

)

[protected, virtual, inherited]

Must be implemented by derived class to flush buffered data.

Parameters:

cbRequested

if non-zero, the derived class should allocate a new buffer with at least the requested size and call setBuffer

Implements ByteOutputStream.

Definition at line 75 of file SegOutputStream.cpp.

References SegNodeLock< Node >::allocatePage(), SegOutputStream::cbMaxPageData, SegStream::cbPageHeader, SegOutputStream::firstPageId, SegPageLock::getCacheAccessor(), SegNodeLock< Node >::getNodeForWrite(), SegPageLock::getPage(), SegStream::getSegment(), SegPageLock::isLocked(), SegOutputStream::lastPageId, SegOutputStream::nPagesAllocated, NULL_PAGE_ID, SegStream::pageLock, ByteOutputStream::setBuffer(), SegPageLock::unlock(), SegOutputStream::updatePage(), WRITE_EAGER_ASYNC, WRITE_LAZY, and ByteOutputStream::writeLatency.

{
    assert(cbRequested <= cbMaxPageData);
    updatePage();
    if (pageLock.isLocked()) {
        if (writeLatency != WRITE_LAZY) {
            pageLock.getCacheAccessor()->flushPage(
                pageLock.getPage(),
                writeLatency == WRITE_EAGER_ASYNC);
        }
        pageLock.unlock();
    }
    if (!cbRequested) {
        return;
    }
    PageId pageId = pageLock.allocatePage();
    ++nPagesAllocated;
    if (firstPageId == NULL_PAGE_ID) {
        firstPageId = pageId;
    } else {
        getSegment()->setPageSuccessor(lastPageId,pageId);
    }
    lastPageId = pageId;
    SegStreamNode &node = pageLock.getNodeForWrite();
    setBuffer(
        reinterpret_cast<PBuffer>(&node)+cbPageHeader,
        cbMaxPageData);
}

void SegOutputStream::closeImpl

(

)

[protected, virtual, inherited]

Must be implemented by derived class to release any resources.

Reimplemented from ByteOutputStream.

Definition at line 104 of file SegOutputStream.cpp.

References SegStream::closeImpl(), and ByteOutputStream::closeImpl().

{
    ByteOutputStream::closeImpl();
    SegStream::closeImpl();
}

SharedSegOutputStream SegOutputStream::newSegOutputStream

(

SegmentAccessor const &

segmentAccessor

)

[static, inherited]

Creates a new SegOutputStream.

Parameters:

segmentAccessor

accessor for the segment in which to store the data

Returns:

shared_ptr to new SegOutputStream

Definition at line 29 of file SegOutputStream.cpp.

References SegStream::segmentAccessor, and SegOutputStream::SegOutputStream().

Referenced by LhxPartitionWriter::open(), SpillOutputStream::spill(), ExternalSortRunAccessor::storeRun(), BTreeTest::testBulkLoad(), BTreeReadersTest::testReaders(), SegStreamTest::testWriteSeg(), VariableBuildLevel::VariableBuildLevel(), and SegBufferWriter::write().

{
    return SharedSegOutputStream(
        new SegOutputStream(segmentAccessor),
        ClosableObjectDestructor());
}

PageId SegOutputStream::getFirstPageId

(

)

const [inherited]

Gets the first PageId allocated.

For non-linear segments, this is required in order to be able to read the data back via SegInputStream.

Returns:

the first PageId allocated, or NULL_PAGE_ID if no data has been written to the stream yet

Definition at line 51 of file SegOutputStream.cpp.

References SegOutputStream::firstPageId.

{
    return firstPageId;
}

BlockNum SegOutputStream::getPageCount

(

)

const [inherited]

Returns:

the number of pages allocated by this stream

Definition at line 56 of file SegOutputStream.cpp.

References SegOutputStream::nPagesAllocated.

{
    return nPagesAllocated;
}

void SegOutputStream::updatePage

(

)

[inherited]

Updates current page header without unlocking; allows a SegInputStream to read the contents from the same thread (but another thread would be locked out).

Definition at line 61 of file SegOutputStream.cpp.

References SegStreamNode::cbData, SegOutputStream::getBytesWrittenThisPage(), SegNodeLock< Node >::getNodeForWrite(), SegPageLock::isLocked(), SegStream::pageLock, and SegOutputStream::writeExtraHeaders().

Referenced by SegOutputStream::flushBuffer().

{
    if (!pageLock.isLocked()) {
        return;
    }
    SegStreamNode &node = pageLock.getNodeForWrite();
    node.cbData = getBytesWrittenThisPage();
    writeExtraHeaders(node);
}

void SegOutputStream::getSegPos

(

SegStreamPosition &

pos

)

[virtual, inherited]

Obtains the current stream position.

Parameters:

pos

receives the position

Implements SegStream.

Definition at line 115 of file SegOutputStream.cpp.

References ByteStream::cbOffset, SegStreamPosition::cbOffset, ByteOutputStream::getBytesAvailable(), SegOutputStream::getBytesWrittenThisPage(), SegOutputStream::lastPageId, CompoundId::MAX_BYTE_OFFSET, SegStreamPosition::segByteId, CompoundId::setByteOffset(), and CompoundId::setPageId().

{
    CompoundId::setPageId(pos.segByteId,lastPageId);
    if (getBytesAvailable()) {
        CompoundId::setByteOffset(pos.segByteId,getBytesWrittenThisPage());
    } else {
        // after a hard page break, use a special sentinel value to indicate
        // the last byte on the page
        CompoundId::setByteOffset(
            pos.segByteId,CompoundId::MAX_BYTE_OFFSET);
    }
    pos.cbOffset = cbOffset;
}

SharedSegment SegStream::getSegment

(

)

const [inherited]

Returns:

segment accessed by this stream

Definition at line 43 of file SegStream.cpp.

References SegmentAccessor::pSegment, and SegStream::segmentAccessor.

Referenced by SegInputStream::closeImpl(), SegOutputStream::flushBuffer(), SegInputStream::readNextBuffer(), SegInputStream::readPrevBuffer(), SegInputStream::SegInputStream(), and SegOutputStream::SegOutputStream().

{
    return segmentAccessor.pSegment;
}

SegmentAccessor const & SegStream::getSegmentAccessor

(

)

const [inherited]

Returns:

segment accessor used by this stream

Definition at line 48 of file SegStream.cpp.

References SegStream::segmentAccessor.

{
    return segmentAccessor;
}

FileSize ByteStream::getOffset

(

)

const [inline, inherited]

Returns:

current offset from beginning of stream

Definition at line 110 of file ByteStream.h.

References ByteStream::cbOffset.

Referenced by SpillOutputStream::getInputStream(), and ByteInputStream::mark().

{
    return cbOffset;
}

bool ClosableObject::isClosed

(

)

const [inline, inherited]

Returns:

whether the object has been closed

Definition at line 58 of file ClosableObject.h.

    {
        return !needsClose;
    }

void ClosableObject::close

(

)

[inherited]

Closes this object, releasing any unallocated resources.

Reimplemented in CollectExecStream, CorrelationJoinExecStream, LcsClusterAppendExecStream, and LcsClusterReplaceExecStream.

Definition at line 39 of file ClosableObject.cpp.

References ClosableObject::closeImpl(), and ClosableObject::needsClose.

Referenced by CacheImpl< PageT, VictimPolicyT >::allocatePages(), LcsRowScanBaseExecStream::closeImpl(), ExecStreamGraphImpl::closeImpl(), FlatFileBuffer::open(), ClosableObjectDestructor::operator()(), and Segment::~Segment().

{
    if (!needsClose) {
        return;
    }
    needsClose = false;
    closeImpl();
}

void ByteOutputStream::setBuffer	(	PBuffer	pBuffer,
		uint	cbBuffer
	)			[inline, protected, inherited]

Sets the current buffer to be written.

Parameters:

	pBuffer	receives start address of new buffer
	cbBuffer	number of bytes in buffer

Definition at line 162 of file ByteOutputStream.h.

References ByteOutputStream::cbWritable, and ByteOutputStream::pNextByte.

Referenced by ByteArrayOutputStream::ByteArrayOutputStream(), ByteArrayOutputStream::clear(), SpillOutputStream::flushBuffer(), SegOutputStream::flushBuffer(), and SpillOutputStream::SpillOutputStream().

{
    pNextByte = pBuffer;
    cbWritable = cbBuffer;
}

uint ByteOutputStream::getBytesAvailable

(

)

const [inline, protected, inherited]

Returns:

number of bytes remaining to be written in current buffer

Definition at line 168 of file ByteOutputStream.h.

References ByteOutputStream::cbWritable.

Referenced by SpillOutputStream::closeImpl(), SpillOutputStream::flushBuffer(), SegOutputStream::getBytesWrittenThisPage(), SegOutputStream::getSegPos(), SpillOutputStream::spill(), and SpillOutputStream::updatePage().

{
    return cbWritable;
}

void ByteOutputStream::writeBytes	(	void const *	pData,
		uint	cbRequested
	)			[inherited]

Writes bytes to the stream.

Parameters:

	pData	source buffer containing bytes to be written
	cbRequested	number of bytes to write

Definition at line 35 of file ByteOutputStream.cpp.

References ByteStream::cbOffset, ByteOutputStream::cbWritable, ByteOutputStream::flushBuffer(), and ByteOutputStream::pNextByte.

Referenced by FtrsTableWriter::execute().

{
    cbOffset += cb;
    if (!cbWritable) {
        flushBuffer(1);
    }
    for (;;) {
        assert(cbWritable);
        if (cb <= cbWritable) {
            memcpy(pNextByte,pData,cb);
            cbWritable -= cb;
            pNextByte += cb;
            return;
        }
        memcpy(pNextByte,pData,cbWritable);
        pData = static_cast<char const *>(pData) + cbWritable;
        cb -= cbWritable;
        cbWritable = 0;
        flushBuffer(1);
    }
}

PBuffer ByteOutputStream::getWritePointer	(	uint	cbRequested,
		uint *	pcbActual = NULL
	)			[inline, inherited]

Copyless alternative for writing bytes to the stream.

Provides direct access to the stream's internal buffer, but doesn't move the stream position (see consumeWritePointer).

Parameters:

	cbRequested	number of contiguous bytes to access; if fewer bytes are currently available in the buffer, the buffer is flushed and a new buffer is returned
	pcbActual	if non-NULL, receives actual number of contiguous writable bytes, which will always be greater than or equal to cbRequested

Returns:

pointer to cbActual bytes of writable buffer space

Definition at line 141 of file ByteOutputStream.h.

References ByteOutputStream::cbWritable, ByteOutputStream::flushBuffer(), and ByteOutputStream::pNextByte.

Referenced by BTreeWriter::deleteCurrent(), BTreeWriter::insertTupleFromBuffer(), and SegOutputStream::SegOutputStream().

{
    if (cbWritable < cbRequested) {
        flushBuffer(cbRequested);
        assert(cbWritable >= cbRequested);
    }
    if (pcbActual) {
        *pcbActual = cbWritable;
    }
    return pNextByte;
}

void ByteOutputStream::consumeWritePointer

(

uint

cbUsed

)

[inline, inherited]

Advances stream position after a call to getWritePointer.

Parameters:

cbUsed

number of bytes to advance; must be less than or equal to the value of cbActual returned by the last call to getWritePointer

Definition at line 154 of file ByteOutputStream.h.

References ByteStream::cbOffset, ByteOutputStream::cbWritable, and ByteOutputStream::pNextByte.

Referenced by BTreeWriter::deleteCurrent(), and BTreeWriter::insertTupleFromBuffer().

{
    assert(cbUsed <= cbWritable);
    cbWritable -= cbUsed;
    pNextByte += cbUsed;
    cbOffset += cbUsed;
}

void ByteOutputStream::hardPageBreak

(

)

[inherited]

Marks the current buffer as complete regardless of how much data it contains.

The exact semantics are dependent on the buffering implementation.

Definition at line 62 of file ByteOutputStream.cpp.

References ByteOutputStream::cbWritable, ByteOutputStream::flushBuffer(), and ByteOutputStream::pNextByte.

{
    flushBuffer(0);
    cbWritable = 0;
    pNextByte = NULL;
}

void ByteOutputStream::setWriteLatency

(

WriteLatency

writeLatency

)

[virtual, inherited]

Changes the write latency.

May not be meaningful for all stream implementations.

Parameters:

writeLatency

new WriteLatency setting

Reimplemented in SpillOutputStream.

Definition at line 69 of file ByteOutputStream.cpp.

References ByteOutputStream::writeLatency.

Referenced by SpillOutputStream::setWriteLatency().

{
    writeLatency = writeLatencyInit;
}

template<class T>

void ByteOutputStream::writeValue

(

T const &

value

)

[inline, inherited]

Writes a fixed-size type to the stream.

Parameters:

value

value to read; type must be memcpy-safe

Definition at line 135 of file ByteOutputStream.h.

Referenced by FtrsTableWriter::describeIndex(), LogicalTxnTest::describeParticipant(), FtrsTableWriter::describeParticipant(), BTreeWriter::describeParticipant(), DatabaseTest::executeIncrementAction(), TupleProjection::writePersistent(), and TupleDescriptor::writePersistent().

    {
        writeBytes(&value,sizeof(value));
    }

---

Member Data Documentation

PseudoUuid CrcSegOutputStream::onlineUuid [private]

Definition at line 41 of file CrcSegOutputStream.h.

Referenced by CrcSegOutputStream(), newCrcSegOutputStream(), and writeExtraHeaders().

boost::crc_32_type CrcSegOutputStream::crcComputer [private]

Definition at line 44 of file CrcSegOutputStream.h.

Referenced by writeExtraHeaders().

PageId SegOutputStream::firstPageId [protected, inherited]

First PageId allocated.

Definition at line 44 of file SegOutputStream.h.

Referenced by SegOutputStream::flushBuffer(), SegOutputStream::getFirstPageId(), and SegOutputStream::SegOutputStream().

PageId SegOutputStream::lastPageId [protected, inherited]

Last PageId allocated.

Definition at line 49 of file SegOutputStream.h.

Referenced by SegOutputStream::flushBuffer(), SegOutputStream::getSegPos(), SegOutputStream::SegOutputStream(), and writeExtraHeaders().

uint SegOutputStream::cbMaxPageData [protected, inherited]

Maximum number of data bytes which can be stored per page (does not include header).

Definition at line 55 of file SegOutputStream.h.

Referenced by SegOutputStream::flushBuffer(), SegOutputStream::getBytesWrittenThisPage(), and SegOutputStream::SegOutputStream().

BlockNum SegOutputStream::nPagesAllocated [protected, inherited]

Counter for getPageCount().

Definition at line 60 of file SegOutputStream.h.

Referenced by SegOutputStream::flushBuffer(), SegOutputStream::getPageCount(), and SegOutputStream::SegOutputStream().

SegmentAccessor SegStream::segmentAccessor [protected, inherited]

Accessor for segment containing stream data.

Definition at line 97 of file SegStream.h.

Referenced by SegStream::closeImpl(), SegStreamAllocation::endWrite(), SegStream::getSegment(), SegStream::getSegmentAccessor(), CrcSegInputStream::lockBufferParanoid(), CrcSegInputStream::newCrcSegInputStream(), newCrcSegOutputStream(), SegInputStream::newSegInputStream(), SegOutputStream::newSegOutputStream(), and SegInputStream::startPrefetch().

SegStreamLock SegStream::pageLock [protected, inherited]

Lock held on current page, if any.

Definition at line 102 of file SegStream.h.

Referenced by SegStream::closeImpl(), SegInputStream::closeImpl(), SegOutputStream::flushBuffer(), SegInputStream::lockBuffer(), CrcSegInputStream::lockBuffer(), CrcSegInputStream::lockBufferParanoid(), SegInputStream::readNextBuffer(), and SegOutputStream::updatePage().

uint SegStream::cbPageHeader [protected, inherited]

Number of bytes in page header.

Definition at line 107 of file SegStream.h.

Referenced by SegOutputStream::flushBuffer(), SegInputStream::lockBuffer(), CrcSegInputStream::lockBufferParanoid(), SegOutputStream::SegOutputStream(), and SegStream::SegStream().

FileSize ByteStream::cbOffset [protected, inherited]

Byte position in stream.

Definition at line 41 of file ByteStream.h.

Referenced by ByteStream::ByteStream(), ByteInputStream::consumeReadPointer(), ByteOutputStream::consumeWritePointer(), ByteStream::getOffset(), SegOutputStream::getSegPos(), SegInputStream::getSegPos(), ByteInputStream::readBytes(), ByteInputStream::reset(), ByteArrayInputStream::resetArray(), ByteInputStream::seekBackward(), SegInputStream::seekSegPos(), and ByteOutputStream::writeBytes().

bool ClosableObject::needsClose [protected, inherited]

Definition at line 44 of file ClosableObject.h.

Referenced by SegStreamAllocation::beginWrite(), ExecStreamGraphImpl::clear(), ClosableObject::ClosableObject(), ClosableObject::close(), FlatFileBuffer::open(), ExecStreamGraphImpl::open(), ExecStream::open(), and ClosableObject::~ClosableObject().

WriteLatency ByteOutputStream::writeLatency [protected, inherited]

Current write latency mode.

Definition at line 51 of file ByteOutputStream.h.

Referenced by SegOutputStream::flushBuffer(), SegOutputStream::SegOutputStream(), ByteOutputStream::setWriteLatency(), and SpillOutputStream::spill().

---

The documentation for this class was generated from the following files:

• /home/pub/open/dev/fennel/segment/CrcSegOutputStream.h
• /home/pub/open/dev/fennel/segment/CrcSegOutputStream.cpp

---