This fileManager is intended to allow different objects to share the same open file. More...

#include <fileManager.h>

Classes
class	buffer
	A buffer is intended to be a temporary workspace in memory. More...

class	cleaner
	A function object to be used to register external cleaners. More...

class	roFile
	This class manages content of a whole (read-only) file. More...

class	storage
	The storage class treats all memory as char*. More...

Public Types
enum	ACCESS_PREFERENCE { MMAP_LARGE_FILES, PREFER_READ, PREFER_MMAP }
	Hint passed to the function `getFile`. More...

Public Member Functions
void	addCleaner (const cleaner *cl)
	Register an external cleaner functor. More...

void	clear ()
	Close all files and remove all records of them. More...

void	flushDir (const char *name)
	Close all files in the named directory, including subdirectories.

void	flushFile (const char *name)
	Close the file, remove the record about it from the file manager.

template<typename T >
int	getFile (const char *name, array_t< T > &arr, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
	Given a file name, place the content in an array_t<T>. More...

int	getFile (const char name, storage *st, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
	Retrieve the file content as a storage object. More...

uint64_t	getMaxMmapBytes () const
	Return the size in bytes of files that are memory mapped.

unsigned int	getMaxOpenMmapFiles () const
	Return the count of files that are memory mapped.

const double &	pageCount () const
	Returns the number of pages accessed by function read from stdlib.h.

void	printStatus (std::ostream &out) const
	Prints status information about the file manager.

void	recordPages (off_t start, off_t stop)
	Given the starting and ending addresses, this function computes the number of pages involved. More...

void	removeCleaner (const cleaner *cl)
	Unregister the cleaner functor. More...

void	signalMemoryAvailable () const
	Signal to the file manager that some memory have been freed. More...

template<typename T >
int	tryGetFile (const char *name, array_t< T > &arr, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
	Given a file name, place the content in an array_t<T>. More...

int	tryGetFile (const char name, storage *st, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
	Try to retrieve the content of the named file. More...

Static Public Member Functions
static int	adjustCacheSize (uint64_t)
	Change the size of memory cache allocated to the file manager. More...

static uint64_t	bytesFree ()
	Return the number of bytes free.

static uint64_t	bytesInUse ()
	Returns the number of bytes currently on records.

static uint64_t	currentCacheSize ()
	A read lock on the file manager. Any object using a file under the. More...

static void	decreaseUse (size_t dec, const char *evt)

static storage *	getFileSegment (const char *name, const int fdes, const off_t b, const off_t e)
	Retrieve a portion of a file content. More...

static time_t	iBeat ()
	Returns the value of a simple counter. It is not thread-safe!

static void	increaseUse (size_t inc, const char *evt)

static fileManager &	instance ()
	Returns a pointer to the one and only file manager. More...

static uint32_t	pageSize ()
	Returns the page size (in bytes) used by the file system.

Protected Member Functions
	fileManager ()
	The protected constructor of the ibis::fileManager class. More...

	fileManager (const fileManager &rhs)

fileManager &	operator= (const fileManager &rhs)

void	recordFile (roFile *)
	Record a newly allocated storage in the two lists. More...

void	unrecordFile (roFile *)
	Remove the file name from the list of files tracked. More...

	~fileManager ()
	Destructor.

Static Protected Attributes
static uint64_t	maxBytes = 0
	The maximum number of bytes allowed.

static unsigned int	maxOpenFiles = 0
	The Maximum number of files that can be kept open at the same time.

static ibis::util::sharedInt64	totalBytes
	The total number of bytes of all managed objects.

Friends
class	roFile

class	storage

Detailed Description

This fileManager is intended to allow different objects to share the same open file.

It does not manage writing of files.

Member Enumeration Documentation

enum ibis::fileManager::ACCESS_PREFERENCE

Hint passed to the function getFile.

The main choice is whether to use memory map or use the read function to access the content of a file.

Constructor & Destructor Documentation

ibis::fileManager::fileManager ( )

protected

The protected constructor of the ibis::fileManager class.

There are three parameters that can be specified in a configuration file to control this object, fileManager.maxBytes, fileManager.maxOpenFiles, and fileManager.minMapSize. If you are unsure of what to do, then don't specify anything – the default values are typically acceptable.

filemanager.maxBytes The maximum number of bytes of all objects under control of the file manager, e.g.,
```
/// fileManager.maxBytes = 500MB
/// 
```
One may specify a number followed by KB, MB, or GB (without space in between). If not specified, this constructor attempts to determine the size of the physical memory available and will use half of the memory for caching FastBit objects.

fileManager.maxOpenFiles This file manager will keep the number of open files below this specified maximum. Note that FastBit usually invokes the lower level function open, which typical can use more file handles than the higher level ones such as fopen. If not specified, it will use three quarters of maximum file halder defined by _SC_OPEN_MAX.

fileManager.minMapSize The minimal size of a file before FastBit will attempt to use memory map on it. For smaller files, it is more efficient to read the whole content into memory rather than keeping a file open. The default value is defined by the macro FASTBIT_MIN_MAP_SIZE.

References ibis::resource::getNumber(), ibis::gParameters(), maxBytes, and maxOpenFiles.

Member Function Documentation

void ibis::fileManager::addCleaner ( const cleaner * cl )

Register an external cleaner functor.

It will be invoked when the file manager runs out of space.

Referenced by ibis::part::part().

int ibis::fileManager::adjustCacheSize ( uint64_t newsize )

static

Change the size of memory cache allocated to the file manager.

Change the class variable maxBytes to the newsize in bytes.

Return 0 if successful, a negative number otherwise.

This function simply changes the maximum bytes allowed, without enforcing this limit. Future operations that require more memory will be subjected to the new cache size limit.

Reducing the cache size while there are on-going operations can have very undesirable effects, therefore this function will not accept a new size if it is less than the current number of bytes in memory. It might be helpful to call ibis::fileManager::clear to reduce the memory usage before changing the cache size.

References ibis::util::envLock, and totalBytes.

void ibis::fileManager::clear ( )

Close all files and remove all records of them.

This function cleans the memory cache used by FastBit.

It clears the two lists of files held by this class and therefore makes the files not accessible to any new objects. Important note, the actual underlying memory may still be present if they are being actively used. This function is effective only if all other operations have ceased!

To force an individual file to be unloaded use ibis::fileManager::flushFile. To force all files in a directory to be unloaded used ibis::fileManager::flushDir.

Note: This function will not do anything if it is not able to acquire a write lock in the file manager object.

References ibis::util::groupby1000(), and totalBytes.

Referenced by fastbit_cleanup(), and ibis::part::rollback().

static uint64_t ibis::fileManager::currentCacheSize ( )

inlinestatic

A read lock on the file manager. Any object using a file under the.

Return the current cache size in bytes.

References maxBytes.

template<typename T >

int ibis::fileManager::getFile	(	const char *	name,
		array_t< T > &	arr,
		ACCESS_PREFERENCE	pref = `MMAP_LARGE_FILES`
	)

Given a file name, place the content in an array_t<T>.

This function waits for memory to become available if there is enough memory to read the file content into memory. The compiler macro FASTBIT_MAX_WAIT_TIME defines the maximum amount of time (in seconds) it may wait.

The return value is zero (0) if the function is successful, otherwise returns a non-zero value.

References ibis::array_t< T >::clear(), ibis::array_t< T >::size(), and ibis::array_t< T >::swap().

int ibis::fileManager::getFile	(	const char *	name,
		storage **	st,
		ACCESS_PREFERENCE	pref = `MMAP_LARGE_FILES`
	)

Retrieve the file content as a storage object.

The object *st returned from this function is owned by the fileManager. The caller should NOT delete *st! This function will wait for the fileManager to unload some in-memory objects if there isn't enough memory for the file.

Upon successful completion of the task, it returns zero; otherwise, it returns a non-zero value to indicate an error and it does not modify the content of storage object.

References ibis::fileManager::storage::begin(), ibis::horometer::CPUTime(), ibis::fileManager::roFile::doRead(), ibis::fileManager::storage::enlarge(), ibis::util::groupby1000(), ibis::fileManager::roFile::isFileMap(), ibis::fileManager::roFile::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), ibis::horometer::stop(), and totalBytes.

ibis::fileManager::storage * ibis::fileManager::getFileSegment	(	const char *	name,
		const int	fdes,
		const off_t	b,
		const off_t	e
	)

static

Retrieve a portion of a file content.

Both the file name and the file descriptor are passed in to this function so that it can make a decision on whether to use a file map or directly read the content into memory. It prefers the read option more because the caller is more like to touch every bytes in an explicitly specified portion of a file. More specifically, it uses file map only if the file of the file segement is 4 * FASTBIT_MIN_MAP_SIZE and the number of mapped files is less than half of the maxOpenFiles.

References ibis::horometer::CPUTime(), ibis::util::groupby1000(), instance(), ibis::fileManager::storage::isFileMap(), ibis::fileManager::storage::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), and ibis::horometer::stop().

ibis::fileManager & ibis::fileManager::instance ( )

static

Returns a pointer to the one and only file manager.

The instance function of the fileManager singleton.

void ibis::fileManager::recordFile ( roFile * st )

protected

Record a newly allocated storage in the two lists.

The caller needs to hold a mutex lock on the file manager to ensure correct operations.

References ibis::fileManager::storage::begin(), ibis::fileManager::storage::end(), ibis::fileManager::storage::filename(), and ibis::fileManager::storage::size().

void ibis::fileManager::recordPages	(	off_t	start,
		off_t	stop
	)

inline

Given the starting and ending addresses, this function computes the number of pages involved.

Used by derived classes to record page accesses.

Referenced by ibis::part::doCompare(), ibis::fileManager::roFile::doRead(), ibis::column::findLower(), ibis::text::findString(), ibis::column::findUpper(), ibis::part::negativeCompare(), ibis::relic::read(), ibis::bin::read(), ibis::direkte::read(), ibis::keywords::read(), ibis::fileManager::storage::read(), ibis::skive::read(), ibis::fade::read(), ibis::range::read(), ibis::fuzz::read(), ibis::bylt::read(), ibis::zona::read(), ibis::ambit::read(), ibis::pale::read(), ibis::pack::read(), ibis::zone::read(), ibis::fuge::read(), ibis::egale::read(), ibis::bundle::readRIDs(), ibis::text::readString(), ibis::text::readStrings1(), ibis::text::readStrings2(), ibis::column::searchSortedOOCC(), ibis::column::searchSortedOOCD(), ibis::column::selectValuesT(), ibis::column::string2int(), and ibis::text::stringSearch().

void ibis::fileManager::removeCleaner ( const cleaner * cl )

Unregister the cleaner functor.

Typically, this is only invoked when the object is being freed. For example, in the destructor of ibis::part, the cleaner associated with the partition is unregistered.

Referenced by ibis::part::clear(), and ibis::part::~part().

void ibis::fileManager::signalMemoryAvailable ( ) const

Signal to the file manager that some memory have been freed.

To be used by clients that are aware of the memory usages of in-memory objects since the in-memory objects based on ibis::fileManager::storage does not produce signals when they are freed.

References instance().

Referenced by ibis::relic::construct().

template<typename T >

int ibis::fileManager::tryGetFile	(	const char *	name,
		array_t< T > &	arr,
		ACCESS_PREFERENCE	pref = `MMAP_LARGE_FILES`
	)

Given a file name, place the content in an array_t<T>.

This function will fail if there isn't enough memory to read the content of the file immediately.

The return value is zero (0) if the function is successful, otherwise returns a non-zero value.

References ibis::array_t< T >::clear(), ibis::array_t< T >::size(), and ibis::array_t< T >::swap().

Referenced by ibis::index::create(), and ibis::column::selectValuesT().

int ibis::fileManager::tryGetFile	(	const char *	name,
		storage **	st,
		ACCESS_PREFERENCE	pref = `MMAP_LARGE_FILES`
	)

Try to retrieve the content of the named file.

The storage object *st returned from this function is owned by the fileManager. The caller is NOT to delete *st. This function will not wait for the fileManager to free any memory if there isn't enough free space available.

It returns 0 to indicate success and a negative value to indicate error. In particular, it returns -102 if there is not enough space to read the whole file into memory.

References ibis::fileManager::storage::begin(), ibis::horometer::CPUTime(), ibis::fileManager::roFile::doRead(), ibis::fileManager::storage::enlarge(), ibis::fileManager::roFile::isFileMap(), ibis::fileManager::roFile::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), ibis::horometer::stop(), and totalBytes.

void ibis::fileManager::unrecordFile ( roFile * st )

protected

Remove the file name from the list of files tracked.

This operation can only be performed on a file whose content has been read into memory, not on a file being mapped.

Note: The caller is expected to hold a mutex lock on the file manager.

References ibis::fileManager::storage::begin(), ibis::fileManager::storage::filename(), and ibis::fileManager::storage::size().

Referenced by ibis::fileManager::roFile::disconnectFile().

The documentation for this class was generated from the following files:

fileManager.h
fileManager.cpp



	Contact us Disclaimers FastBit source code FastBit mailing list archive

Classes

Public Types

Public Member Functions

Static Public Member Functions

Protected Member Functions

Static Protected Attributes

Friends

Detailed Description

Member Enumeration Documentation

Constructor & Destructor Documentation

Member Function Documentation