Classes | Public Types | Public Member Functions | Static Public Member Functions | Protected Member Functions | Static Protected Attributes | Friends | List of all members
ibis::fileManager Class Reference

This fileManager is intended to allow different objects to share the same open file. More...

#include <fileManager.h>

Classes

class  buffer
 A buffer is intended to be a temporary workspace in memory. More...
 
class  cleaner
 A function object to be used to register external cleaners. More...
 
class  roFile
 This class manages content of a whole (read-only) file. More...
 
class  storage
 The storage class treats all memory as char*. More...
 

Public Types

enum  ACCESS_PREFERENCE { MMAP_LARGE_FILES, PREFER_READ, PREFER_MMAP }
 Hint passed to the function getFile. More...
 

Public Member Functions

void addCleaner (const cleaner *cl)
 Register an external cleaner functor. More...
 
void clear ()
 Close all files and remove all records of them. More...
 
void flushDir (const char *name)
 Close all files in the named directory, including subdirectories.
 
void flushFile (const char *name)
 Close the file, remove the record about it from the file manager.
 
template<typename T >
int getFile (const char *name, array_t< T > &arr, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Given a file name, place the content in an array_t<T>. More...
 
int getFile (const char *name, storage **st, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Retrieve the file content as a storage object. More...
 
uint64_t getMaxMmapBytes () const
 Return the size in bytes of files that are memory mapped.
 
unsigned int getMaxOpenMmapFiles () const
 Return the count of files that are memory mapped.
 
const double & pageCount () const
 Returns the number of pages accessed by function read from stdlib.h.
 
void printStatus (std::ostream &out) const
 Prints status information about the file manager.
 
void recordPages (off_t start, off_t stop)
 Given the starting and ending addresses, this function computes the number of pages involved. More...
 
void removeCleaner (const cleaner *cl)
 Unregister the cleaner functor. More...
 
void signalMemoryAvailable () const
 Signal to the file manager that some memory have been freed. More...
 
template<typename T >
int tryGetFile (const char *name, array_t< T > &arr, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Given a file name, place the content in an array_t<T>. More...
 
int tryGetFile (const char *name, storage **st, ACCESS_PREFERENCE pref=MMAP_LARGE_FILES)
 Try to retrieve the content of the named file. More...
 

Static Public Member Functions

static int adjustCacheSize (uint64_t)
 Change the size of memory cache allocated to the file manager. More...
 
static uint64_t bytesFree ()
 Return the number of bytes free.
 
static uint64_t bytesInUse ()
 Returns the number of bytes currently on records.
 
static uint64_t currentCacheSize ()
 A read lock on the file manager. Any object using a file under the. More...
 
static void decreaseUse (size_t dec, const char *evt)
 
static storagegetFileSegment (const char *name, const int fdes, const off_t b, const off_t e)
 Retrieve a portion of a file content. More...
 
static time_t iBeat ()
 Returns the value of a simple counter. It is not thread-safe!
 
static void increaseUse (size_t inc, const char *evt)
 
static fileManagerinstance ()
 Returns a pointer to the one and only file manager. More...
 
static uint32_t pageSize ()
 Returns the page size (in bytes) used by the file system.
 

Protected Member Functions

 fileManager ()
 The protected constructor of the ibis::fileManager class. More...
 
 fileManager (const fileManager &rhs)
 
fileManageroperator= (const fileManager &rhs)
 
void recordFile (roFile *)
 Record a newly allocated storage in the two lists. More...
 
void unrecordFile (roFile *)
 Remove the file name from the list of files tracked. More...
 
 ~fileManager ()
 Destructor.
 

Static Protected Attributes

static uint64_t maxBytes = 0
 The maximum number of bytes allowed.
 
static unsigned int maxOpenFiles = 0
 The Maximum number of files that can be kept open at the same time.
 
static ibis::util::sharedInt64 totalBytes
 The total number of bytes of all managed objects.
 

Friends

class roFile
 
class storage
 

Detailed Description

This fileManager is intended to allow different objects to share the same open file.

It does not manage writing of files.

Member Enumeration Documentation

Hint passed to the function getFile.

The main choice is whether to use memory map or use the read function to access the content of a file.

Constructor & Destructor Documentation

ibis::fileManager::fileManager ( )
protected

The protected constructor of the ibis::fileManager class.

There are three parameters that can be specified in a configuration file to control this object, fileManager.maxBytes, fileManager.maxOpenFiles, and fileManager.minMapSize. If you are unsure of what to do, then don't specify anything – the default values are typically acceptable.

  • filemanager.maxBytes The maximum number of bytes of all objects under control of the file manager, e.g.,
    /// fileManager.maxBytes = 500MB
    /// 
    One may specify a number followed by KB, MB, or GB (without space in between). If not specified, this constructor attempts to determine the size of the physical memory available and will use half of the memory for caching FastBit objects.
  • fileManager.maxOpenFiles This file manager will keep the number of open files below this specified maximum. Note that FastBit usually invokes the lower level function open, which typical can use more file handles than the higher level ones such as fopen. If not specified, it will use three quarters of maximum file halder defined by _SC_OPEN_MAX.
  • fileManager.minMapSize The minimal size of a file before FastBit will attempt to use memory map on it. For smaller files, it is more efficient to read the whole content into memory rather than keeping a file open. The default value is defined by the macro FASTBIT_MIN_MAP_SIZE.

References ibis::resource::getNumber(), ibis::gParameters(), maxBytes, and maxOpenFiles.

Member Function Documentation

void ibis::fileManager::addCleaner ( const cleaner cl)

Register an external cleaner functor.

It will be invoked when the file manager runs out of space.

Referenced by ibis::part::part().

int ibis::fileManager::adjustCacheSize ( uint64_t  newsize)
static

Change the size of memory cache allocated to the file manager.

Change the class variable maxBytes to the newsize in bytes.

Return 0 if successful, a negative number otherwise.

This function simply changes the maximum bytes allowed, without enforcing this limit. Future operations that require more memory will be subjected to the new cache size limit.

Reducing the cache size while there are on-going operations can have very undesirable effects, therefore this function will not accept a new size if it is less than the current number of bytes in memory. It might be helpful to call ibis::fileManager::clear to reduce the memory usage before changing the cache size.

References ibis::util::envLock, and totalBytes.

void ibis::fileManager::clear ( )

Close all files and remove all records of them.

This function cleans the memory cache used by FastBit.

It clears the two lists of files held by this class and therefore makes the files not accessible to any new objects. Important note, the actual underlying memory may still be present if they are being actively used. This function is effective only if all other operations have ceased!

To force an individual file to be unloaded use ibis::fileManager::flushFile. To force all files in a directory to be unloaded used ibis::fileManager::flushDir.

Note
This function will not do anything if it is not able to acquire a write lock in the file manager object.

References ibis::util::groupby1000(), and totalBytes.

Referenced by fastbit_cleanup(), and ibis::part::rollback().

static uint64_t ibis::fileManager::currentCacheSize ( )
inlinestatic

A read lock on the file manager. Any object using a file under the.

Return the current cache size in bytes.

References maxBytes.

template<typename T >
int ibis::fileManager::getFile ( const char *  name,
array_t< T > &  arr,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Given a file name, place the content in an array_t<T>.

This function waits for memory to become available if there is enough memory to read the file content into memory. The compiler macro FASTBIT_MAX_WAIT_TIME defines the maximum amount of time (in seconds) it may wait.

The return value is zero (0) if the function is successful, otherwise returns a non-zero value.

References ibis::array_t< T >::clear(), ibis::array_t< T >::size(), and ibis::array_t< T >::swap().

Referenced by ibis::column::actualMinMax(), ibis::relic::append(), ibis::bin::append(), ibis::direkte::append(), ibis::category::append(), ibis::range::append(), ibis::mesa::append(), ibis::ambit::append(), ibis::pale::append(), ibis::pack::append(), ibis::zone::append(), ibis::bin::binning(), ibis::bin::binningT(), ibis::bin::binOrderT(), ibis::bundle1::bundle1(), ibis::bundles::bundles(), ibis::column::computeMax(), ibis::column::computeMin(), ibis::column::computeSum(), ibis::bin::construct(), ibis::range::construct(), ibis::ambit::construct(), ibis::egale::construct(), ibis::blob::countRawBytes(), ibis::blob::getBlob(), ibis::column::getDoubleArray(), ibis::column::getFloatArray(), ibis::column::getIntArray(), ibis::column::getNullMask(), ibis::column::getRawData(), ibis::category::getString(), ibis::roster::icSearch(), ibis::roster::locate(), ibis::index::mapValues(), ibis::bak2::mapValues(), ibis::bitvector64::read(), ibis::array_t< T >::read(), ibis::bitvector::read(), ibis::bundle::readRIDs(), ibis::query::readRIDs(), ibis::part::reorder(), ibis::part::rollback(), ibis::column::searchSorted(), ibis::column::selectBytes(), ibis::column::selectDoubles(), ibis::column::selectFloats(), ibis::column::selectInts(), ibis::text::selectLongs(), ibis::column::selectLongs(), ibis::blob::selectRawBytes(), ibis::column::selectShorts(), ibis::column::selectUBytes(), ibis::column::selectUInts(), ibis::column::selectULongs(), ibis::column::selectUShorts(), ibis::column::selectValuesT(), ibis::column::truncateData(), and ibis::roster::writeSorted().

int ibis::fileManager::getFile ( const char *  name,
storage **  st,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Retrieve the file content as a storage object.

The object *st returned from this function is owned by the fileManager. The caller should NOT delete *st! This function will wait for the fileManager to unload some in-memory objects if there isn't enough memory for the file.

Upon successful completion of the task, it returns zero; otherwise, it returns a non-zero value to indicate an error and it does not modify the content of storage object.

References ibis::fileManager::storage::begin(), ibis::horometer::CPUTime(), ibis::fileManager::roFile::doRead(), ibis::fileManager::storage::enlarge(), ibis::util::groupby1000(), ibis::fileManager::roFile::isFileMap(), ibis::fileManager::roFile::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), ibis::horometer::stop(), and totalBytes.

ibis::fileManager::storage * ibis::fileManager::getFileSegment ( const char *  name,
const int  fdes,
const off_t  b,
const off_t  e 
)
static

Retrieve a portion of a file content.

Both the file name and the file descriptor are passed in to this function so that it can make a decision on whether to use a file map or directly read the content into memory. It prefers the read option more because the caller is more like to touch every bytes in an explicitly specified portion of a file. More specifically, it uses file map only if the file of the file segement is 4 * FASTBIT_MIN_MAP_SIZE and the number of mapped files is less than half of the maxOpenFiles.

References ibis::horometer::CPUTime(), ibis::util::groupby1000(), instance(), ibis::fileManager::storage::isFileMap(), ibis::fileManager::storage::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), and ibis::horometer::stop().

ibis::fileManager & ibis::fileManager::instance ( )
static

Returns a pointer to the one and only file manager.

The instance function of the fileManager singleton.

Referenced by ibis::column::actualMinMax(), ibis::relic::append(), ibis::bin::append(), ibis::direkte::append(), ibis::category::append(), ibis::column::append(), ibis::range::append(), ibis::mesa::append(), ibis::ambit::append(), ibis::pale::append(), ibis::pack::append(), ibis::zone::append(), ibis::part::append1(), ibis::part::append2(), ibis::part::appendToBackup(), ibis::bord::backup(), ibis::bin::binning(), ibis::bin::binningT(), ibis::bin::binOrderT(), ibis::bundle1::bundle1(), ibis::bundles::bundles(), bytesFree(), ibis::query::clear(), ibis::part::clear(), ibis::column::column(), ibis::part::commit(), ibis::column::computeMax(), ibis::column::computeMin(), ibis::column::computeSum(), ibis::relic::construct(), ibis::bin::construct(), ibis::range::construct(), ibis::ambit::construct(), ibis::egale::construct(), ibis::blob::countRawBytes(), ibis::index::create(), ibis::part::deactivate(), ibis::fileManager::roFile::disconnectFile(), ibis::part::doCompare(), ibis::fileManager::roFile::doRead(), ibis::part::emptyCache(), ibis::fileManager::roFile::endUse(), ibis::query::estimate(), ibis::query::evaluate(), fastbit_cleanup(), ibis::column::findLower(), ibis::text::findString(), ibis::column::findUpper(), ibis::part::get1DBins_(), ibis::blob::getBlob(), ibis::column::getDoubleArray(), getFileSegment(), ibis::column::getFloatArray(), ibis::column::getIntArray(), ibis::column::getNullMask(), ibis::column::getRawData(), ibis::category::getString(), ibis::roster::icSearch(), ibis::init(), ibis::util::intersect(), ibis::roster::locate(), ibis::index::mapValues(), ibis::bak2::mapValues(), ibis::part::negativeCompare(), ibis::part::barrel::open(), ibis::part::vault::open(), ibis::query::orderPairs(), ibis::part::part(), ibis::part::purgeInactive(), ibis::column::purgeIndexFile(), ibis::part::reactivate(), ibis::relic::read(), ibis::bin::read(), ibis::direkte::read(), ibis::bitvector64::read(), ibis::array_t< T >::read(), ibis::keywords::read(), ibis::bitvector::read(), ibis::fileManager::storage::read(), ibis::skive::read(), ibis::fade::read(), ibis::range::read(), ibis::fuzz::read(), ibis::bylt::read(), ibis::zona::read(), ibis::ambit::read(), ibis::pale::read(), ibis::pack::read(), ibis::zone::read(), ibis::fuge::read(), ibis::egale::read(), ibis::bundle::readRIDs(), ibis::query::readRIDs(), ibis::part::readRIDs(), ibis::text::readString(), ibis::text::readStrings1(), ibis::text::readStrings2(), ibis::direkte::remapKeys(), ibis::query::removeFiles(), ibis::part::reorder(), ibis::part::rollback(), ibis::text::saveSelected(), ibis::column::saveSelected(), ibis::column::searchSorted(), ibis::column::searchSortedOOCC(), ibis::column::searchSortedOOCD(), ibis::filter::select(), ibis::table::select(), ibis::column::selectBytes(), ibis::column::selectDoubles(), ibis::column::selectFloats(), ibis::column::selectInts(), ibis::text::selectLongs(), ibis::column::selectLongs(), ibis::blob::selectRawBytes(), ibis::column::selectShorts(), ibis::column::selectUBytes(), ibis::column::selectUInts(), ibis::column::selectULongs(), ibis::column::selectUShorts(), ibis::column::selectValuesT(), signalMemoryAvailable(), ibis::fileManager::storage::storage(), ibis::column::string2int(), ibis::text::stringSearch(), ibis::column::truncateData(), ibis::roster::write(), ibis::relic::write(), ibis::bin::write(), ibis::direkte::write(), ibis::keywords::write(), ibis::skive::write(), ibis::slice::write(), ibis::fade::write(), ibis::sbiad::write(), ibis::sapid::write(), ibis::range::write(), ibis::fuzz::write(), ibis::mesa::write(), ibis::bylt::write(), ibis::zona::write(), ibis::ambit::write(), ibis::pale::write(), ibis::pack::write(), ibis::zone::write(), ibis::fuge::write(), ibis::egale::write(), ibis::moins::write(), ibis::entre::write(), ibis::column::writeData(), ibis::tafel::writeMetaData(), ibis::roster::writeSorted(), and ibis::part::~part().

void ibis::fileManager::recordFile ( roFile st)
protected

Record a newly allocated storage in the two lists.

The caller needs to hold a mutex lock on the file manager to ensure correct operations.

References ibis::fileManager::storage::begin(), ibis::fileManager::storage::end(), ibis::fileManager::storage::filename(), and ibis::fileManager::storage::size().

void ibis::fileManager::recordPages ( off_t  start,
off_t  stop 
)
inline
void ibis::fileManager::removeCleaner ( const cleaner cl)

Unregister the cleaner functor.

Typically, this is only invoked when the object is being freed. For example, in the destructor of ibis::part, the cleaner associated with the partition is unregistered.

Referenced by ibis::part::clear(), and ibis::part::~part().

void ibis::fileManager::signalMemoryAvailable ( ) const

Signal to the file manager that some memory have been freed.

To be used by clients that are aware of the memory usages of in-memory objects since the in-memory objects based on ibis::fileManager::storage does not produce signals when they are freed.

References instance().

Referenced by ibis::relic::construct().

template<typename T >
int ibis::fileManager::tryGetFile ( const char *  name,
array_t< T > &  arr,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Given a file name, place the content in an array_t<T>.

This function will fail if there isn't enough memory to read the content of the file immediately.

The return value is zero (0) if the function is successful, otherwise returns a non-zero value.

References ibis::array_t< T >::clear(), ibis::array_t< T >::size(), and ibis::array_t< T >::swap().

Referenced by ibis::index::create(), and ibis::column::selectValuesT().

int ibis::fileManager::tryGetFile ( const char *  name,
storage **  st,
ACCESS_PREFERENCE  pref = MMAP_LARGE_FILES 
)

Try to retrieve the content of the named file.

The storage object *st returned from this function is owned by the fileManager. The caller is NOT to delete *st. This function will not wait for the fileManager to free any memory if there isn't enough free space available.

It returns 0 to indicate success and a negative value to indicate error. In particular, it returns -102 if there is not enough space to read the whole file into memory.

References ibis::fileManager::storage::begin(), ibis::horometer::CPUTime(), ibis::fileManager::roFile::doRead(), ibis::fileManager::storage::enlarge(), ibis::fileManager::roFile::isFileMap(), ibis::fileManager::roFile::printStatus(), ibis::horometer::realTime(), ibis::fileManager::storage::size(), ibis::horometer::start(), ibis::horometer::stop(), and totalBytes.

void ibis::fileManager::unrecordFile ( roFile st)
protected

Remove the file name from the list of files tracked.

This operation can only be performed on a file whose content has been read into memory, not on a file being mapped.

Note
The caller is expected to hold a mutex lock on the file manager.

References ibis::fileManager::storage::begin(), ibis::fileManager::storage::filename(), and ibis::fileManager::storage::size().

Referenced by ibis::fileManager::roFile::disconnectFile().


The documentation for this class was generated from the following files:

Make It A Bit Faster
Contact us
Disclaimers
FastBit source code
FastBit mailing list archive