Monday, December 6, 2010

Design Assumptions of Google File System:

  • components that often fail. It must constantly monitor
    itself and detect, tolerate, and recover promptly from
    component failures on a routine basis.
    expect a few million files, each typically 100 MB or
    larger in size. Multi-GB files are the common case
    and should be managed efficiently. Small files must be
    supported, but we need not optimize for them.
    large streaming reads and small random reads. In
    large streaming reads, individual operations typically
    read hundreds of KBs, more commonly 1 MB or more.
    Successive operations from the same client often read
    through a contiguous region of a file. A small random
    read typically reads a few KBs at some arbitrary
    offset. Performance-conscious applications often batch
    and sort their small reads to advance steadily through
    the file rather than go backan d forth.
    that append data to files. Typical operation sizes are
    similar to those for reads. Once written, files are seldom
    modified again. Small writes at arbitrary positions
    in a file are supported but do not have to be
    efficient.
    for multiple clients that concurrently append
    to the same file. Our files are often used as producerconsumer
    queues or for many-way merging. Hundreds
    of producers, running one per machine, will concurrently
    append to a file. Atomicity with minimal synchronization
    overhead is essential. The file may be
    read later, or a consumer may be reading through the
    file simultaneously.
    latency. Most of our target applications place a premium
    on processing data in bulka t a high rate, while
    few have stringent response time requirements for an
    individual read or write.
    High sustained bandwidth is more important than low
    The system must efficiently implement well-defined semantics
    The workloads also have many large, sequential writes
    The workloads primarily consist of two kinds of reads:
    The system stores a modest number of large files. We
    The system is built from many inexpensive commodity

No comments:

Post a Comment