wiki:Dev/Technical/ZEO

Version 55 (modified by dgalimbe, 3 years ago) (diff)

--

  1. ZEO
    1. 1. Introduction
    2. 1.1 Caches
    3. 2. Connection
      1. 2.1 Picklecache
      2. 2.2 Picklecache's strategy
      3. 2.3 How to change the Picklecache cache size
    4. 3. ZEO client
      1. 3.1 ZEO Client cache
      2. 3.2 ZEO Client cache's strategy
      3. 3.3 How to change the ZEO Client cache size
      4. 3.3 How to choose the correct cache size
    5. 4. References

ZEO

1. Introduction

ZODB does not support concurrencies processes. Only one application can open the file storage at a time. ZEO is a client-server system for sharing a single storage among many clients. Normally, a ZODB storage can only be used by a single process. When you use ZEO, the storage is opened in the ZEO server process. Client programs connect to this process using a ZEO Client Storage.

http://www.gliffy.com/pubdoc/2060766/M.png

In principle each request made to the Apache Server creates a new process (and therefore all global variables, class variables and singleton are set again), and then each thread of this process owns its own connection.

1.1 Caches

There are always two caches in play when using ZEO.

  • A Connection object has "a pickle cache", which is an in-memory cache. By default, it holds 400 objects across transaction boundaries, regardless of how much RAM they consume.
  • A second-level cache (the ZEO client cache, see §3.1), seeing only requests not satisfied by the pickle cache. It's a disk-based cache, and defaults to a maximum size of 20MB

2. Connection

2.1 Picklecache

Each DB Connection has a separate Picklecache. The Cache serves two purposes. It acts like a memo for unpickling. It also keeps recent objects in memory under the assumption that they may be used again. As already mentioned in §1.1, the size of the cache is the number of objects that can be cached, by default this number is 400.

2.2 Picklecache's strategy

The least-recently used objects are booted out at transaction boundaries.

2.3 How to change the Picklecache cache size

The Picklecache's cache size can be changed through the initialization of the MaKaCDB object, and this can be done as follows (assuming one wants to set it to 800 objects): 

class DBMgr:
    def __init__( self, hostname=None, port=None ):
        ...
        self._db=MaKaCDB(self._storage, cache_size = 800)
        ....
    ...

3. ZEO client

Zeo Clients create a connection with the Zeo Server each time they need to read or write in the ZODB.

Indico uses ZOE and ZODB without the ZOPE support, therefore the configuration for the ZEO-Client is directly made inside the code, at its initialization. More precisely in the DBMgr class when creating the Client Storage class. The Client Storage class accepts these configuration values (the complete list and description can be found at: ​http://svn.zope.org/ZODB/trunk/doc/zeo.txt?rev=82954&view=markup)

  • cache-size: the maximun size of the client cache, in bytes
  • client: enables persistent cache files. If it is not specified, the client creates a temprary cache that will only be used by the current by the current object (in our case for the current process)
  • var: the directory where persistent cache file are stored. By default, if they are persistent, the cache files are stored in the current directoy

3.1 ZEO Client cache

Each ZEO client keeps an on-disk cache because fetching an object over the network can be much slower than fetching the object from a local File Storage. The client cache can be:

  • Persistent: the cache file is retained for user after process restarts (in order to define a cache as persistent one has to define the configuration parameter 'client' in the Client Storage initialization (see §3).
  • Transient: the cache uses temporary files that are removed when the client storage is closed

3.2 ZEO Client cache's strategy

The ZEO Client cache uses a FIFO strategy. In others words there is a pointer inside the cache that goes around the file, circularly, forever.

3.3 How to change the ZEO Client cache size

The default value of the zeo client cache size is 20MB. If one wants to modify this value, one has to add a 'cache-size' parameter in the construction of the DBMgr's storage field. For examble if one wants to set 50Mb (5242880 bytes) of cache size, the code will be: 

class DBMgr:
    def __init__( self, hostname=None, port=None ):
        ...
        self._storage=ClientStorage((hostname, port), username=cfg.getDBUserName(), password=cfg.getDBPassword(),
                      realm=cfg.getDBRealm(), cache_size=5242880)
        ....
    ...

3.3 How to choose the correct cache size

Follow these instructions: ​http://svn.zope.org/ZODB/trunk/doc/zeo-client-cache-tracing.txt

4. References