Datajoint Pipeline Architecture

The data pipelines in use in the Moser group require numerous computers to communicate and operate with each other. The components are mostly shared between the two pipelines, but with some key differences

Shared Components

  • Database server: a physical computer hosting a MySQL database server. This stores the tables that make up the pipeline

  • mLIMs: Animal colony management.

  • Network drive (\\forskning.it.ntnu.no\ntnu\mh-kin\moser\): The network drive where raw recordings are stored

  • Object Storage (ceph): An s3-compatible storage system for storing processed “blob” data for the database. Accessed only through the Datajoint/database interface.

  • dj-GUI: The joint web interface (source code repository) hosted by a virtual machine.

  • dj-docs: the pipeline documentation

  • Opexebo: Python-based analysis functions (source code repository)

  • BNT: Matlab-based analysis functions (source code repository)

Electrophysiology Pipeline

  • dj-elphys: the source code repository for the ephys pipeline.

  • dj-worker-python: Virtual machine running Python-based pipeline workers

  • dj-worker-matlab: Virtual machine running Matlab-based pipeline workers

../_images/datajoint_architecture.png

Imaging Pipeline

  • dj-moser-imaging: the source code repository for the imaging pipeline

  • Dirk: Intermediary storage server. File server hosted in Kavli Institute