How to contribute¶
The pipeline infrastructure at Kavli is a large and complex software undertaking, with numerous moving parts, stored in numerous code repositories.
Have you found a bug, or just something that could be improved? Contributions are welcome
The simplest option is to tell one of the maintainers (see below)
Better: open an issue in the relevent code repository
Best: Fork the repository, make your changes, and make a Pull Request to have your changes reviewed and merged back into the main code base.
Components of the pipeline¶
User interfaces¶
Documentation¶
The documentation is written in a mixture of MarkDown and ReStructured Text. New pages should be written in ReStructured Text, and where substantial modifications are made to MarkDown formatted pages, they should be converted.
Problems can be reported to Simon Ball, and fixes are welcomed as Pull Requests to the documentation repository.
Documentation is hosted on ReadTheDocs.
Web GUI¶
The web gui is written in Python using the Flask
microservices framework and served via nginx
. The code base and static and template objects are stored in the dj-GUI
repository
Imaging: Session Viewer¶
The session viewer is written in Python using the PyQT5 framework, and stored in the dj-imaging-user
repository. Originally written by Horst Obenhaus, now maintained by Simon Ball.
Ephys: Kavli Gui¶
The Kavli Gui is written in Python using the PyQT5 framework, and stored in the kavli-gui
repository. Written and maintained by Simon Ball
Backend code¶
Pipeline population logic¶
The pipelines are defined by a large amount of “business logic”: this defines what happens to populate each table, and any other assistant logic needed to support those population functions. The code is separated by pipeline:
The Ephys pipeline code is stored in the
dj-elphys
repository, and includes loader functions for reading each of the raw data file types currently in use. Written and maintained by Thinh Nguyen and Simon BallThe Imaging pipeline code is stored in the
dj-moser-imaging
repository. Originally written by Horst Obenhaus, and maintained by Horst Obenhaus, Thinh Nguyen and Simon Ball
Analysis code¶
Code to perform the various screening analyses (essentially, everything after combining position and spike data to produce SignalTracking) are stored in two separate repositories. Unlike most of the pipeline code, these libraries are open source.
Behavoural Neurology Toolbox: the original Matlab toolbox in use at Kavli. Originally written by Vadim Frolov, and now maintained by Simon Ball
Opexebo: the Python translation of much of the Matlab toolbox. Written and maintained by Simon Ball. Documentation is available on ReadTheDocs
Assistant tools and helpers¶
Various assistant tools have been written to help administer the pipeline. Where that code is pipeline-specific, it is stored with the population logic. Additionally, the dj-utils
repository stores assorted tools:
Database user management
Ephemeral server deployment configuration (written with Ansible)
Database server configuration details
Pipeline initialisation code.
External components¶
The pipelines also depend on several external components, over which the pipeline maintainers have reduced or no control:
mlims: Kavli institute colony management system
File storage: raw data is typically stored on the
forskning
redundant network storage drive.Object storage: intermediate processed data is stored on an s3-compatible storage system run by NTNU (
ceph
based)Openstack: Worker servers are virtual machines run in NTNU’s Openstack infrastructure. VMs are configured by the configuration code in
dj-utils
.