Manage optogenetics experiments with DataJoint Elements¶
In this tutorial, we will walk through storing optogenetic stimulus data with the DataJoint Workflow for Optogenetics.
We will explain the following concepts as they relate to this workflow:
- What is an Element versus a Workflow?
- Plot the workflow with
dj.Diagram - Insert data into tables
- Query table contents
- Fetch table contents
- Run the workflow for your experiments
For detailed documentation and tutorials on general DataJoint principles that support collaboration, automation, reproducibility, and visualizations:
DataJoint Interactive Tutorials - Fundamentals including table tiers, query operations, fetch operations, automated computations with the
makefunction, etc.DataJoint Core - Documentation - Relational data model principles
Let's start by importing the packages necessary to run this workflow.
import datajoint as dj
The DataJoint Workflow for Optogenetics is assembled from 4 DataJoint Elements¶
| Element | Source Code | Documentation | Description |
|---|---|---|---|
| Element Lab | Link | Link | Lab management related information, such as Lab, User, Project, Protocol, Source. |
| Element Animal | Link | Link | General animal metadata and surgery information. |
| Element Session | Link | Link | General information of experimental sessions. |
| Element Optogenetics | Link | Link | Optogenetics stimulus and timing data. |
Each workflow is composed of multiple Elements. Each Element contains 1 or more modules, and each module declares its own schema in the database.
The Elements are imported within the workflow_optogenetics.pipeline script.
By importing the modules for the first time, the schemas and tables will be created in the database. Once created, importing modules will not create schemas and tables again, but the existing schemas/tables can be accessed.
The schema diagram (shown below) is a good reference for understanding the order of the tables within the workflow.
Let's activate the Elements.
from workflow_optogenetics.pipeline import lab, subject, surgery, session, optogenetics, Device
[2023-05-05 03:13:02,092][WARNING]: lab.Project and related tables will be removed in a future version of Element Lab. Please use the project schema. [2023-05-05 03:13:02,117][INFO]: Connecting root@fakeservices.datajoint.io:3306 [2023-05-05 03:13:02,146][INFO]: Connected root@fakeservices.datajoint.io:3306
Diagram¶
We can plot the diagram of tables within multiple schemas and their dependencies using dj.Diagram(). For details, see the documentation.
(
dj.Diagram(subject.Subject)
+ dj.Diagram(surgery.Implantation)
+ dj.Diagram(session.Session)
+ dj.Diagram(Device)
+ dj.Diagram(optogenetics)
)
While the diagram above seems complex at first, it becomes more clear when it's approached as a hierarchy of tables that define the order in which the workflow expects to receive data in each of its tables.
The tables higher up in the diagram such as subject.Subject() should be the first to receive data.
Data is manually entered into the green, rectangular tables with the insert1() method.
Tables connected by a solid line depend on entries from the table above it.
There are 5 table tiers in DataJoint. Some of these tables appear in the diagram above.
| Table tier | Color and shape | Description |
|---|---|---|
| Manual table | Green box | Data entered from outside the pipeline, either by hand or with external helper scripts. |
| Lookup table | Gray box | Small tables containing general facts and settings of the data pipeline; not specific to any experiment or dataset. |
| Imported table | Blue oval | Data ingested automatically inside the pipeline but requiring access to data outside the pipeline. |
| Computed table | Red circle | Data computed automatically entirely inside the pipeline. |
| Part table | Plain text | Part tables share the same tier as their master table. |
Insert entries into manual tables¶
In this section, we will insert metadata about an animal subject, experiment session, and optogenetic stimulation parameters.
Let's start with the first schema and table in the schema diagram (i.e. subject.Subject table).
Each module (e.g. subject) contains a schema object that enables interaction with the schema in the database.
subject.schema
Schema `neuro_subject`
The table classes in the module correspond to a table in the database.
subject.Subject()
| subject | subject_nickname | sex | subject_birth_date | subject_description |
|---|---|---|---|---|
Total: 0
We can view the table dependencies and the attributes we need to insert by using the functions .describe() and .heading. The describe() function displays the table definition with foreign key references and the heading function displays the attributes of the table definition. These are particularly useful functions if you are new to DataJoint Elements and are unsure of the attributes required for each table.
print(subject.Subject.describe())
'subject : varchar(8) \n---\nsubject_nickname="" : varchar(64) \nsex : enum(\'M\',\'F\',\'U\') \nsubject_birth_date : date \nsubject_description="" : varchar(1024) \n'
subject.Subject.heading
#
subject : varchar(8) #
---
subject_nickname="" : varchar(64) #
sex : enum('M','F','U') #
subject_birth_date : date #
subject_description="" : varchar(1024) #
We will insert data into the subject.Subject table.
subject.Subject.insert1(
dict(
subject="subject1",
sex="F",
subject_birth_date="2020-01-01",
subject_description="Optogenetic pilot subject",
)
)
subject.Subject()
| subject | subject_nickname | sex | subject_birth_date | subject_description |
|---|---|---|---|---|
| subject1 | F | 2020-01-01 | Optogenetic pilot subject |
Total: 1
Let's continue inserting in the other manual tables. The Session table is next.
print(session.Session.describe())
'-> subject.Subject\nsession_id : int \n---\nsession_datetime : datetime \n'
session.Session.heading
# subject : varchar(8) # session_id : int # --- session_datetime : datetime #
The cells above show the dependencies and attributes for the session.Session table.
Notice that describe shows the dependencies of the table on upstream tables (i.e. foreign key references). The Session table depends on the upstream Subject table.
Whereas heading lists all the attributes of the Session table, regardless of
whether they are declared in an upstream table.
session.Session.insert1(
dict(
subject="subject1",
session_id="1",
session_datetime="2022-04-04 12:22:15.032"
)
)
session.Session()
| subject | session_id | session_datetime |
|---|---|---|
| subject1 | 1 | 2022-04-04 12:22:15 |
Total: 1
The OptoProtocol table's attributes include the Session and Device tables. Let's insert into the Device table.
Device.insert1(
dict(
device="OPTG_8",
modality="Optogenetics",
description="8 channel pulse sequence device",
)
)
Device()
| device | modality | description |
|---|---|---|
| OPTG_4 | Optogenetics | Doric Pulse Sequence Generator |
| OPTG_8 | Optogenetics | 8 channel pulse sequence device |
Total: 2
The surgery.Implantation table's attribute includes the User table. Let's insert into the User table.
lab.User.insert1(
dict(user="User1")
)
lab.User()
| user username, short identifier | user_email | user_cellphone | user_fullname Full name used to uniquely identify an individual |
|---|---|---|---|
| User1 |
Total: 1
The Implantation table's attributes includes the CoordinateReference and Hemisphere tables. Let's view the contents of these lookup tables, which have default contents.
surgery.CoordinateReference()
| reference |
|---|
| bregma |
| dura |
| lambda |
| sagittal_suture |
| sinus |
| skull_surface |
Total: 6
surgery.Hemisphere()
| hemisphere Brain region hemisphere |
|---|
| left |
| middle |
| right |
Total: 3
Insert a new entry for the location of the optogenetics probe.
surgery.BrainRegion.insert1(
dict(
region_acronym="dHP",
region_name="Dorsal Hippocampus")
)
surgery.BrainRegion()
| region_acronym Brain region shorthand | region_name Brain region full name |
|---|---|
| dHP | Dorsal Hippocampus |
Total: 1
surgery.Implantation.insert1(
dict(
subject="subject1",
implant_date="2022-04-01 12:13:14",
implant_type="opto",
target_region="dHP",
target_hemisphere="left",
surgeon="User1",
)
)
surgery.Implantation.Coordinate.insert1(
dict(
subject="subject1",
implant_date="2022-04-01 12:13:14",
implant_type="opto",
target_region="dHP",
target_hemisphere="left",
ap="-7.9", # [mm] anterior-posterior distance
ap_ref="bregma",
ml="-1.8", # [mm] medial axis distance
ml_ref="bregma",
dv="5", # [mm] dorso-ventral axis distance
dv_ref="skull_surface",
theta="11.5", # [0, 180] degree rotation about ml-axis relative to z
phi="0", # [0, 360] degree rotation about dv-axis relative to x
beta=None, # [-180, 180] degree rotation about shank relative to anterior
)
)
surgery.Implantation.Coordinate()
| subject | implant_date surgery date | implant_type Short name for type of implanted device | target_region Brain region shorthand | target_hemisphere Brain region hemisphere | ap (mm) anterior-posterior; ref is 0 | ap_ref | ml (mm) medial axis; ref is 0 | ml_ref | dv (mm) dorso-ventral axis; ventral negative | dv_ref | theta (deg) rot about ml-axis [0, 180] wrt z | phi (deg) rot about dv-axis [0, 360] wrt x | beta (deg) rot about shank [-180, 180] wrt anterior |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| subject1 | 2022-04-01 12:13:14 | opto | dHP | left | -7.9 | bregma | -1.8 | bregma | 5.0 | skull_surface | 11.5 | 0.0 | nan |
Total: 1
We'll add information to describe the stimulus, including waveform shape and stimulation parameters.
optogenetics.OptoWaveform.insert1(
dict(
waveform_name="square_10",
waveform_type="square",
waveform_description="Square waveform: 10%/90% on/off cycle",
)
)
# Square is one part table of OptoWaveform.
# For sine and ramp waveforms, see the corresponding tables.
optogenetics.OptoWaveform.Square.insert1(
dict(
waveform_name="square_10",
on_proportion=0.10,
off_proportion=0.90)
)
optogenetics.OptoWaveform.Square()
| waveform_name | on_proportion Proportion of stimulus on time within a cycle | off_proportion Proportion of stimulus off time within a cycle |
|---|---|---|
| square_10 | 0.10 | 0.90 |
Total: 1
optogenetics.OptoStimParams.insert1(
dict(
opto_params_id=1,
waveform_name="square_10",
wavelength=470,
light_intensity=10.2,
frequency=1,
duration=241,
)
)
optogenetics.OptoStimParams()
| opto_params_id | waveform_name | wavelength (nm) wavelength of optical stimulation light | power (mW) total power from light source | light_intensity (mW/mm2) power for given area | frequency (Hz) frequency of the waveform | duration (ms) duration of each optical stimulus |
|---|---|---|---|---|---|---|
| 1 | square_10 | 470 | None | 10.20 | 1.0 | 241.0 |
Total: 1
Next, we'll describe the session in which these parameters are used in OptoProtocol.
optogenetics.OptoProtocol.insert1(
dict(
subject="subject1",
session_id="1",
protocol_id="1",
opto_params_id="1",
implant_date="2022-04-01 12:13:14",
implant_type="opto",
target_region="dHP",
target_hemisphere="left",
device="OPTG_4",
)
)
optogenetics.OptoProtocol()
| subject | session_id | protocol_id | opto_params_id | implant_date surgery date | implant_type Short name for type of implanted device | target_region Brain region shorthand | target_hemisphere Brain region hemisphere | device | protocol_description description of optogenetics protocol |
|---|---|---|---|---|---|---|---|---|---|
| subject1 | 1 | 1 | 1 | 2022-04-01 12:13:14 | opto | dHP | left | OPTG_4 |
Total: 1
We can describe the timing of these stimulations in OptoEvent.
optogenetics.OptoEvent.insert1(
dict(
subject="subject1",
session_id=1,
protocol_id=1,
stim_start_time=241,
stim_end_time=482,
)
)
optogenetics.OptoEvent()
| subject | session_id | protocol_id | stim_start_time (s) stimulus start time relative to session start | stim_end_time (s) stimulus end time relative session start |
|---|---|---|---|---|
| subject1 | 1 | 1 | 241.0 | 482.0 |
Total: 1
We can insert a second set of timing information for the stimulation.
optogenetics.OptoEvent.insert1(
dict(
subject="subject1",
session_id=1,
protocol_id=1,
stim_start_time=543,
stim_end_time=797,
)
)
optogenetics.OptoEvent()
| subject | session_id | protocol_id | stim_start_time (s) stimulus start time relative to session start | stim_end_time (s) stimulus end time relative session start |
|---|---|---|---|---|
| subject1 | 1 | 1 | 241.0 | 482.0 |
| subject1 | 1 | 1 | 543.0 | 797.0 |
Total: 2
Query¶
Queries allow you to view the contents of the database. The simplest query is the instance of the table class.
optogenetics.OptoEvent()
| subject | session_id | protocol_id | stim_start_time (s) stimulus start time relative to session start | stim_end_time (s) stimulus end time relative session start |
|---|---|---|---|---|
| subject1 | 1 | 1 | 241.0 | 482.0 |
| subject1 | 1 | 1 | 543.0 | 797.0 |
Total: 2
With the & operator, we will restrict the contents of the OptoEvent table to those entries with a stim_start_time of 543.
optogenetics.OptoEvent & "stim_start_time=543"
| subject | session_id | protocol_id | stim_start_time (s) stimulus start time relative to session start | stim_end_time (s) stimulus end time relative session start |
|---|---|---|---|---|
| subject1 | 1 | 1 | 543.0 | 797.0 |
Total: 1
DataJoint queries can be a highly flexible tool with several operators. The next operator we will explore is join which combines matching information from tables.
optogenetics.OptoProtocol * optogenetics.OptoStimParams
| subject | session_id | protocol_id | opto_params_id | implant_date surgery date | implant_type Short name for type of implanted device | target_region Brain region shorthand | target_hemisphere Brain region hemisphere | device | protocol_description description of optogenetics protocol | waveform_name | wavelength (nm) wavelength of optical stimulation light | power (mW) total power from light source | light_intensity (mW/mm2) power for given area | frequency (Hz) frequency of the waveform | duration (ms) duration of each optical stimulus |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| subject1 | 1 | 1 | 1 | 2022-04-01 12:13:14 | opto | dHP | left | OPTG_4 | square_10 | 470 | None | 10.20 | 1.0 | 241.0 |
Total: 1
Fetch¶
The fetch and fetch1 methods download the data from the query object into the workspace.
Below we will run fetch() without any arguments to return all attributes of all entries in the table.
optogenetics.OptoEvent.fetch(as_dict=True)
[{'subject': 'subject1',
'session_id': 1,
'protocol_id': 1,
'stim_start_time': 241.0,
'stim_end_time': 482.0},
{'subject': 'subject1',
'session_id': 1,
'protocol_id': 1,
'stim_start_time': 543.0,
'stim_end_time': 797.0}]
Next, we will fetch the entry with a stim_start_time of 543 with the fetch1 method, which returns a dictionary containing all attributes of one entry in the table.
(optogenetics.OptoEvent & "stim_start_time=543").fetch1()
{'subject': 'subject1',
'session_id': 1,
'protocol_id': 1,
'stim_start_time': 543.0,
'stim_end_time': 797.0}
Next steps¶
Follow the steps below to run this workflow for your experiments:
- Create a fork of this repository to your GitHub account.
- Clone the repository to your local machine and configure for use with the instructions in the User Guide.
- The DataJoint team offers free Office Hours to help you setup this workflow.
- If you have any questions, please reach out at support@datajoint.com.