{ "cells": [ { "cell_type": "markdown", "id": "6838deb9-c543-4732-a4ed-3cf4046a54d3", "metadata": {}, "source": [ "# Dataset Definition DataFrame\n", "\n", "The purpose is to show:\n", "- how a custom dataframe can be used to define a dataset for processing\n", "- that a processing config can be generated based on that dataframe.\n", " - Specically, the stations level of the processing config, which contains information about which stations and runs are available.\n", "- that we can re-use a run with different start and end times (provided that these data are available)\n", " - i.e. say there is a long run at some station, we can use an early chunk and a later chunk from the same run, and omit some intermediate time interal from processing simply by creating one row of the dataset dataframe per run-chunk to process.\n", " \n", "\n", "There are examples of using ConfigCreator to generate an entire processing config in `operate_aurora.ipynb`." ] }, { "cell_type": "markdown", "id": "e73ab089-fc34-4d5a-a88c-4e43d5874ab7", "metadata": {}, "source": [ "A user can pass to the processing config a dataframe with information about which runs to process.\n", "\n", "Here is a simple example of how to do that." ] }, { "cell_type": "code", "execution_count": 1, "id": "af5fa6bb-7f64-47fb-a6a7-8cd6d116546f", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "from mt_metadata.transfer_functions.processing.aurora import Processing" ] }, { "cell_type": "markdown", "id": "edea2ce4-9174-4903-809f-6be4c8674709", "metadata": {}, "source": [ "An example of creating a dataframe from scratch with the required columns to pass to `Processing`.\n", "\n", "Here we consider three stations, `mt01`, `rr01`, `rr02`, each having three runs, labelled `000`, `001`, `002`.\n", "Note that we do not need to specify the information about the actual start and end times of the runs that were acquired in the field, the start and end times specified here correspond to time intervals to _process_. The information about the actual start and end times of the field data acquisition are stored elsewhere, in an MTH5 archive." ] }, { "cell_type": "code", "execution_count": 2, "id": "03cab6f9-6d03-4483-9199-663d3fda922c", "metadata": {}, "outputs": [], "source": [ "starts = [\"2020-01-01T00:00:00\", \"2020-02-02T00:00:00\"]\n", "ends = [\"2020-01-31T12:00:00\", \"2020-02-28T12:00:00\"]\n", "\n", "data_list = []\n", "\n", "for i_run in range(3):\n", " run_id = f\"{i_run}\".zfill(3) # note that the run_id could be different for the different stations\n", " for start, end in zip(starts, ends):\n", " entry = {\n", " \"station\": \"mt01\",\n", " \"run\": run_id,\n", " \"start\": start,\n", " \"end\": end,\n", " \"mth5_path\": r\"/home/mth5_path.h5\" ,\n", " \"sample_rate\": 10,\n", " \"input_channels\": [\"hx\", \"hy\"],\n", " \"output_channels\": [\"hz\", \"ex\", \"ey\"],\n", " \"remote\": False\n", " }\n", " \n", " data_list.append(entry)\n", " \n", " rr_entry_01 = {\n", " \"station\": \"rr01\",\n", " \"run\": run_id,\n", " \"start\": start,\n", " \"end\": end,\n", " \"mth5_path\": r\"/home/mth5_path.h5\" ,\n", " \"sample_rate\": 10,\n", " \"input_channels\": [\"hx\", \"hy\"],\n", " \"output_channels\": [\"hz\", \"ex\", \"ey\"],\n", " \"remote\": True\n", " }\n", " data_list.append(rr_entry_01)\n", " \n", " rr_entry_02 = {\n", " \"station\": \"rr02\",\n", " \"run\": run_id,\n", " \"start\": start,\n", " \"end\": end,\n", " \"mth5_path\": r\"/home/mth5_path.h5\" ,\n", " \"sample_rate\": 10,\n", " \"input_channels\": [\"hx\", \"hy\"],\n", " \"output_channels\": [\"hz\", \"ex\", \"ey\"],\n", " \"remote\": True\n", " }\n", " data_list.append(rr_entry_02)\n", "\n", "\n", "dataset_df = pd.DataFrame(data_list)\n", "dataset_df.start = pd.to_datetime(dataset_df.start, utc=True)\n", "dataset_df.end = pd.to_datetime(dataset_df.end, utc=True)" ] }, { "cell_type": "markdown", "id": "cc1074bb-8e37-4a10-a5b4-612ab5b81d7a", "metadata": {}, "source": [ "Here is the `dataset_dataframe`. If this is passed to the Processing object `p`, then `p` will know how to create station metadata. Note that we can specify more than one remote. The ability to work with multiple remotes is something that maybe implemented in future, for now only the first remote station will be used in the processing." ] }, { "cell_type": "code", "execution_count": 3, "id": "e18a4300-6aaa-4345-87b2-31a133c0a544", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stationrunstartendmth5_pathsample_rateinput_channelsoutput_channelsremote
0mt010002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]False
1rr010002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
2rr020002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
3mt010002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]False
4rr010002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
5rr020002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
6mt010012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]False
7rr010012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
8rr020012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
9mt010012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]False
10rr010012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
11rr020012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
12mt010022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]False
13rr010022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
14rr020022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
15mt010022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]False
16rr010022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
17rr020022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510[hx, hy][hz, ex, ey]True
\n", "
" ], "text/plain": [ " station run start end \\\n", "0 mt01 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "1 rr01 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "2 rr02 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "3 mt01 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "4 rr01 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "5 rr02 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "6 mt01 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "7 rr01 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "8 rr02 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "9 mt01 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "10 rr01 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "11 rr02 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "12 mt01 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "13 rr01 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "14 rr02 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "15 mt01 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "16 rr01 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "17 rr02 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "\n", " mth5_path sample_rate input_channels output_channels remote \n", "0 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] False \n", "1 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "2 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "3 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] False \n", "4 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "5 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "6 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] False \n", "7 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "8 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "9 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] False \n", "10 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "11 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "12 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] False \n", "13 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "14 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "15 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] False \n", "16 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True \n", "17 /home/mth5_path.h5 10 [hx, hy] [hz, ex, ey] True " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset_df" ] }, { "cell_type": "markdown", "id": "4c86ab9c-a2de-4ef0-910f-f07b1912a329", "metadata": {}, "source": [ "Initialize an empty processing object" ] }, { "cell_type": "code", "execution_count": 4, "id": "e8a20066-a343-4345-829b-08a5490dd6c4", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{\n", " \"processing\": {\n", " \"channel_nomenclature.ex\": \"ex\",\n", " \"channel_nomenclature.ey\": \"ey\",\n", " \"channel_nomenclature.hx\": \"hx\",\n", " \"channel_nomenclature.hy\": \"hy\",\n", " \"channel_nomenclature.hz\": \"hz\",\n", " \"decimations\": [],\n", " \"id\": null,\n", " \"stations.local.id\": null,\n", " \"stations.local.mth5_path\": null,\n", " \"stations.local.remote\": false,\n", " \"stations.local.runs\": [],\n", " \"stations.remote\": []\n", " }\n", "}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "p = Processing()\n", "p" ] }, { "cell_type": "markdown", "id": "09bc82ae-e367-4f1c-9dd6-2444ea3e15eb", "metadata": {}, "source": [ "### Create the `Stations` container" ] }, { "cell_type": "code", "execution_count": 5, "id": "9342098a-2391-4fa9-a02d-dfaeec3750cf", "metadata": {}, "outputs": [], "source": [ "p.stations.from_dataset_dataframe(dataset_df)" ] }, { "cell_type": "markdown", "id": "d7ed8513-0dd3-47df-a580-59d0a468920f", "metadata": {}, "source": [ "Now `p` has all the station and run information." ] }, { "cell_type": "code", "execution_count": 6, "id": "b1e46a2b-05cd-441b-a265-1387fbae4420", "metadata": { "tags": [] }, "outputs": [ { "data": { "text/plain": [ "{\n", " \"processing\": {\n", " \"channel_nomenclature.ex\": \"ex\",\n", " \"channel_nomenclature.ey\": \"ey\",\n", " \"channel_nomenclature.hx\": \"hx\",\n", " \"channel_nomenclature.hy\": \"hy\",\n", " \"channel_nomenclature.hz\": \"hz\",\n", " \"decimations\": [],\n", " \"id\": null,\n", " \"stations.local.id\": \"mt01\",\n", " \"stations.local.mth5_path\": \"/home/mth5_path.h5\",\n", " \"stations.local.remote\": false,\n", " \"stations.local.runs\": [\n", " {\n", " \"run\": {\n", " \"id\": \"000\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"run\": {\n", " \"id\": \"001\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"run\": {\n", " \"id\": \"002\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " }\n", " ],\n", " \"stations.remote\": [\n", " {\n", " \"station\": {\n", " \"id\": \"rr01\",\n", " \"mth5_path\": \"/home/mth5_path.h5\",\n", " \"remote\": true,\n", " \"runs\": [\n", " {\n", " \"run\": {\n", " \"id\": \"000\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"run\": {\n", " \"id\": \"001\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"run\": {\n", " \"id\": \"002\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"station\": {\n", " \"id\": \"rr02\",\n", " \"mth5_path\": \"/home/mth5_path.h5\",\n", " \"remote\": true,\n", " \"runs\": [\n", " {\n", " \"run\": {\n", " \"id\": \"000\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"run\": {\n", " \"id\": \"001\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " },\n", " {\n", " \"run\": {\n", " \"id\": \"002\",\n", " \"input_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hx\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"hy\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"output_channels\": [\n", " {\n", " \"channel\": {\n", " \"id\": \"hz\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ex\",\n", " \"scale_factor\": 1.0\n", " }\n", " },\n", " {\n", " \"channel\": {\n", " \"id\": \"ey\",\n", " \"scale_factor\": 1.0\n", " }\n", " }\n", " ],\n", " \"sample_rate\": 10.0,\n", " \"time_periods\": [\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-01-31T12:00:00+00:00\",\n", " \"start\": \"2020-01-01T00:00:00+00:00\"\n", " }\n", " },\n", " {\n", " \"time_period\": {\n", " \"end\": \"2020-02-28T12:00:00+00:00\",\n", " \"start\": \"2020-02-02T00:00:00+00:00\"\n", " }\n", " }\n", " ]\n", " }\n", " }\n", " ]\n", " }\n", " }\n", " ]\n", " }\n", "}" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "p" ] }, { "cell_type": "markdown", "id": "b59450a9-489a-454d-a3fc-9eefc05cfa01", "metadata": {}, "source": [ "We can recover the dataframe from `p` by asking it for a `dataset_dataframe`" ] }, { "cell_type": "code", "execution_count": 7, "id": "c11a1e09-b0d1-45a5-95a2-11f4ec7bd382", "metadata": { "tags": [] }, "outputs": [], "source": [ "df2 = p.stations.to_dataset_dataframe()" ] }, { "cell_type": "markdown", "id": "3ec17b4e-c79a-4663-8026-4182bde5a710", "metadata": {}, "source": [ "The new dataframe `df2` contains the same information as the original, but is not sorted exactly the same" ] }, { "cell_type": "code", "execution_count": 8, "id": "cc8bb293-0861-4483-ac51-806a06b3c9cb", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stationrunstartendmth5_pathsample_rateinput_channelsoutput_channelsremotechannel_scale_factors
0mt010002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
1mt010002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
2mt010012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
3mt010012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
4mt010022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
5mt010022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
6rr010002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
7rr010002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
8rr010012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
9rr010012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
10rr010022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
11rr010022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
12rr020002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
13rr020002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
14rr020012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
15rr020012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
16rr020022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
17rr020022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
\n", "
" ], "text/plain": [ " station run start end \\\n", "0 mt01 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "1 mt01 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "2 mt01 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "3 mt01 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "4 mt01 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "5 mt01 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "6 rr01 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "7 rr01 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "8 rr01 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "9 rr01 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "10 rr01 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "11 rr01 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "12 rr02 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "13 rr02 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "14 rr02 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "15 rr02 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "16 rr02 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "17 rr02 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "\n", " mth5_path sample_rate input_channels output_channels remote \\\n", "0 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "1 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "2 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "3 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "4 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "5 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "6 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "7 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "8 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "9 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "10 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "11 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "12 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "13 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "14 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "15 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "16 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "17 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "\n", " channel_scale_factors \n", "0 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "1 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "2 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "3 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "4 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "5 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "6 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "7 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "8 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "9 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "10 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "11 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "12 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "13 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "14 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "15 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "16 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "17 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df2" ] }, { "cell_type": "markdown", "id": "b67cab05-d5dc-4bcb-b4a8-bc44fe3d3b88", "metadata": {}, "source": [ "To recover the original dataframe, (in this specific example) sort by run, start and station" ] }, { "cell_type": "code", "execution_count": 9, "id": "536bdbd7-9b05-4506-a0df-026f51d1c839", "metadata": {}, "outputs": [], "source": [ "df2.sort_values(by=[\"run\", \"start\", \"station\"], inplace=True)" ] }, { "cell_type": "code", "execution_count": 10, "id": "5c9af394-0e7c-4e03-8f5f-f395fa6cbabd", "metadata": { "tags": [] }, "outputs": [], "source": [ "df2.reset_index(drop=True, inplace=True)" ] }, { "cell_type": "code", "execution_count": 11, "id": "b10794da-7cba-40ce-8a82-63c1b01eb826", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
stationrunstartendmth5_pathsample_rateinput_channelsoutput_channelsremotechannel_scale_factors
0mt010002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
1rr010002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
2rr020002020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
3mt010002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
4rr010002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
5rr020002020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
6mt010012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
7rr010012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
8rr020012020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
9mt010012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
10rr010012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
11rr020012020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
12mt010022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
13rr010022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
14rr020022020-01-01 00:00:00+00:002020-01-31 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
15mt010022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]False{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
16rr010022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
17rr020022020-02-02 00:00:00+00:002020-02-28 12:00:00+00:00/home/mth5_path.h510.0[hx, hy][hz, ex, ey]True{'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '...
\n", "
" ], "text/plain": [ " station run start end \\\n", "0 mt01 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "1 rr01 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "2 rr02 000 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "3 mt01 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "4 rr01 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "5 rr02 000 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "6 mt01 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "7 rr01 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "8 rr02 001 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "9 mt01 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "10 rr01 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "11 rr02 001 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "12 mt01 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "13 rr01 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "14 rr02 002 2020-01-01 00:00:00+00:00 2020-01-31 12:00:00+00:00 \n", "15 mt01 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "16 rr01 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "17 rr02 002 2020-02-02 00:00:00+00:00 2020-02-28 12:00:00+00:00 \n", "\n", " mth5_path sample_rate input_channels output_channels remote \\\n", "0 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "1 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "2 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "3 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "4 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "5 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "6 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "7 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "8 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "9 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "10 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "11 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "12 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "13 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "14 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "15 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] False \n", "16 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "17 /home/mth5_path.h5 10.0 [hx, hy] [hz, ex, ey] True \n", "\n", " channel_scale_factors \n", "0 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "1 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "2 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "3 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "4 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "5 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "6 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "7 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "8 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "9 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "10 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "11 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "12 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "13 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "14 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "15 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "16 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... \n", "17 {'hx': 1.0, 'hy': 1.0, 'hz': 1.0, 'ex': 1.0, '... " ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df2" ] }, { "cell_type": "markdown", "id": "d811ab1c-65a5-4cb7-96ce-ba4db3714ab5", "metadata": {}, "source": [ "We can see that the dataframes are equal:" ] }, { "cell_type": "code", "execution_count": 12, "id": "a34966f5-440c-47cf-91d5-7907b865dcd5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "(df2[dataset_df.columns]==dataset_df).all().all()" ] }, { "cell_type": "code", "execution_count": null, "id": "696908f4-18c5-42d3-b69d-aa834eff5241", "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "id": "ff6780ab-ac40-43ae-85c1-8c6e7be2fa88", "metadata": {}, "source": [ "# Excercise:\n", "\n", "Below is an example of getting a channel_summary dataframe from an mth5, which can be used to inform the choice of a dataset dataframe.\n", "\n", "Run the cells below and then create a dataset definition from the channel summary that will process two chunks of data, 3 days at the start of the second run, and 4 days at then end of the last run" ] }, { "cell_type": "code", "execution_count": 13, "id": "96466a08-f11e-4005-83b1-aab5fda88f3d", "metadata": {}, "outputs": [], "source": [ "from mth5.mth5 import MTH5\n", "from mt_metadata import MT_EXPERIMENT_MULTIPLE_RUNS\n", "from mt_metadata.timeseries import Experiment" ] }, { "cell_type": "code", "execution_count": 14, "id": "6c189aad-4e9f-47a4-ab2f-0a2eb33bc4a1", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "PosixPath('/home/kkappler/software/irismt/mt_metadata/mt_metadata/data/mt_xml/multi_run_experiment.xml')" ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "MT_EXPERIMENT_MULTIPLE_RUNS" ] }, { "cell_type": "code", "execution_count": 15, "id": "469594bd-528e-4e41-a6c4-4cc3be560ff5", "metadata": { "tags": [] }, "outputs": [], "source": [ "experiment = Experiment()\n", "experiment.from_xml(MT_EXPERIMENT_MULTIPLE_RUNS)" ] }, { "cell_type": "code", "execution_count": 16, "id": "2ba8e29f-ac0f-4c7b-bca7-06472de2adb7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33m\u001b[1m2024-08-28T15:52:24.361188-0700 | WARNING | mth5.mth5 | open_mth5 | test_dataset_definition.h5 will be overwritten in 'w' mode\u001b[0m\n", "\u001b[1m2024-08-28T15:52:24.913025-0700 | INFO | mth5.mth5 | _initialize_file | Initialized MTH5 0.2.0 file test_dataset_definition.h5 in mode w\u001b[0m\n" ] }, { "data": { "text/plain": [ "/:\n", "====================\n", " |- Group: Experiment\n", " --------------------\n", " |- Group: Reports\n", " -----------------\n", " |- Group: Standards\n", " -------------------\n", " --> Dataset: summary\n", " ......................\n", " |- Group: Surveys\n", " -----------------\n", " --> Dataset: channel_summary\n", " ..............................\n", " --> Dataset: tf_summary\n", " ........................." ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m = MTH5()\n", "m.open_mth5(\"test_dataset_definition.h5\", \"w\")" ] }, { "cell_type": "code", "execution_count": 17, "id": "27e3d16a-298e-4905-b322-69d4d1ca1270", "metadata": { "tags": [] }, "outputs": [], "source": [ "m.from_experiment(experiment)" ] }, { "cell_type": "code", "execution_count": 18, "id": "dcb654f6-def8-4ebe-880d-f8f4a737bb81", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
surveystationrunlatitudelongitudeelevationcomponentstartendn_samplessample_ratemeasurement_typeazimuthtiltunitshas_datahdf5_referencerun_hdf5_referencestation_hdf5_reference
0CONUS SouthUTS14a37.563198-113.3016632490.775ex2020-07-05 23:19:41+00:002020-07-06 00:11:55+00:0031341.0electric11.1933620.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
1CONUS SouthUTS14a37.563198-113.3016632490.775ey2020-07-05 23:19:41+00:002020-07-06 00:11:55+00:0031341.0electric101.1933620.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
2CONUS SouthUTS14a37.563198-113.3016632490.775hx2020-07-05 23:19:41+00:002020-07-06 00:11:55+00:0031341.0magnetic11.1933620.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
3CONUS SouthUTS14a37.563198-113.3016632490.775hy2020-07-05 23:19:41+00:002020-07-06 00:11:55+00:0031341.0magnetic101.1933620.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
4CONUS SouthUTS14a37.563198-113.3016632490.775hz2020-07-05 23:19:41+00:002020-07-06 00:11:55+00:0031341.0magnetic0.00000090.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
5CONUS SouthUTS14b37.563198-113.3016632490.775ex2020-07-06 00:32:41+00:002020-07-20 17:43:45+00:0012714641.0electric11.1933680.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
6CONUS SouthUTS14b37.563198-113.3016632490.775ey2020-07-06 00:32:41+00:002020-07-20 17:43:45+00:0012714641.0electric101.1933680.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
7CONUS SouthUTS14b37.563198-113.3016632490.775hx2020-07-06 00:32:41+00:002020-07-20 17:43:45+00:0012714641.0magnetic11.1933680.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
8CONUS SouthUTS14b37.563198-113.3016632490.775hy2020-07-06 00:32:41+00:002020-07-20 17:43:45+00:0012714641.0magnetic101.1933680.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
9CONUS SouthUTS14b37.563198-113.3016632490.775hz2020-07-06 00:32:41+00:002020-07-20 17:43:45+00:0012714641.0magnetic0.00000090.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
10CONUS SouthUTS14c37.563198-113.3016632490.775ex2020-07-20 18:54:26+00:002020-07-28 16:38:25+00:006830391.0electric11.1933670.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
11CONUS SouthUTS14c37.563198-113.3016632490.775ey2020-07-20 18:54:26+00:002020-07-28 16:38:25+00:006830391.0electric101.1933670.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
12CONUS SouthUTS14c37.563198-113.3016632490.775hx2020-07-20 18:54:26+00:002020-07-28 16:38:25+00:006830391.0magnetic11.1933670.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
13CONUS SouthUTS14c37.563198-113.3016632490.775hy2020-07-20 18:54:26+00:002020-07-28 16:38:25+00:006830391.0magnetic101.1933670.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
14CONUS SouthUTS14c37.563198-113.3016632490.775hz2020-07-20 18:54:26+00:002020-07-28 16:38:25+00:006830391.0magnetic0.00000090.0countsFalse<HDF5 object reference><HDF5 object reference><HDF5 object reference>
\n", "
" ], "text/plain": [ " survey station run latitude longitude elevation component \\\n", "0 CONUS South UTS14 a 37.563198 -113.301663 2490.775 ex \n", "1 CONUS South UTS14 a 37.563198 -113.301663 2490.775 ey \n", "2 CONUS South UTS14 a 37.563198 -113.301663 2490.775 hx \n", "3 CONUS South UTS14 a 37.563198 -113.301663 2490.775 hy \n", "4 CONUS South UTS14 a 37.563198 -113.301663 2490.775 hz \n", "5 CONUS South UTS14 b 37.563198 -113.301663 2490.775 ex \n", "6 CONUS South UTS14 b 37.563198 -113.301663 2490.775 ey \n", "7 CONUS South UTS14 b 37.563198 -113.301663 2490.775 hx \n", "8 CONUS South UTS14 b 37.563198 -113.301663 2490.775 hy \n", "9 CONUS South UTS14 b 37.563198 -113.301663 2490.775 hz \n", "10 CONUS South UTS14 c 37.563198 -113.301663 2490.775 ex \n", "11 CONUS South UTS14 c 37.563198 -113.301663 2490.775 ey \n", "12 CONUS South UTS14 c 37.563198 -113.301663 2490.775 hx \n", "13 CONUS South UTS14 c 37.563198 -113.301663 2490.775 hy \n", "14 CONUS South UTS14 c 37.563198 -113.301663 2490.775 hz \n", "\n", " start end n_samples \\\n", "0 2020-07-05 23:19:41+00:00 2020-07-06 00:11:55+00:00 3134 \n", "1 2020-07-05 23:19:41+00:00 2020-07-06 00:11:55+00:00 3134 \n", "2 2020-07-05 23:19:41+00:00 2020-07-06 00:11:55+00:00 3134 \n", "3 2020-07-05 23:19:41+00:00 2020-07-06 00:11:55+00:00 3134 \n", "4 2020-07-05 23:19:41+00:00 2020-07-06 00:11:55+00:00 3134 \n", "5 2020-07-06 00:32:41+00:00 2020-07-20 17:43:45+00:00 1271464 \n", "6 2020-07-06 00:32:41+00:00 2020-07-20 17:43:45+00:00 1271464 \n", "7 2020-07-06 00:32:41+00:00 2020-07-20 17:43:45+00:00 1271464 \n", "8 2020-07-06 00:32:41+00:00 2020-07-20 17:43:45+00:00 1271464 \n", "9 2020-07-06 00:32:41+00:00 2020-07-20 17:43:45+00:00 1271464 \n", "10 2020-07-20 18:54:26+00:00 2020-07-28 16:38:25+00:00 683039 \n", "11 2020-07-20 18:54:26+00:00 2020-07-28 16:38:25+00:00 683039 \n", "12 2020-07-20 18:54:26+00:00 2020-07-28 16:38:25+00:00 683039 \n", "13 2020-07-20 18:54:26+00:00 2020-07-28 16:38:25+00:00 683039 \n", "14 2020-07-20 18:54:26+00:00 2020-07-28 16:38:25+00:00 683039 \n", "\n", " sample_rate measurement_type azimuth tilt units has_data \\\n", "0 1.0 electric 11.193362 0.0 counts False \n", "1 1.0 electric 101.193362 0.0 counts False \n", "2 1.0 magnetic 11.193362 0.0 counts False \n", "3 1.0 magnetic 101.193362 0.0 counts False \n", "4 1.0 magnetic 0.000000 90.0 counts False \n", "5 1.0 electric 11.193368 0.0 counts False \n", "6 1.0 electric 101.193368 0.0 counts False \n", "7 1.0 magnetic 11.193368 0.0 counts False \n", "8 1.0 magnetic 101.193368 0.0 counts False \n", "9 1.0 magnetic 0.000000 90.0 counts False \n", "10 1.0 electric 11.193367 0.0 counts False \n", "11 1.0 electric 101.193367 0.0 counts False \n", "12 1.0 magnetic 11.193367 0.0 counts False \n", "13 1.0 magnetic 101.193367 0.0 counts False \n", "14 1.0 magnetic 0.000000 90.0 counts False \n", "\n", " hdf5_reference run_hdf5_reference station_hdf5_reference \n", "0 \n", "1 \n", "2 \n", "3 \n", "4 \n", "5 \n", "6 \n", "7 \n", "8 \n", "9 \n", "10 \n", "11 \n", "12 \n", "13 \n", "14 " ] }, "execution_count": 18, "metadata": {}, "output_type": "execute_result" } ], "source": [ "m.channel_summary.clear_table()\n", "m.channel_summary.summarize()\n", "channel_df = m.channel_summary.to_dataframe()\n", "channel_df" ] }, { "cell_type": "code", "execution_count": 19, "id": "fc097d5f-2b8e-4d15-8927-ed7edbd1fcd0", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1m2024-08-28T15:52:26.355757-0700 | INFO | mth5.mth5 | close_mth5 | Flushing and closing test_dataset_definition.h5\u001b[0m\n" ] } ], "source": [ "m.close_mth5()" ] }, { "cell_type": "markdown", "id": "ab418a1e-f39e-4ef6-8b0b-53a5f071ceaa", "metadata": {}, "source": [ "From the above channel summary we can see that there are three runs, at station UTS14, the first being a few minutes long, the second about two weeks, and the third around 1 week." ] }, { "cell_type": "markdown", "id": "63bb9502-4d29-4b29-848d-247698be13d6", "metadata": {}, "source": [ "Insert code here for creating custom dataset" ] } ], "metadata": { "kernelspec": { "display_name": "aurora-test", "language": "python", "name": "aurora-test" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 5 }