Processing Configuration¶
The processing_config (“config”) together with the kernel dataset represent the key inputs to the processing pipeline. The processing config is based on the mt_metadata.base.Base class. This means it is a container that has a JSON or dictionary representation.
It is the purpose of the config to encapsulate all the parameters required for processing. There are many parameters and effort has been given to selecting some “reasonable” default values so that users do not need to worry about all of these parameters if they don’t want to.
The ProcessingConfig is expected to evolve with aurora as new functionalities become available. This is one reason why such a genreic data srtuctrue was selected In this tutorial, we will use the synthetic dataset example to show some of the features of the config object.
Hopefully, it will be fairly easy to add other parameters to the config, such as: - coherencey sorting - polarization sorting - ARMA prewhitening - Other tools from the community
There are two main ways one would normally build the object: 1. Use the ConfigCreator class 2. Edit existing config.json files
But one can also initialize a Processing object, which is what is done inside ConfigCreator
[1]:
from mt_metadata.transfer_functions.processing.aurora import Processing
[2]:
p = Processing()
[3]:
p
[3]:
{
"processing": {
"channel_nomenclature.ex": "ex",
"channel_nomenclature.ey": "ey",
"channel_nomenclature.hx": "hx",
"channel_nomenclature.hy": "hy",
"channel_nomenclature.hz": "hz",
"decimations": [],
"id": null,
"stations.local.id": null,
"stations.local.mth5_path": null,
"stations.local.remote": false,
"stations.local.runs": [],
"stations.remote": []
}
}
Using ConfigCreator¶
[4]:
from aurora.config import BANDS_DEFAULT_FILE
from aurora.config.config_creator import ConfigCreator
The config creator does not need any arguments to initialize, but it uses a KernelDataset to generate a processing object with station and run information, and default settings are applied to the processing parameters.
[5]:
cc = ConfigCreator()
The ConfigCreator class generates a processing config with default arguments if it is provided with a KernelDataset
Example of making a KernelDataset from an mth5¶
[6]:
from aurora.test_utils.synthetic.paths import SyntheticTestPaths
from mtpy.processing import RunSummary, KernelDataset
synthetic_test_paths = SyntheticTestPaths()
MTH5_PATH = synthetic_test_paths.mth5_path
Make an example file to work with
[7]:
mth5_path = MTH5_PATH.joinpath("test12rr.h5")
if mth5_path.exists():
pass
else:
synthetic_test_paths.mkdirs()
from mth5.data.make_mth5_from_asc import create_test12rr_h5
create_test12rr_h5(target_folder=MTH5_PATH)
[8]:
run_summary = RunSummary()
run_summary.from_mth5s([mth5_path,])
run_summary.df
24:08:28T15:59:26 | INFO | line:761 |mth5.mth5 | close_mth5 | Flushing and closing /home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5
[8]:
| channel_scale_factors | duration | end | has_data | input_channels | mth5_path | n_samples | output_channels | run | sample_rate | start | station | survey | run_hdf5_reference | station_hdf5_reference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 39999.0 | 1980-01-01 11:06:39+00:00 | True | [hx, hy] | /home/kkappler/software/irismt/aurora/data/syn... | 40000 | [ex, ey, hz] | 001 | 1.0 | 1980-01-01 00:00:00+00:00 | test1 | EMTF Synthetic | <HDF5 object reference> | <HDF5 object reference> |
| 1 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 39999.0 | 1980-01-01 11:06:39+00:00 | True | [hx, hy] | /home/kkappler/software/irismt/aurora/data/syn... | 40000 | [ex, ey, hz] | 001 | 1.0 | 1980-01-01 00:00:00+00:00 | test2 | EMTF Synthetic | <HDF5 object reference> | <HDF5 object reference> |
[9]:
kernel_dataset = KernelDataset()
kernel_dataset.from_run_summary(run_summary, "test1", "test2")
kernel_dataset.df
24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column fc, adding and setting dtype to <class 'bool'>.
24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column remote, adding and setting dtype to <class 'bool'>.
24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column run_dataarray, adding and setting dtype to <class 'object'>.
24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column stft, adding and setting dtype to <class 'object'>.
24:08:28T15:59:26 | INFO | line:250 |mtpy.processing.kernel_dataset | _add_columns | KernelDataset DataFrame needs column mth5_obj, adding and setting dtype to <class 'object'>.
[9]:
| channel_scale_factors | duration | end | has_data | input_channels | mth5_path | n_samples | output_channels | run | sample_rate | start | station | survey | run_hdf5_reference | station_hdf5_reference | fc | remote | run_dataarray | stft | mth5_obj | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 39999.0 | 1980-01-01 11:06:39+00:00 | True | [hx, hy] | /home/kkappler/software/irismt/aurora/data/syn... | 40000 | [ex, ey, hz] | 001 | 1.0 | 1980-01-01 00:00:00+00:00 | test1 | EMTF Synthetic | <HDF5 object reference> | <HDF5 object reference> | False | False | None | None | None |
| 1 | {'ex': 1.0, 'ey': 1.0, 'hx': 1.0, 'hy': 1.0, '... | 39999.0 | 1980-01-01 11:06:39+00:00 | True | [hx, hy] | /home/kkappler/software/irismt/aurora/data/syn... | 40000 | [ex, ey, hz] | 001 | 1.0 | 1980-01-01 00:00:00+00:00 | test2 | EMTF Synthetic | <HDF5 object reference> | <HDF5 object reference> | False | True | None | None | None |
[10]:
kernel_dataset.mini_summary
[10]:
| survey | station | run | start | end | duration | |
|---|---|---|---|---|---|---|
| 0 | EMTF Synthetic | test1 | 001 | 1980-01-01 00:00:00+00:00 | 1980-01-01 11:06:39+00:00 | 39999.0 |
| 1 | EMTF Synthetic | test2 | 001 | 1980-01-01 00:00:00+00:00 | 1980-01-01 11:06:39+00:00 | 39999.0 |
Create Config from KernelDataset¶
[11]:
config = cc.create_from_kernel_dataset(kernel_dataset)
24:08:28T15:59:26 | INFO | line:108 |aurora.config.config_creator | determine_band_specification_style | Bands not defined; setting to EMTF BANDS_DEFAULT_FILE
[12]:
config
[12]:
{
"processing": {
"band_setup_file": "/home/kkappler/software/irismt/aurora/aurora/config/emtf_band_setup/bs_test.cfg",
"band_specification_style": "EMTF",
"channel_nomenclature.ex": "ex",
"channel_nomenclature.ey": "ey",
"channel_nomenclature.hx": "hx",
"channel_nomenclature.hy": "hy",
"channel_nomenclature.hz": "hz",
"decimations": [
{
"decimation_level": {
"anti_alias_filter": "default",
"bands": [
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.23828125,
"frequency_min": 0.19140625,
"index_max": 30,
"index_min": 25
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.19140625,
"frequency_min": 0.15234375,
"index_max": 24,
"index_min": 20
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.15234375,
"frequency_min": 0.12109375,
"index_max": 19,
"index_min": 16
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.12109375,
"frequency_min": 0.09765625,
"index_max": 15,
"index_min": 13
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.09765625,
"frequency_min": 0.07421875,
"index_max": 12,
"index_min": 10
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.07421875,
"frequency_min": 0.05859375,
"index_max": 9,
"index_min": 8
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.05859375,
"frequency_min": 0.04296875,
"index_max": 7,
"index_min": 6
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 0,
"frequency_max": 0.04296875,
"frequency_min": 0.03515625,
"index_max": 5,
"index_min": 5
}
}
],
"decimation.factor": 1.0,
"decimation.level": 0,
"decimation.method": "default",
"decimation.sample_rate": 1.0,
"estimator.engine": "RME_RR",
"estimator.estimate_per_channel": true,
"extra_pre_fft_detrend_type": "linear",
"input_channels": [
"hx",
"hy"
],
"method": "fft",
"min_num_stft_windows": 2,
"output_channels": [
"ex",
"ey",
"hz"
],
"pre_fft_detrend_type": "linear",
"prewhitening_type": "first difference",
"recoloring": true,
"reference_channels": [
"hx",
"hy"
],
"regression.max_iterations": 10,
"regression.max_redescending_iterations": 2,
"regression.minimum_cycles": 10,
"regression.r0": 1.5,
"regression.tolerance": 0.005,
"regression.u0": 2.8,
"regression.verbosity": 0,
"save_fcs": false,
"window.clock_zero_type": "ignore",
"window.num_samples": 128,
"window.overlap": 32,
"window.type": "boxcar"
}
},
{
"decimation_level": {
"anti_alias_filter": "default",
"bands": [
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 1,
"frequency_max": 0.0341796875,
"frequency_min": 0.0263671875,
"index_max": 17,
"index_min": 14
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 1,
"frequency_max": 0.0263671875,
"frequency_min": 0.0205078125,
"index_max": 13,
"index_min": 11
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 1,
"frequency_max": 0.0205078125,
"frequency_min": 0.0166015625,
"index_max": 10,
"index_min": 9
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 1,
"frequency_max": 0.0166015625,
"frequency_min": 0.0126953125,
"index_max": 8,
"index_min": 7
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 1,
"frequency_max": 0.0126953125,
"frequency_min": 0.0107421875,
"index_max": 6,
"index_min": 6
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 1,
"frequency_max": 0.0107421875,
"frequency_min": 0.0087890625,
"index_max": 5,
"index_min": 5
}
}
],
"decimation.factor": 4.0,
"decimation.level": 1,
"decimation.method": "default",
"decimation.sample_rate": 0.25,
"estimator.engine": "RME_RR",
"estimator.estimate_per_channel": true,
"extra_pre_fft_detrend_type": "linear",
"input_channels": [
"hx",
"hy"
],
"method": "fft",
"min_num_stft_windows": 2,
"output_channels": [
"ex",
"ey",
"hz"
],
"pre_fft_detrend_type": "linear",
"prewhitening_type": "first difference",
"recoloring": true,
"reference_channels": [
"hx",
"hy"
],
"regression.max_iterations": 10,
"regression.max_redescending_iterations": 2,
"regression.minimum_cycles": 10,
"regression.r0": 1.5,
"regression.tolerance": 0.005,
"regression.u0": 2.8,
"regression.verbosity": 0,
"save_fcs": false,
"window.clock_zero_type": "ignore",
"window.num_samples": 128,
"window.overlap": 32,
"window.type": "boxcar"
}
},
{
"decimation_level": {
"anti_alias_filter": "default",
"bands": [
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 2,
"frequency_max": 0.008544921875,
"frequency_min": 0.006591796875,
"index_max": 17,
"index_min": 14
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 2,
"frequency_max": 0.006591796875,
"frequency_min": 0.005126953125,
"index_max": 13,
"index_min": 11
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 2,
"frequency_max": 0.005126953125,
"frequency_min": 0.004150390625,
"index_max": 10,
"index_min": 9
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 2,
"frequency_max": 0.004150390625,
"frequency_min": 0.003173828125,
"index_max": 8,
"index_min": 7
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 2,
"frequency_max": 0.003173828125,
"frequency_min": 0.002685546875,
"index_max": 6,
"index_min": 6
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 2,
"frequency_max": 0.002685546875,
"frequency_min": 0.002197265625,
"index_max": 5,
"index_min": 5
}
}
],
"decimation.factor": 4.0,
"decimation.level": 2,
"decimation.method": "default",
"decimation.sample_rate": 0.0625,
"estimator.engine": "RME_RR",
"estimator.estimate_per_channel": true,
"extra_pre_fft_detrend_type": "linear",
"input_channels": [
"hx",
"hy"
],
"method": "fft",
"min_num_stft_windows": 2,
"output_channels": [
"ex",
"ey",
"hz"
],
"pre_fft_detrend_type": "linear",
"prewhitening_type": "first difference",
"recoloring": true,
"reference_channels": [
"hx",
"hy"
],
"regression.max_iterations": 10,
"regression.max_redescending_iterations": 2,
"regression.minimum_cycles": 10,
"regression.r0": 1.5,
"regression.tolerance": 0.005,
"regression.u0": 2.8,
"regression.verbosity": 0,
"save_fcs": false,
"window.clock_zero_type": "ignore",
"window.num_samples": 128,
"window.overlap": 32,
"window.type": "boxcar"
}
},
{
"decimation_level": {
"anti_alias_filter": "default",
"bands": [
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 3,
"frequency_max": 0.00274658203125,
"frequency_min": 0.00213623046875,
"index_max": 22,
"index_min": 18
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 3,
"frequency_max": 0.00213623046875,
"frequency_min": 0.00164794921875,
"index_max": 17,
"index_min": 14
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 3,
"frequency_max": 0.00164794921875,
"frequency_min": 0.00115966796875,
"index_max": 13,
"index_min": 10
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 3,
"frequency_max": 0.00115966796875,
"frequency_min": 0.00079345703125,
"index_max": 9,
"index_min": 7
}
},
{
"band": {
"center_averaging_type": "geometric",
"closed": "left",
"decimation_level": 3,
"frequency_max": 0.00079345703125,
"frequency_min": 0.00054931640625,
"index_max": 6,
"index_min": 5
}
}
],
"decimation.factor": 4.0,
"decimation.level": 3,
"decimation.method": "default",
"decimation.sample_rate": 0.015625,
"estimator.engine": "RME_RR",
"estimator.estimate_per_channel": true,
"extra_pre_fft_detrend_type": "linear",
"input_channels": [
"hx",
"hy"
],
"method": "fft",
"min_num_stft_windows": 2,
"output_channels": [
"ex",
"ey",
"hz"
],
"pre_fft_detrend_type": "linear",
"prewhitening_type": "first difference",
"recoloring": true,
"reference_channels": [
"hx",
"hy"
],
"regression.max_iterations": 10,
"regression.max_redescending_iterations": 2,
"regression.minimum_cycles": 10,
"regression.r0": 1.5,
"regression.tolerance": 0.005,
"regression.u0": 2.8,
"regression.verbosity": 0,
"save_fcs": false,
"window.clock_zero_type": "ignore",
"window.num_samples": 128,
"window.overlap": 32,
"window.type": "boxcar"
}
}
],
"id": "test1-rr_test2_sr1",
"stations.local.id": "test1",
"stations.local.mth5_path": "/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5",
"stations.local.remote": false,
"stations.local.runs": [
{
"run": {
"id": "001",
"input_channels": [
{
"channel": {
"id": "hx",
"scale_factor": 1.0
}
},
{
"channel": {
"id": "hy",
"scale_factor": 1.0
}
}
],
"output_channels": [
{
"channel": {
"id": "ex",
"scale_factor": 1.0
}
},
{
"channel": {
"id": "ey",
"scale_factor": 1.0
}
},
{
"channel": {
"id": "hz",
"scale_factor": 1.0
}
}
],
"sample_rate": 1.0,
"time_periods": [
{
"time_period": {
"end": "1980-01-01T11:06:39+00:00",
"start": "1980-01-01T00:00:00+00:00"
}
}
]
}
}
],
"stations.remote": [
{
"station": {
"id": "test2",
"mth5_path": "/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5",
"remote": true,
"runs": [
{
"run": {
"id": "001",
"input_channels": [
{
"channel": {
"id": "hx",
"scale_factor": 1.0
}
},
{
"channel": {
"id": "hy",
"scale_factor": 1.0
}
}
],
"output_channels": [
{
"channel": {
"id": "ex",
"scale_factor": 1.0
}
},
{
"channel": {
"id": "ey",
"scale_factor": 1.0
}
},
{
"channel": {
"id": "hz",
"scale_factor": 1.0
}
}
],
"sample_rate": 1.0,
"time_periods": [
{
"time_period": {
"end": "1980-01-01T11:06:39+00:00",
"start": "1980-01-01T00:00:00+00:00"
}
}
]
}
}
]
}
}
]
}
}
You can see the entire config by executing the cell below, or you can cut and paste the json code into a json editor (e.g. https://jsoneditoronline.org) and then you can look at it heirarchically.
You can also transform the processing object to a json string
[13]:
json_string = config.to_json()
[14]:
json_string
[14]:
'{\n "processing": {\n "band_setup_file": "/home/kkappler/software/irismt/aurora/aurora/config/emtf_band_setup/bs_test.cfg",\n "band_specification_style": "EMTF",\n "channel_nomenclature.ex": "ex",\n "channel_nomenclature.ey": "ey",\n "channel_nomenclature.hx": "hx",\n "channel_nomenclature.hy": "hy",\n "channel_nomenclature.hz": "hz",\n "decimations": [\n {\n "decimation_level": {\n "anti_alias_filter": "default",\n "bands": [\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.23828125,\n "frequency_min": 0.19140625,\n "index_max": 30,\n "index_min": 25\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.19140625,\n "frequency_min": 0.15234375,\n "index_max": 24,\n "index_min": 20\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.15234375,\n "frequency_min": 0.12109375,\n "index_max": 19,\n "index_min": 16\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.12109375,\n "frequency_min": 0.09765625,\n "index_max": 15,\n "index_min": 13\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.09765625,\n "frequency_min": 0.07421875,\n "index_max": 12,\n "index_min": 10\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.07421875,\n "frequency_min": 0.05859375,\n "index_max": 9,\n "index_min": 8\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.05859375,\n "frequency_min": 0.04296875,\n "index_max": 7,\n "index_min": 6\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 0,\n "frequency_max": 0.04296875,\n "frequency_min": 0.03515625,\n "index_max": 5,\n "index_min": 5\n }\n }\n ],\n "decimation.factor": 1.0,\n "decimation.level": 0,\n "decimation.method": "default",\n "decimation.sample_rate": 1.0,\n "estimator.engine": "RME_RR",\n "estimator.estimate_per_channel": true,\n "extra_pre_fft_detrend_type": "linear",\n "input_channels": [\n "hx",\n "hy"\n ],\n "method": "fft",\n "min_num_stft_windows": 2,\n "output_channels": [\n "ex",\n "ey",\n "hz"\n ],\n "pre_fft_detrend_type": "linear",\n "prewhitening_type": "first difference",\n "recoloring": true,\n "reference_channels": [\n "hx",\n "hy"\n ],\n "regression.max_iterations": 10,\n "regression.max_redescending_iterations": 2,\n "regression.minimum_cycles": 10,\n "regression.r0": 1.5,\n "regression.tolerance": 0.005,\n "regression.u0": 2.8,\n "regression.verbosity": 0,\n "save_fcs": false,\n "window.clock_zero_type": "ignore",\n "window.num_samples": 128,\n "window.overlap": 32,\n "window.type": "boxcar"\n }\n },\n {\n "decimation_level": {\n "anti_alias_filter": "default",\n "bands": [\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 1,\n "frequency_max": 0.0341796875,\n "frequency_min": 0.0263671875,\n "index_max": 17,\n "index_min": 14\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 1,\n "frequency_max": 0.0263671875,\n "frequency_min": 0.0205078125,\n "index_max": 13,\n "index_min": 11\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 1,\n "frequency_max": 0.0205078125,\n "frequency_min": 0.0166015625,\n "index_max": 10,\n "index_min": 9\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 1,\n "frequency_max": 0.0166015625,\n "frequency_min": 0.0126953125,\n "index_max": 8,\n "index_min": 7\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 1,\n "frequency_max": 0.0126953125,\n "frequency_min": 0.0107421875,\n "index_max": 6,\n "index_min": 6\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 1,\n "frequency_max": 0.0107421875,\n "frequency_min": 0.0087890625,\n "index_max": 5,\n "index_min": 5\n }\n }\n ],\n "decimation.factor": 4.0,\n "decimation.level": 1,\n "decimation.method": "default",\n "decimation.sample_rate": 0.25,\n "estimator.engine": "RME_RR",\n "estimator.estimate_per_channel": true,\n "extra_pre_fft_detrend_type": "linear",\n "input_channels": [\n "hx",\n "hy"\n ],\n "method": "fft",\n "min_num_stft_windows": 2,\n "output_channels": [\n "ex",\n "ey",\n "hz"\n ],\n "pre_fft_detrend_type": "linear",\n "prewhitening_type": "first difference",\n "recoloring": true,\n "reference_channels": [\n "hx",\n "hy"\n ],\n "regression.max_iterations": 10,\n "regression.max_redescending_iterations": 2,\n "regression.minimum_cycles": 10,\n "regression.r0": 1.5,\n "regression.tolerance": 0.005,\n "regression.u0": 2.8,\n "regression.verbosity": 0,\n "save_fcs": false,\n "window.clock_zero_type": "ignore",\n "window.num_samples": 128,\n "window.overlap": 32,\n "window.type": "boxcar"\n }\n },\n {\n "decimation_level": {\n "anti_alias_filter": "default",\n "bands": [\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 2,\n "frequency_max": 0.008544921875,\n "frequency_min": 0.006591796875,\n "index_max": 17,\n "index_min": 14\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 2,\n "frequency_max": 0.006591796875,\n "frequency_min": 0.005126953125,\n "index_max": 13,\n "index_min": 11\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 2,\n "frequency_max": 0.005126953125,\n "frequency_min": 0.004150390625,\n "index_max": 10,\n "index_min": 9\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 2,\n "frequency_max": 0.004150390625,\n "frequency_min": 0.003173828125,\n "index_max": 8,\n "index_min": 7\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 2,\n "frequency_max": 0.003173828125,\n "frequency_min": 0.002685546875,\n "index_max": 6,\n "index_min": 6\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 2,\n "frequency_max": 0.002685546875,\n "frequency_min": 0.002197265625,\n "index_max": 5,\n "index_min": 5\n }\n }\n ],\n "decimation.factor": 4.0,\n "decimation.level": 2,\n "decimation.method": "default",\n "decimation.sample_rate": 0.0625,\n "estimator.engine": "RME_RR",\n "estimator.estimate_per_channel": true,\n "extra_pre_fft_detrend_type": "linear",\n "input_channels": [\n "hx",\n "hy"\n ],\n "method": "fft",\n "min_num_stft_windows": 2,\n "output_channels": [\n "ex",\n "ey",\n "hz"\n ],\n "pre_fft_detrend_type": "linear",\n "prewhitening_type": "first difference",\n "recoloring": true,\n "reference_channels": [\n "hx",\n "hy"\n ],\n "regression.max_iterations": 10,\n "regression.max_redescending_iterations": 2,\n "regression.minimum_cycles": 10,\n "regression.r0": 1.5,\n "regression.tolerance": 0.005,\n "regression.u0": 2.8,\n "regression.verbosity": 0,\n "save_fcs": false,\n "window.clock_zero_type": "ignore",\n "window.num_samples": 128,\n "window.overlap": 32,\n "window.type": "boxcar"\n }\n },\n {\n "decimation_level": {\n "anti_alias_filter": "default",\n "bands": [\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 3,\n "frequency_max": 0.00274658203125,\n "frequency_min": 0.00213623046875,\n "index_max": 22,\n "index_min": 18\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 3,\n "frequency_max": 0.00213623046875,\n "frequency_min": 0.00164794921875,\n "index_max": 17,\n "index_min": 14\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 3,\n "frequency_max": 0.00164794921875,\n "frequency_min": 0.00115966796875,\n "index_max": 13,\n "index_min": 10\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 3,\n "frequency_max": 0.00115966796875,\n "frequency_min": 0.00079345703125,\n "index_max": 9,\n "index_min": 7\n }\n },\n {\n "band": {\n "center_averaging_type": "geometric",\n "closed": "left",\n "decimation_level": 3,\n "frequency_max": 0.00079345703125,\n "frequency_min": 0.00054931640625,\n "index_max": 6,\n "index_min": 5\n }\n }\n ],\n "decimation.factor": 4.0,\n "decimation.level": 3,\n "decimation.method": "default",\n "decimation.sample_rate": 0.015625,\n "estimator.engine": "RME_RR",\n "estimator.estimate_per_channel": true,\n "extra_pre_fft_detrend_type": "linear",\n "input_channels": [\n "hx",\n "hy"\n ],\n "method": "fft",\n "min_num_stft_windows": 2,\n "output_channels": [\n "ex",\n "ey",\n "hz"\n ],\n "pre_fft_detrend_type": "linear",\n "prewhitening_type": "first difference",\n "recoloring": true,\n "reference_channels": [\n "hx",\n "hy"\n ],\n "regression.max_iterations": 10,\n "regression.max_redescending_iterations": 2,\n "regression.minimum_cycles": 10,\n "regression.r0": 1.5,\n "regression.tolerance": 0.005,\n "regression.u0": 2.8,\n "regression.verbosity": 0,\n "save_fcs": false,\n "window.clock_zero_type": "ignore",\n "window.num_samples": 128,\n "window.overlap": 32,\n "window.type": "boxcar"\n }\n }\n ],\n "id": "test1-rr_test2_sr1",\n "stations.local.id": "test1",\n "stations.local.mth5_path": "/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5",\n "stations.local.remote": false,\n "stations.local.runs": [\n {\n "run": {\n "id": "001",\n "input_channels": [\n {\n "channel": {\n "id": "hx",\n "scale_factor": 1.0\n }\n },\n {\n "channel": {\n "id": "hy",\n "scale_factor": 1.0\n }\n }\n ],\n "output_channels": [\n {\n "channel": {\n "id": "ex",\n "scale_factor": 1.0\n }\n },\n {\n "channel": {\n "id": "ey",\n "scale_factor": 1.0\n }\n },\n {\n "channel": {\n "id": "hz",\n "scale_factor": 1.0\n }\n }\n ],\n "sample_rate": 1.0,\n "time_periods": [\n {\n "time_period": {\n "end": "1980-01-01T11:06:39+00:00",\n "start": "1980-01-01T00:00:00+00:00"\n }\n }\n ]\n }\n }\n ],\n "stations.remote": [\n {\n "station": {\n "id": "test2",\n "mth5_path": "/home/kkappler/software/irismt/aurora/data/synthetic/mth5/test12rr.h5",\n "remote": true,\n "runs": [\n {\n "run": {\n "id": "001",\n "input_channels": [\n {\n "channel": {\n "id": "hx",\n "scale_factor": 1.0\n }\n },\n {\n "channel": {\n "id": "hy",\n "scale_factor": 1.0\n }\n }\n ],\n "output_channels": [\n {\n "channel": {\n "id": "ex",\n "scale_factor": 1.0\n }\n },\n {\n "channel": {\n "id": "ey",\n "scale_factor": 1.0\n }\n },\n {\n "channel": {\n "id": "hz",\n "scale_factor": 1.0\n }\n }\n ],\n "sample_rate": 1.0,\n "time_periods": [\n {\n "time_period": {\n "end": "1980-01-01T11:06:39+00:00",\n "start": "1980-01-01T00:00:00+00:00"\n }\n }\n ]\n }\n }\n ]\n }\n }\n ]\n }\n}'
Which can be saved:
[15]:
with open("config.json", "w") as fid:
fid.write(json_string)
Default Parameters¶
The default config parameters are listed below:
[16]:
input_channels = ["hx", "hy"],
output_channels = ["hz", "ex", "ey"],
estimator = None,
emtf_band_file = BANDS_DEFAULT_FILE,
What can user change with the Config?¶
Channel Nomenclature
for example ex,ey maybe called e1, e2 in the mth5
This is handled by passing a channel_nomenclature keyword argument.
Examples of systems that might need this are LEMI and Phoenix
Windowing Parameters
Window shape (family)
Window length
Sliding window overlap
Clock-Zero (optional)
Choice of Stations
This is currently done via KernelDataset
Scale Factors for Individual Channels
allows for correcting, or trying to add simple, frequency independent gain corrections
Frequency Bands
Group the Fourier coefficients into bands to be processed together and averaged for a TF estimate
currently via an emtf-style band_setup file
Number of Decimation Levels
currently via an emtf-style band_setup file
Regression Estimator Engine
Currently only choices are RME, RME_RR (Regression M-estimate) and Remote reference regresson M-estimate
Regression Parameters
Maximum number of iterations
Maximum number of redecending iterations
Minimum number of frequecny cycles
EMTF Band Setup File¶
The frequency bands will eventually be setup in a variety of ways, but currently aurora supports only specification of bands either by explicit construction or via EMTF “band setup” files.
[17]:
BANDS_DEFAULT_FILE
[17]:
PosixPath('/home/kkappler/software/irismt/aurora/aurora/config/emtf_band_setup/bs_test.cfg')
Here is the content of a typical EMTF band setup file:
25 1 25 30 1 20 24 1 16 19 1 13 15 1 10 12 1 8 9 1 6 7 1 5 5 2 14 17 2 11 13 2 9 10 2 7 8 2 6 6 2 5 5 3 14 17 3 11 13 3 9 10 3 7 8 3 6 6 3 5 5 4 18 22 4 14 17 4 10 13 4 7 9 4 5 6These legacy files have the following significance; The first line, 25 indicates the number of bands, and there are 25 lines following, one line per frequency band.
Each line comprises three numbers:
decimation_level, first_FC_index, last_FC_index
where “FC” stands for Fourier coefficient
The decimation factor applied at each level was controlled in EMTF by a sepearate file, called decset.cfg. In the old EMTF codes, this controlled the window length, overlap, the decimation factor, and the corners of the anti-alias filter applied before downsampling.
The decimation factor in EMTF was almost always 4, and the default behaviour of the ConfigCreator is to assume a decimation factor of 4 at each level, but this can be changed manually.