# Run a 3DXRD workflow

## Get started

To check the different options to install and start processing _3DXRD_ in graphical user interface kindly follow the [Installation](./installation.md) page.

:::

Then, open the Orange canvas by running

```bash
ewoks-canvas
```

The typical 3DXRD workflow can then be opened by navigation to _Help_ _>_ _Example workflows_. You will get the following workflow

![3DXRD workflow](_static/tuto/3dxrd_workflow.png)

:::{attention}
From now on, remember to save regularly your workflow!
:::

## Peak segmentation

The first step of the workflow is to extract diffraction peaks from the raw data, a process known as segmentation.

### Loading Data

Double-click the **Peaks segmentation** node to open the widget.

![Empty Peaks segmentation window](_static/tuto/empty_segmentation.png)

- **Select Master File**: At the top-left, click _Browse_ to select the Bliss master file containing your _3DXRD_ data.

- **Auto-Configuration**: Once loaded, the widget automatically detects _motor_ and _detector_ names. If a dropdown has only one valid option, it will be disabled to prevent errors.

- **Analyse Folder**: You can modify the _Analyse Folder Path_ where the segmented output files will be stored.

### Configuring the Segmenter

The widget provides a real-time preview of the segmentation for a single frame.

- **Visual Feedback**: The left panel shows the background-corrected image; the right panel shows the raw image with markers indicating detected peaks.
- **Algorithm Selection**: By default, the widget uses the _Gaussian Peak Search_ algorithm. You can change the algorithm via the dropdown in the _Segmentation Parameters box_. Changing the algorithm will automatically update the settings box with relevant parameters.

:::{tip}
If you are unsure about a parameter, hover over the field to see a detailed hint.
:::

### Frame Selection and Tuning

- **Frame Selection**: The widget initially targets the frame with the highest intensity (_Frame: auto_). Use the slider in the _Omega and frame selection_ section to choose a specific _frame index_. The corresponding _omega_ value will be displayed next to the slider.
- **Live Updates**: Any change to parameters in the _Segmentation Parameters_ or _Correction files_ sections will automatically re-trigger the segmentation for the current preview frame.

(validating-segmentation)=
### Validating segmentation

Now, the aim is to find the right segmentation parameters for your sample. For this, change the parameters and observe how the detected peak positions evolve.

Ideally, you should get something like this:

![Image showing an example of a segmented frame with good segmentation parameters](_static/tuto/good_segmentation.png)

Namely:
- ✅ Detected peaks are centered on diffraction spots
- ✅ No peaks are detected outside of diffraction spots
- ✅ One-to-one correspondance with detected peaks and diffraction spots

For comparison, we show here an example where segmentation parameters are not adapted:

![Image showing an example of a segmented frame with bad segmentation parameters](_static/tuto/bad_segmentation.png)

Namely:
- ❌ Multiple peaks are detected for one diffraction spot
- ❌ Detected peaks are mislaligned with respect to the diffraction spot
- ❌ Peaks are detected although there is no corresponding diffraction spot

### Executing Full Scan
Once the parameters are tuned for a single frame, click _Execute Segment Scan_ to process the entire stack.

While the scan is running, the widget will be disabled and a progress bar will indicate the status. Upon completion, a window will display the positions of all segmented peaks, color-coded by their omega rotation angle.
![Peaks segmentation window](_static/tuto/segmentation.png)

```{note}
While not mandatory, it is possible to renormalize frame intensities during the full segmentation. For this, provide the name of the monitor dataset via the _Monitor_ dropdown.

The monitor data can be inspected by clicking _Show Monitor Data_.
```

The progress bar at the bottom will show the progress of the segmentation. Once the segmentation is complete, you will get a pop-up with a plot showing the full segmentation results

![Image showing the full segmentation result](_static/tuto/full_segmentation_result.png)

```{tip}
If you closed the window showing the full segmentation results, you can show it again by clicking on the 👁️ button near the _Execute Segment Scan_.
```

These results are automatically saved in a HDF5 file located in the analysis folder chosen above, in a subfolder that will depend on the Bliss _sample_ and _dataset_ name, mirroring the `RAW_DATA` structure. See the  {ref}`Data format page <output>` for more information.

All tasks results will be saved in this output file inside different groups. Segmented results will be saved under the group `<scan_number>/segmented_3d_peaks`.

If you are happy with the result, you can move to the next widget _Detector correction_.

Else, change the parameters, inspect the results for single frame (see [Validating segmentation](#validating-segmentation) above) and re-run the segmentation until it is satisfactory. The _Overwrite Segmentation Result_ will have to be checked to overwrite previous results contained in the group `segmented_3d_peaks` of the output file.

## Detector correction

Real detector modules are not completely flat and this can lead to errors in the estimation of the coordinates of peaks in the reciprocal space. The aim of the _Detector correction_ is to correct those errors.

Double-click on the _Detector correction_ node, you will get the following window

![Image showing an empty detector correction widget](_static/tuto/empty_detector_correction.png)

On the left, you can choose between a correction using a **Spline file** or **two e2dx/e2dy files**. Pick the appropriate one and load the correction file(s) via the corresponding _Browse_ button(s).

Click then on _Execute detector correction_. The plot on the right will update with two different set of points:

![Image showing the detector correction widget after task completion](_static/tuto/detector_correction.png)

The first set of points (blue points in the picture) correspond to the uncorrected peak positions from the previous _Peaks Segmentation_ task while the second set (red crosses in the picture) correspond to the corrected peak positions according to the loaded detector correction files. The plot therefore allows to visualize the effect of the detector correction.

The results will be saved in the scan group of the output file under `spatial_corrected_peaks`.

The following tasks will use the corrected peaks positions.

## Geometry transformation

The peaks positions are given in the detector frame of reference. For further data treatment, it is needed to project the positions in the lab frame of reference. For this, you have to supply geometric parameters describing your setup in the _Geometry transformation_ widget.

Double-click on the _Geometry transformation_ node, you will get the following window

![Image showing an empty geometry transformation widget](_static/tuto/empty_geometry_transformation.png)

To supply the parameters, load the `.par` file obtained from [calibration](./calibration.md) by clicking on _Browse_ on the top-left _Geometry settings_ section.

Once the geometry parameters are set, click on _Compute Geometry Vectors for Peaks_. The plot on the right will then show the computed azimuthal angle vs. reciprocal distance of all peaks.

The results will be saved in the scan group of the output file under `spatial_corrected_peaks`.

![Image showing the geometry transformation widget after task completion](_static/tuto/geometry_transformation.png)

If the geometry was well parametrized, peaks should be more or less grouped in vertical lines that correspond to the diffraction rings. 

To help diagnosis, you can load a file containing the expected lattice parameters. For this, check the _Show lattice rings_ checkbox and click _Browse_ to load a `.par` or `.cif` file (see [File formats](./data_formats.md)). The corresponding lattice rings will appear on the plot as colored vertical lines.

![Image showing the geometry transformation widget after task completion with lattice rings](_static/tuto/geometry_transformation_with_rings.png)

If the peaks positions match the lattice rings (as shown on the screenshot above), you can move to the next widget.

```{warning}
If this is not the case, you can edit the geometry parameters manually by clicking on _Edit parameters_ in the _Geometry settings_ group. Remember to check _Overwrite_ to overwrite the previous results contained in the output file.

But the best thing would be to redo a [calibration](./calibration.md) to produce a new `.par` file.
```

## Filtering

```{attention}
Did you save your workflow?

By opening a saved workflow, you can resume working at the last executed task.
```

Before moving to **Indexing** (i.e. attributing peaks to grains), it is essential to filter the peaks found by the segmentation steps to keep only the ones that are relevant. The filtering goes through three steps:

1. **Lattice filtering**: keep only peaks that are close to the expected peaks for a given lattice. A lattice file needs to be provided.
1. **Intensity filtering**: keep only a fraction of peaks to keep based on intensity.
1. **Index filtering**: same as lattice filtering but ring indices must be provided to keep only the peaks close to those rings.

All these widgets operate according the same principle: some peaks are given as input, the task filters some of them and returns the peaks that will be kept for the next operations.

### Lattice filtering

Double-click on the _Lattice Filtering_ node, you will get the following window

![Image showing the initial state of the lattice filtering widget](_static/tuto/initial_lattice_filtering.png)

The peaks from the previously run _Geometry Transformation_ task will be displayed as crosses on the plot (displayed again as computed azimuthal angle vs. reciprocal distance). These are the input or unfiltered peaks.

The aim of the widget is to keep only the peaks that are close a lattice ring. This allows to discard peaks that do not have crystallographic meaning.

For this, you have to load a lattice file corresponding to the expected lattice of the sample grains. Click _Browse_ on the top-left and chose a `.par` or `.cif` file containing the lattice parameters.

Once this is done, lattice rings will be shown as colored lines on the plot. Click now _Filter Peaks_ to start filtering according to these rings.

After task completion, the peaks that will be kept will appear as round markers.

![Image showing the lattice filtering widget after task completion](_static/tuto/lattice_filtering.png)

You can now check if all the relevant peaks are kept. If not, tune the _Tolerance Settings_ below:
- _Reciprocal Distance Tolerance_: this is the maximum distance between a peak and a ring for the peak to be kept. Increasing it makes the filtering more tolerant (more peaks are kept).
- _Max Reciprocal Distance_: this is an optional cutoff value. If the checkbox is checked, all peaks above this reciprocal distance will be discarded.

As usual, remember to check _Overwrite_ if you run the task once again. You can also choose to change the _Output NeXus Group Name_ to save the peaks in a different group name (default: `lattice_filtered_peaks`).

### Intensity filtering

Double-click on the _Intensity Filtering_ node, you will get the following window

![Image showing the initial state of the intensity filtering widget](_static/tuto/initial_intensity_filtering.png)

Again, the peaks from the previous task (_Lattice Filtering_) will be displayed as crosses on the plot. But this time, the plot is showing the intensity of the peaks against the reciprocal distance.

This widget allows to filter a fraction of the peaks to keep only the ones with the highest intensity. 

By default, the _Intensity Fraction_ parameter is set to `1.0` so that all peaks are kept. Low-intensity peaks can be discarded by reducing this value 

For example, `0.5` will only discard half the peaks, keeping the ones with highest intensity. But usually, the fraction should be between `0.95` and `0.99` depending on the dataset.

By clicking on _Filter Peaks_, the results will be saved in the output file and the plot will be updated as kept peaks will be appear as round markers.

![Image showing the intensity filtering widget after task completion](_static/tuto/intensity_filtering.png)

If you do run the task again, check _Overwrite_ to overwrite to existing results or change the _Output NeXus Group Name_ to save the peaks in a different group name (default: `intensity_filtered_peaks`).

### Index filtering

Double-click on the _Index Filtering_ node, you will get the following window

![Image showing the initial state of the index filtering widget](_static/tuto/initial_index_filtering.png)

The peaks from _Intensity Filtering_ will be displayed as crosses on the plot (as intensity of the peaks vs. the reciprocal distance). 

In addition, the _Lattice file_ field will be filled with the lattice used during the _Lattice Filtering_ task so that lattice rings will also be shown on the plot (as colored lines).

In this widget, you will choose to keep only peaks for some of the diplayed rings. The rings are identified by their index starting from the left: the leftmost one is `0`, the one after is `1`, and so on.

Select rings by typing their indices as comma-separated values in the _Select Rings_ field. The _Reciprocal Distance Tolerance_ should be set to the same value set in _Lattice Filtering_ (default: `0.01`).

When clicking _Filter according the selected rings_, the results will be saved in the output file (default: `indexer_filtered_peaks`) and the plot will be updated as kept peaks will be appear as round markers.

![Image showing the initial state of the index filtering widget](_static/tuto/index_filtering.png)

As for previous filtering tasks, the output group name can be changed with the _Output NeXus Group Name_ field and _Overwrite_ can be checked to overwrite existing results when re-running the task.

## Grid indexing

Now that you have filtered the peaks to keep only those that are relevant, you can proceed to the task of generating grains, a process called indexing. This process uses a grid-based algorithm.

### Initial Setup

Double-click the _Grid Indexing_ node to open the interface.

The **Incoming Peaks** tab displays the peaks passed from the previous filtering task (_Index filtering_). Before starting the process, configure the parameters in the control panel on the left:

![Image showing the initial state of the grid indexing widget](_static/tuto/initial_grid_indexing.png)


- **Sample and Beam Info**: These parameters define the spatial grid for grain generation.
1. **Dimensions**: The grid should roughly match the illuminated area: _Sample Width_ × _Sample Depth_ × _Beam Height_.
1. **Grid Step**: This defines the minimum distance between grains.
- **Grid Index Settings**: These control the generation algorithm. Hover over any parameter to view a tooltip with detailed information.


```{admonition} _Ring 1_ and _Ring 2_

The generation algorithm runs for every couple (`r1`,`r2`) with `r1` the ring indices of _Ring 1_ and `r2` the ring indices of _Ring 2_.

**Example**: if _Ring 1_ is `0, 1` and _Ring 2_ `0, 1`, the algorithm will run for the following couples: `(0, 0)`, `(0, 1)`, `(1, 0)` and `(1, 1)`

Also, do mind that **_Ring 2_ should be a subset of _Ring 1_**.
```

### Execution and Monitoring

Once the parameters are set, click _Run grid indexing_. A window **Grid Index Logs** will pop-up and periodically update with grain statistics, grain positions and tolerance levels.

As the computation progresses, the **Generated Grains** tab on the right will update. The size of the grain will represent the mean intensity of the associated peaks while the color will represent the number of associated peaks. 

When finished, the final list of grains is displayed in the 3D plot and saved in the output file's scan group under `grid_indexed_grains`.

Use the 3D plot to explore the spatial distribution and orientation of the generated grains to evaluate the results (see next section) before moving to the final task.

### Evaluating results

Manual validation remains a crucial step in the 3DXRD data processing. You should have a rough idea of the crystallinity of your sample to evaluate if the number of found grains is right or not. 

Note also that the spheres are approximations and not representative of the real grain shapes.
 
For example, the following picture shows a result where the generated grains are actually not realistic.

![Image showing badly generated grains from Grid Indexing](_static/tuto/bad_grid_indexing.png)

A good grain indexing usually shows fewer grains with a better repartition of the segmented peaks and less overlap:

![Image showing the generated grains plot of the grid indexing widget](_static/tuto/grid_indexing.png)

A **high grain count** may indicate noise being interpreted as real signal. Restarting from the _Peaks segmentation_ step and making sure that no peaks are detected outside of diffraction spots may improve the indexing.

If **grains overlap a lot**, you may need to run the _Grid Indexing_ task by choosing different, more well-behaved, rings for _Ring 1_ and _Ring 2_. Double-checking your geometry parameters by redoing a [calibration](./calibration.md) and restarting from the _Geometry transformation_ step may also be useful.

```{admonition} Info
A slight grain overlap may still be observed even with good parameters since the sphere representation of grain is only an approximation.
```

### Troubleshooting

If you receive an error stating the grain list is empty:
* Try **decreasing the _Number of Peaks_** requirement so that more grains are retained during indexing.
* Review your **segmentation and filtering** steps to ensure a sufficient number of peaks were passed through.
* Verify that the **lattice parameters** used during filtering are correct for your material.

## Refine grain mapping

The objective of this final task is to iteratively refine the crystallographic and spatial parameters of the grains obtained during the Grid Indexing stage. This ensures higher accuracy in the final reconstructed grain map.

### Setup

Double-click the _Refine Grain Mapping_ node to open the interface.

![Image showing the initial state of the refine grain mapping widget](_static/tuto/initial_refine_grain_mapping.png)

In the _Refinement settings panel_, you can configure the optimization algorithm. As with previous widgets, hover over any parameter for a detailed tooltip.

A very important parameter is the **Tolerances** field where multiple tolerances must be given, separated by commas. The iterative refinement algorithm will go through each tolerance subsequently to optimize the grain orientations and positions. In other words, the number of tolerances represent the number of iterations of the algorithm and the tolerances should be decreasing to refine grain parameters further and further.

### Execution

Once your parameters are configured, double-click on _Refine Grain Mapping_.

During the process, a pop-up window will track the progress of the iterations and the 3D plot and statistics will update periodically as the refined grain parameters are computed.

Once the computation finishes, the plot on the right will be updated one last time with the refined grains and these will be saved in the scan group of the output file under `make_map_grains`.

![Image showing the the refine grain mapping widget after task completion](_static/tuto/refine_grain_mapping.png)

## Final words

You have now reached the end of the Ewoks3DXRD workflow tutorial! You should now have an HDF5 file containing all the results, looking like the one below (viewed here with [myHDF5](https://myhdf5.hdfgroup.org/)) 

![Image showing a HDF5 file with all groups produced by Ewoks3DXRD](_static/tuto/final_hdf5.png)

You can inspect the results outside of the workflow by using the CLI `ewoks3dxrd-grain-vis`

```bash
ewoks3dxrd-grain-vis <path_to_output_file> -e <scan_number (default: 1.1)>
```

See the [Visualization page](./result_visualization.md) for more info on this.

Processing 3DXRD data is a complex task and there is a lot of information left out from this tutorial for the sake of simplicity. Dedicated sections will be created in the future for deeper dives on certain operations and concepts.

In the mean time, you can still open an [issue on our Gitlab repository](https://gitlab.esrf.fr/workflow/ewoksapps/ewoks3dxrd/-/issues/new) if you encountered any issue during this tutorial or would like to provide feedback.

Thanks for reading!