> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chainbase.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Decentralized DORA Tuning

### Abstract

$D^2ORA$ (Decentralized Weight-Decomposed Low-Rank Adaptation) extends the DoRA algorithm to a decentralized environment, enabling distributed training across multiple servers.

### $D^2ORA$ Algorithm

#### Step 1: Initialization

We first initialize the parameter matrices of Theia and the low-rank matrices.

* **Magnitude Vector Initialization**:
  Initialize the magnitude vector $m$ using the column-wise norm of the pre-trained weight matrix $W_0$:
  $$
  m = ||W_0||_c
  $$

* **Directional Matrix Initialization**:
  Set the directional matrix $V$ to the pre-trained weight matrix $W_0$:
  $$
  V = W_0
  $$

* **Low-Rank Matrices Initialization**:
  Initialize the low-rank matrices $B$ and $A$ for the LoRA method.

#### Step 2: Decomposition

Decompose the pre-trained weight matrix $W_0$ into its magnitude and direction components. The pre-trained weight is directly derived from the NLP model.

* **Magnitude Decomposition**:
  $$
  m = ||W_0||_c
  $$

* **Directional Decomposition**:
  $$
  V = W_0 / ||W_0||_c
  $$

#### Step 3: Distributed Training

For each server in the decentralized network, perform the following:

* **Distribution**:
  Distribute the initialized $m$, $V$, $B$, and $A$ to all servers.

* **Iterations**:
  For each iteration $t$ from 1 to $T$:

  * **Weight Update Calculation**:
    Compute the weight update $\Delta W$ using the low-rank adaptation method:
    $$
    \Delta W = BA
    $$

  * **Magnitude Vector Update**:
    Update the magnitude vector $m$:
    $$
    m = m - \eta \nabla_m L(m, V)
    $$

  * **Directional Matrix Update**:
    Update the low-rank matrices $B$ and $A$ with gradient descent:
    $$
    B = B - \eta \nabla_B L(B, A)
    $$
    $$
    A = A - \eta \nabla_A L(B, A)
    $$

#### Step 4: Aggregation

After all iterations are completed, collect updates from all servers and aggregate them:

* **Magnitude Vector Aggregation**:
  $$
  m' = \text{aggregate}(m)
  $$

* **Directional Matrix Aggregation**:
  $$
  V' = \text{aggregate}(V)
  $$

* **Low-Rank Matrices Aggregation**:
  $$
  B' = \text{aggregate}(B)
  $$
  $$
  A' = \text{aggregate}(A)
  $$

#### Step 5: Merging

Merge the updated components to form the final weight matrix:

* **Final Weights Calculation**:
  $$
  W' = m' \frac{V'}{||V'||_c} + B'A'
  $$

#### Step 6: Return Updated Weights

Return the updated weights $W'$ as the final output of the algorithm. We use it as the weight parameters of Theia.

### Summary

D^2ORA enhances the capabilities of DoRA by enabling decentralized training across multiple servers. This approach leverages blockchain technology to ensure trustworthiness, making it suitable for training AI models in a distributed and secure environment.