The use of the Transformer-XL and bi-directional recurrent neural networks in cyberbullying detection for Bengali text offers multiple advantages, notably in contextual sensitivity, sequential understanding, and resilience to limited-resource language constraints. Transformer-XL is chosen for its superior handling of long-range dependencies, enabling it to capture nuanced contextual relationships critical in identifying cyberbullying language patterns. Its segment-level recurrence mechanism effectively extends attention spans, which is especially useful for processing long or contextually complex texts. In parallel, the BiGRU-BiLSTM layers bring a bi-directional sequential learning approach, allowing the model to learn from both past and future tokens, thus capturing the sequential dynamics that often characterize abusive language.

In our study, the Transformer-XL with BiGRU-BiLSTM model (Fusion Transformer-XL) is deployed to maximize these strengths. Extensive preprocessing steps include data cleaning, label encoding, and upsampling for class balancing, while data augmentation techniques enhance the robustness of the training dataset. Text tokenization is managed through a pre-trained tokenizer to preserve semantic context. For performance evaluation, accuracy and F1-score are calculated, providing a comprehensive view of the model's predictive power, especially in the face of imbalanced data. Additionally, k-fold cross-validation ensures generalizability and robustness across diverse data subsets.

To further clarify model decisions, Local Interpretable Model-agnostic Explanation (LIME) is applied, providing transparent explanations of the model’s predictions. A cross-dataset evaluation on an English-language dataset further tests adaptability, reinforcing the model’s potential for multilingual application. This model setup and evaluation method underscore the effectiveness and reliability of combining Transformer-XL with bi-directional recurrent layers for detecting cyberbullying, particularly in under-resourced languages like Bengali.