Multi-Agent Automated Feature Engineering for High-Dimensional Big Data

Abstract

Feature engineering remains one of the most critical yet time-consuming bottlenecks in building effective machine learning pipelines, especially in high-dimensional big data environments where the feature space is vast, noisy, and often poorly understood. Manual feature engineering demands significant domain expertise, is difficult to scale, and frequently fails to uncover complex, non-linear relationships hidden within the data. This paper proposes a Multi-Agent Framework for Automated Feature Engineering (MAFE) designed to address these challenges through intelligent automation, specialization, and inter-agent coordination.

Functionally, the framework operates by deploying a population of autonomous agents, each assigned a specialized role in the feature transformation pipeline. These roles include feature generators, feature selectors, redundancy eliminators, and performance evaluators. Agents interact through a competitive-collaborative mechanism — competing to propose the most predictive feature subsets while collaborating by sharing high-value transformations via a shared knowledge pool. A master orchestrator agent governs agent interactions, resolves conflicts, and enforces computational constraints, ensuring the system remains efficient and scalable across large datasets.

On the technical side, each agent is powered by reinforcement learning policies that iteratively refine transformation strategies based on reward signals derived from downstream model performance metrics such as AUC, F1-score, and cross-validation accuracy. The framework integrates graph-based feature dependency modeling to detect and eliminate multicollinearity, while a meta-learning module accelerates convergence by transferring knowledge from previously solved feature engineering tasks. Distributed computing support via Apache Spark enables the framework to handle datasets exceeding millions of rows and thousands of features without significant performance degradation.

Empirical evaluations conducted across diverse benchmark datasets — including financial, genomic, and IoT domains — demonstrate that MAFE consistently outperforms both manual feature engineering approaches and existing AutoML baselines in predictive accuracy, feature interpretability, and computational efficiency. This work makes a significant contribution to the AutoML landscape by presenting a robust, adaptive, and production-ready solution to one of data science's most persistent challenges."