Before training a large language model, raw corpus must pass through parsing, cleaning, deduplication, quality scoring, tokenization, and sample assembly. At petabyte scale, engineering teams face not only mounting compute bills but also hundreds of tables, ever-growing feature sets, and anomalous data points that can silently invalidate entire batch runs. This year's VLDB Industrial Track Best Paper zeroes in on a stage of LLM training that is critical yet often overlooked: data preparation.
Ant Group's unified wide-table system, OmniTable, already manages more than 35PB and over 305 billion LLM training records in production, spanning data domains including web text, code, PDFs, and SFT. In one real-world SFT data preparation task, the end-to-end cycle was shortened from approximately 14 days to 2.5 days, while manual operational steps dropped from 45 to 12. This result did not come from a faster machine — it came from changing how data engineers organize data and features.
Traditional LLM data processing is built around physical tables: after a data source is ingested, parsing results land in one table, cleaning produces another, and quality scores, domain labels, and deduplication signatures continue spawning yet more tables — with web, code, PDF, and SFT pipelines each maintaining their own separate stacks. As data sources and features keep growing, the maintenance burden explodes. The paper documents a real-world case: to add a single feature, engineers had to wrangle 106 tables on a task canvas, yet those tables only stored results, rarely recording how those results were derived.
OmniTable's core principle is logical unification with physical separation. At the logical layer, each row represents a traceable data entity, and each column holds the state of a given processing stage or a derived feature. Two types of system fields keep columns aligned: a global primary key ensures the same data record shares a consistent identifier across different sources and stages, while ingestion batch, provenance, and version are all recorded. Data backfills, point lookups, and lineage tracing thus gain stable anchor points. The physical layer can split rows, split columns, merge small files, or build materialized views — all without changing the upper-level schema.
Feature computation shifts from drawing task diagrams to declaring target columns. The system directly inspects the status of those columns on the target batch: already-completed results are reused, and missing ancestor columns enter the execution plan. Once a task completes successfully, the system atomically registers the batch, feature columns, version, and column-level lineage. Information that once lived scattered across scripts, scheduling platforms, and manual records now converges into a single metadata plane. Engineers define feature semantics; the system handles dependency expansion, execution routing, and result submission.
Unstructured corpus inevitably contains anomalous encodings, overlong texts, or corrupted content. Once data reaches hundreds of millions of records, even an extremely low anomaly rate produces a large number of bad samples. OmniTable isolates common UDF failures at the record level: each invocation carries a timeout and memory check, anomalous records have their sample IDs logged and written as NULL, and all other records continue processing. In a controlled comparison on a 500GB feature task with roughly 600 million records, enabling fault isolation allowed about 99.995% of records to complete in a single pass at approximately 6.2 hours with no manual intervention — whereas the legacy process required three rounds of troubleshooting and re-submission, totaling roughly 52 hours.
Feature computation for LLM data varies widely in compute characteristics: text length checks and rule-based filtering suit CPU or SQL engines, while model inference may be offloaded to GPUs. OmniTable selects backends across Spark, MaxCompute SQL, and GPU inference platforms based on declarations, operator profiles, and cluster load, while also fusing features that read the same column into a single scan. In an operator fusion experiment spanning approximately 2.5PB and over 300 billion records, scan counts dropped from 8 to 1, CPU time decreased by 55.9%, and end-to-end runtime fell from 38 hours to 14 hours.
As LLM training enters the petabyte era, the data engineering challenge is no longer about getting a single task to run — it is about keeping continuously growing data, features, and computations manageable over the long term. What OmniTable seeks to save is not just machine runtime, but also the time engineers spend hunting for tables, patching tasks, and debugging anomalies. The logic of unified wide tables plus a single metadata plane also offers a valuable reference for platform-grade infrastructure that coordinates computing power and data: if compute scheduling and data orchestration can share one observable, traceable foundation, enterprise-grade AI iteration at scale will truly accelerate. StarWar Cloud's sustained investment in compute scheduling and platform engineering is precisely about turning this kind of data-compute synergy into a standardized foundational capability.