fix(datalake): fix 5 critical bugs in transformation pipeline (!8) · Merge requests · Infraestructura / Portal Caabsa

Infraestructura requested to merge diego-dev into main Apr 16, 2026

Bug 1: replace non-existent build_cte_chain() calls with compile_pipeline_sql() and compile_preview_sql() in datalake.py

Bug 2: _gen_rename now preserves non-renamed columns via input_columns config or SELECT * fallback in pipeline_compiler.py

Bug 3: cast config shape aligned to compiler expectation (casts list-of-dicts instead of columns dict) in data_studio.py

Bug 4: compile_preview_sql accepts limit=None to avoid LIMIT 0 in count queries that always returned 0

Bug 5: replaced asyncio.create_task in sync route with FastAPI BackgroundTasks to prevent RuntimeError on notification failures

Also: added add_column and fixed aggregate config in data_studio, fixed up_to_step_order to use step_order instead of len()

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

fix(datalake): fix 5 critical bugs in transformation pipeline