Research & Experiments | Showcase

Opening

This section documents hypotheses that didn't pan out alongside those still in flight — without definitive conclusions. Reporting failures builds credibility, provides value to others on similar paths, and invites collaboration on unsolved problems.

Experiment 1 · Active

Causal Discovery for Business Optimisation

Can observational data reveal true cause-and-effect — moving decisions beyond correlation mining?

Ongoing

🔗

Problem & PC Algorithm

Correlation ≠ causation. "Customers using feature X have higher retention" → push to all → no improvement: users self-selected. The hypothesis: causal graphs from observational data identify high-leverage interventions.

PC Algorithm (Peter and Clark):

Start assuming all variables connected
Remove edges when conditionally independent
Orient edges via v-structures and d-separation
Output causal DAG — undirected edges where direction is indeterminate

Synthetic data: recovers most relationships; struggles with bidirectional causation; fails on non-linear relationships. Real data: often contradicts domain knowledge, indicating assumption violations.

⚠️

Assumption Failures in Business

Three core assumptions routinely break in practice:

Causal sufficiency — unmeasured confounders everywhere: competitor actions, economic conditions, personal circumstances
Acyclicity — feedback loops are the norm: satisfaction → behaviour → satisfaction
Faithfulness — near-cancellation effects: two causal paths with opposite signs summing to near-zero correlation

Methods being tested to relax assumptions:

ICA-based for hidden confounding (needs strong distributional assumptions)
Dynamic causal discovery for contemporaneous causation
Combine A/B test data to validate observational graphs

Practical Workflow

From algorithm to actionable insight

1 Algorithms generate candidate causal graphs as hypotheses, not conclusions
2 Domain experts review graphs, flagging structures that make no business sense
3 Design targeted experiments testing uncertain causal claims
4 Refine graphs with experimental evidence — observational discovery guides experimentation, not replaces it

Intermediate Findings & Next Steps

Even wrong graphs add value

Incorrect structures prompt questions about why they're incorrect — often revealing previously unrecognised confounders. Automated discovery + human expertise outperforms either alone.

Ongoing directions

Diagnostics flagging when assumptions likely fail
Domain knowledge as constraints (impossible edges, required orderings)
Cost-effectiveness vs. A/B testing for decisions of varying strategic importance

Experiment 2 · Concluded

Transfer Learning for Time Series

Pre-training on thousands of products to enable accurate early-lifecycle forecasting — compelling hypothesis, disappointing results.

Failed

💡

Hypothesis

New products lack years of data for accurate forecasting. Pre-train on thousands of existing products, fine-tune on weeks of new product data — enable accurate forecasts earlier in the lifecycle.

Approaches tried:

Global model — all products simultaneously, fine-tune final layers
Meta-learning — learn to adapt quickly from minimal data
Few-shot learning — compare to similar products

📉

Results

Transfer learning: 3–5% improvement over seasonal naïve — at high complexity and computational cost.

Simple baseline won: find the 3 most similar products (category, price point, demographics) → average patterns scaled to early sales → 8% improvement, far simpler.

🏆

Why Simple Won

Transfer fails when patterns aren't genuinely shared:

Seasonality varies by category (ice cream vs. soup)
Promotional response depends on brand and pricing
Lifecycle stages differ across products
Global model learned a "mushy average" fitting many products weakly

The simple approach uses specific patterns from genuinely comparable products — avoiding false transfer from unrelated ones.

Lessons from Failure

1 · Test simple baselines first

They are often surprisingly strong and should always be benchmarked before complex methods.

2 · Transfer requires shared structure

Source and target domains must actually share transferable patterns — which proves less true than hoped.

3 · Complexity has a cost

3% better but 3× more complex and fragile often represents net negative value in production.

Experiment 3 · Active

LLM-Assisted Feature Engineering

Can GPT-4 and similar models auto-generate feature transformations that match — or exceed — human practitioners?

In Progress

⚙️

Approach & Filter Criteria

Provide the LLM with schema, sample rows, prediction task, business context, and feature engineering docs. It generates Python code for:

Interaction features — products and ratios of existing features
Domain-specific features — time aggregations, categorical encodings
Mathematical transforms — logs, powers, binning

Code executes in a sandbox; features are filtered by:

Suspicious predictive power → likely leakage
Insufficient variance to discriminate outcomes
High correlation with existing features → redundancy
Requires future data unavailable at prediction time

📊

Early Results & Human vs. LLM Strengths

~30% of generated features pass basic sanity checks; ~10% prove genuinely useful in models. After 10–20 cycles, LLM feature sets are roughly comparable to human practitioners in predictive value.

Human advantages:

Domain knowledge for theoretically grounded features
Temporal reasoning to prevent leakage
Efficiency — high-quality candidates without iteration

LLM advantages:

Systematic exploration of large transformation spaces
Willingness to try unconventional combinations humans dismiss

Ongoing Directions

Explainability

Can LLMs explain generated features in ways domain experts can validate?

Multi-agent critique

One LLM generates features; another critiques for leakage and sensibility.

Domain fine-tuning

Fine-tune on domain-specific examples to improve the quality of suggestions over time.

Early conclusion: LLM-assisted feature engineering is a useful collaborative tool — LLMs generate candidates, humans filter for domain sensibility and validate. It won't replace practitioners.

Experiment 4 · Active

Interpretability via Visual Analytics

Interpretability is a spectrum — can rich interactive visualisations move complex models far enough to satisfy practical needs?

Ongoing

📈

Visualisation Tools

Tools being developed for three model types:

Classification — visual response surfaces, decision boundaries, and feature interactions as inputs vary
Time series — decompose predictions into trend, seasonality, and external regressors to show each forecast driver
Recommendation systems — visualise embedding space; geometric proximity makes collaborative filtering interpretable

✅

Business Evidence

Stakeholders who resisted "black box" models showed acceptance once visualisations revealed sensible behaviour:

Credit models relying on payment history — not demographic proxies
Demand forecasts responding correctly to promotions and holidays
Recommendation systems grouping products in intuitive ways

Visualisations didn't make models fully transparent — but provided sufficient interpretability for informed trust.

⚠️

Technical Challenges

Low-latency model evaluation required — limits to fast models or model distillation
High-dimensional inputs (hundreds of features) → slices and projections only, not full space
Genuinely interpretable charts require cognitive load and visual perception expertise data scientists often lack

Ongoing Work

Interpretability constraints

Training models with limited depth, monotonicity, and additive structure to enable clearer visualisation without sacrificing too much accuracy.

Automated chart recommendation

Systems that suggest appropriate visualisations by model type, prediction task, and user goals.

User studies

Measuring whether visualisations actually improve understanding — not just look impressive. A crucial distinction often overlooked.

This is our highest priority research stream: the most accurate model provides no value if stakeholders won't trust its predictions.

Key Insight — Transparency about what doesn't work provides as much value as reporting successes—simple baselines often outperform complex methods once implementation and maintenance costs are considered.

Key Takeaways

Real research involves failures and ongoing experiments without definitive conclusions
Transparency about what doesn't work provides as much value as successes
Simple baselines often outperform complex methods once implementation costs are considered
Emerging techniques show promise but require careful validation and domain integration
Collaborative human-AI approaches often outperform either alone

← Previous: Open Source Contributions Explore the Lab →

Research & Experiments: Honest Exploration