Deep Learning for Tabular Data

DSS5104

Overview

The goal of this assignment is to critically explore and evaluate recent deep learning methods for tabular data prediction. While deep learning has achieved dramatic success in domains like vision and language, tabular data, often structured, heterogeneous, and small-to-medium in size, has long remained a domain dominated by tree-based models such as XGBoost. However, in recent years, new deep learning architectures specifically tailored for tabular data have emerged, including TabNet, NODE, and FT-Transformer and a host of others. These models leverage the strengths of deep learning, such as representation learning and attention mechanisms, to tackle tabular prediction tasks.

In this assignment, you will evaluate the performance, robustness, and interpretability of a selection of these methods using publicly available implementations on a variety of non-trivial tabular prediction tasks. You are not expected to re-implement these models from scratch, but you are expected to understand how to use them properly and interpret their behavior.

Assignment Objectives

  • Understand and implement recent deep learning methods for tabular data
  • Critically assess their performance across multiple datasets
  • Compare their results against classical baselines like gradient boosting and random forests.
  • Analyze the strengths, weaknesses, and practical challenges of deep learning models in the tabular setting
  • Reflect on when and why these methods may be preferable, or not, to classical approaches

Datasets and Prediction Tasks

You must choose at least five datasets for tabular regression and/or classification. These should be:

  • Non-trivial in size, dimensionality, or complexity
  • From diverse domains (e.g., finance, healthcare, retail, education, etc.)
  • Publicly available

Key Questions to Address

In your report, aim to answer:

  1. Performance: How well do deep learning models perform relative to classical baselines? Are there specific datasets or characteristics where they shine or fail?
  2. Scalability: How do training times and resource demands compare?
  3. Sensitivity: How sensitive are results to hyperparameters, preprocessing, or random seed?
  4. Generalization: Do deep models overfit more easily on small tabular datasets?
  5. Practical Utility: Would you recommend using these models in practice? Why or why not?

Deliverables

  1. A written report in PDF format
    • Your report should be at most 10 pages long, although a much shorter report is possible (and preferable if you are concise).
    • No code should be included in the report.
  2. Your code
    • Either as a zip file or link to a GitHub repository
    • Clearly organized notebooks/scripts for:
      • Data preparation
      • Model training/evaluation
      • Metric reporting
      • Visualizations and interpretability analysis
    • Must be reproducible

Final Thoughts

This assignment encourages you to step beyond traditional modeling and engage with the frontier of machine learning for structured data. While deep learning for tabular data remains a debated and fast-evolving space, it is essential for practitioners to develop the critical skill of evaluating tools, not just using them.

You will gain hands-on experience that helps answer:

  • Are these models ready for production use?
  • When are they worth the added complexity?
  • What types of problems benefit from deep tabular models?

By the end, you should be able to advocate for or against these methods with informed reasoning, grounded in your own experiments.