Pylon: Semantic Table Union Search in Data Lakes

Understand

The large size and fast growth of data repositories, such as data lakes, has spurred the need for data discovery to help analysts find related data.

• The problem has become challenging as (i) a user typically does not know what datasets exist in an enormous data repository; and (ii) there is usually a lack of a unified data model to capture the interrelationships between heterogeneous datasets from disparate sources.
• In this work, we address one important class of discovery needs: finding union-able tables.
• The task is to find tables in a data lake that can be unioned with a given query table.