Regarding HDF5 and DataFrames, I think the standard Panas interface does load the file into memory but it is possible to implement the DataFrame API in a way that fetches the data lazily from disk. This is done in the dask and bcolz packages.