Having dealt with a lot of big data I often came to realization that we actually did not need most of it.

Try being intentional and smart in front of your data pipeline and purge data that is not useful. Too many times people store data "just in case" and that case never happens years later.