7 Ways to Handle Large Data Files for Machine Learning
- by 7wData
Exploring and applying machine learning algorithms to datasets that are too large to fit into memory is pretty common.
This leads to questions like:
In this post, I want to offer some common suggestions you may want to consider.
Some machine learning tools or libraries may be limited by a default memory configuration.
Check if you can re-configure your tool or library to allocate more memory.
A good example is Weka, where you can increase the memory as a parameter when starting the application.
Are you sure you need to work with all of the data?
Take a random sample of your data, such as the first 1,000 or 100,000 rows. Use this smaller sample to work through your problem before fitting a final model on all of your data (using progressive data loading techniques).
I think this is a good practice in general for machine learning to give you quick spot-checks of algorithms and turnaround of results.
You may also consider performing a sensitivity analysis of the amount of data used to fit one algorithm compared to the model skill. Perhaps there is a natural point of diminishing returns that you can use as a heuristic size of your smaller sample.
Do you have to work on your computer?
Perhaps you can get access to a much larger computer with an order of magnitude more memory.
For example, a good option is to rent compute time on a cloud service like Amazon Web Services that offers machines with tens of gigabytes of RAM for less than a US dollar per hour.
I have found this approach very useful in the past.
Is your data stored in raw ASCII text, like a CSV file?
Perhaps you can speed up data loading and use less memory by using another data format. A good example is a binary format like GRIB, NetCDF, or HDF.
There are many command line tools that you can use to transform one data format into another that do not require the entire dataset to be loaded into memory.
Using another format may allow you to store the data in a more compact form that saves memory, such as 2-byte integers, or 4-byte floats.
[Social9_Share class=”s9-widget-wrapper”]
Upcoming Events
Shift Difficult Problems Left with Graph Analysis on Streaming Data
29 April 2024
12 PM ET – 1 PM ET
Read MoreCategories
You Might Be Interested In
Machine Learning Leading to Revolution in Clinical Data Management
16 Mar, 2020“What I’ve been looking at for the past few years is how things are evolving within the clinical trial space, …
6 Big data Trends in 2022 Democratizing ‘Real-time’ Data
28 Feb, 2022The high-volume information, that demands innovative and effective processing for better insight and decision making, is what defines big data. …
Digital transformation: 5 early warning signs you’re moving too slowly
27 Sep, 2020Due to COVID-19, most industries have had to make significant changes in how they do business. Firms can no longer …
Recent Jobs
Do You Want to Share Your Story?
Bring your insights on Data, Visualization, Innovation or Business Agility to our community. Let them learn from your experience.