Opened 9 years ago

Last modified 9 years ago

#432 new feature

Split big files using user-defined bins

Reported by: srkline Owned by: srkline
Priority: minor Milestone: TISANE
Component: SANS Reduction Keywords:
Cc: Blocking:


As a first stab at processing:

For very big files that need to be split for processing, one possibility is to split and decimate the original file to get an overview of the whole file. Then, you can figure out what the optimal binning setup is for the data. Finally, you want to process the full set of events, not the decimated ones. So re-split the original event file using the "custom" binning and number of splits, rather than linear. One sticking point is that the correspondence between (time) and (MB of file) is approximate, since there are many types of events. But it's roughly linear (on log-log).

Change History (1)

comment:1 Changed 9 years ago by srkline

Depending on the file size, the file > 500MB may be loaded. I don't know the upper limit, and it is certainly going to be machine-dependent. Even on a fast machine, the graphics display are intolerably slow for a 500 MB event file (approx 44 Million XY events). But it possible could be loaded and without viewing the event details or derivative, binned and exported.

Note: See TracTickets for help on using tickets.