Dynamically splitting output into files while processing
Sometimes I just want to write a script because it can be useful or because I’m curious to figure out how to solve a problem. Most of the times when I’m scripting one of the above is the motive to start and create a script for something and sometimes the purposes is completely unclear at first sight.
Sitting at a department store doing a bit of work, I was adjusting a Shell script that I’ve wrote. The shell script was generating a lot of output that serves as input for another process. Since the script needs to process thousands of entries I needed to wait for the script to finish.
Instead of waiting a couple of hours for the processing to be done I was looking for a way to automatically split the output of a shell script into a file after a certain amount of entries processed. In other words dynamically create different text files with the output generated by the script depending on the amount of processed entries.
The example script on the Wiki is a simplified version of a 30 minute scripting session, during which one of the employees of the department store asked if the food had tasted good and she could take away my serving tray.
The function of the example script is ‘very simple’, during the while loop the script counts till 10 and checks while processing an entry (in this case a number) if the amount of lines in the output file has been reached using the ‘wc -l’ command. Off course this can be changed to something else, like a certain value or word using the ‘grep’ command. For example if you want to generate a XML file and you want to check how many entries the generated XML file has you can use: ‘ grep -oc “Your XML entry open or close pattern” ‘ .
If the value of the counted lines is less then the amount that is set to split a file the current entry is added to the current file, else a new file is generated. The output filename in this script is dynamically generated using a begin and end number.
If you think you can generate the same output easier, faster etc. be my guest!