Print

Print


Why are there so many kernel threads in my processing?

I have a Bash script, attached. At its core it:

 * reads each line of the input file (sans the first one)
 * parses the line into fields
 * escapes single quote characters in a few fields
 * builds an SQL INSERT statement
 * appends the statement to a variable
 * when done with all lines, the variable is output as a file

This process seems too slow, and I can't figure out why. Granted, some of the input files may be megabytes in size, and some of my input files have more fields than others.

When I use htop to observe what is going on, I see half the processing is red (kernel processes).

Can you give me some advice as to why things are so slow? Is cat too big? Is tail too big? Is the parsing labor intensive? Do I have too many piped processes? Is sed inherently slow?

Attached is a a screen dump illustrating the state of the processors.

Any help would be appreciated. How can I increase the thru-put of my script? 

--
Eric Morgan
University of Notre Dame