bash/shell scripting ideas

[zahc]

I have a config file like so:

Code: [Select]

<some dumb markup>
FOO,this,is,a,CSV,file,CHART1
BAR,this,is,a,CSV,file,CHART2
...
<more dumb markup
I have a shell script that wgets and filters this like so:

Code: [Select]

wget -O - $URL | grep -E '*[A-Z]{2}' > mycharts.data
This works dandy but now I want to exclude certain configs from mycharts.data. In other words, I want to print lines to mycharts.data ONLY if they DO NOT any tokens in a blacklist file. If I can do that, I can avoid adding blacklisting code to every other program that accesses mycharts.data. The blacklist file is simple; it's just a list of charts to be blacklisted.

Blacklist file:

Code: [Select]

CHART1
CHARTXyz
TRAHC1
...
I can't use a hard-coded regexp because the blacklist file can be different every time. I'm failing. I don't know whether to use sed or awk or whatever for the match. I tried something like this

Code: [Select]

wget -O - | grep -E '*[A-Z]{2}' | awk '!/BLACK|LISTED|CHARTS/' > mycharts.data

I need a way to loop through the regexpen stored in the blacklist file, but I'm not having any luck.


[zahc]

I'd like to put something in the pipeline, but I'll settle for this, which is working. Just delete lines from the final result afterward.

Code: [Select]

while read blacklist
do
    sed -i '/.*'$blacklist'.*/d' mycharts.data
done < blacklist.txt
This requires looping over the whole results file one time for each blacklisted chart, but at least it's working.

[GigaBuist]

Without testing...

Code: [Select]

wget -O - $URL | grep -E '*[A-Z]{2}' | grep -v -f blacklist.txt > mycharts.data

Tested:  Doesn't work.

OK.  This should be pretty simple.  I'm going to keep plugging.  Is mixing Perl into the equation possible?  This is sorta the reason Larry Wall made the beast.

But this should still be easy with a simple shell script.  Hurmph.  You have me intrigued.

[zahc]

I could use perl. However, that would be cheating. I have previously argued against turning this entire script into a perl script, so using inline perl would by hypocritical.

Half my problem was not understanding shell-quoting. For the longest time I couldn't get shell variables into sed or awk statements, because they are single-quoted. The breakthrough was realizing I can do

Code: [Select]

sed '/.*'$blacklist'.*/d'and the shell will pop in the variable first. Now that I know this, I could probably arrive at an awk statement that I could put in the pipeline.

[GigaBuist]

Ok, wasn't as weird as I thought.  Here's my terminal output w/ commands.  Should be easy to follow.

Code: [Select]

justinb@justinb-1005HA:~$ cat pats.txt
HI
WORLD
BLACK
LIST
ME
LAPD
justinb@justinb-1005HA:~$ cat black.txt
BLACK
LIST
ME
justinb@justinb-1005HA:~$ cat pats.txt | grep -Evf black.txt
HI
WORLD
LAPD


[zahc]

I should have known there was a simple grep option to do exactly what I need. I should rtfm more.

       -P, --perl-regexp
         Interpret PATTERN as a Perl regular expression.

wha? That is awesome-sauce right there.

[roo_ster]

Neat.

I loves me some tcsh, awk & sed, but moving on to perl if necessary is no shame.  Right tool for the job & all.  Besides, if you can make the perl work from the command line, it isn't cheating.  It is only cheating if you have to drop back, punt, and write a perl script in another text file.

Heck, I have begun prototyping tcsh, awk, sed, and perl code to parse & work over output from my models with the express intent for my code guy (soon to be Dr. Code Guy) to later convert into python for incorporation into the official postprocessor.  I figure that is better than waving my hands about, saying "I want it to do this-and-such" and then gripe when it doesn't grab the data I want and format it the way I want.

I do sometimes throw him for a loop with my funky meta- and generative code when I use tcsh/linux utilities.  (He comes from the Windows world.)  When he figures it out, his response many times is, "It would take me half a day to write that line in the script in C or Python."  Which is why I prototype them and store them to the side until he has time to make it happen and make it faster.

I should have known there was a simple grep option to do exactly what I need. I should rtfm more.
wha? That is awesome-sauce right there.

Yeah, me, too. 

I just re-read grep's man page last month, while scripting and was like, "Really?  No shinola?" a few places.  Some pow-er-ful stuff in some of these utilities.