Data Mining Tool: 64bit, Linux OS

[roo_ster]

Looking for a data mining tool.

It will be combing through gigabytes worth of text files and manipulating the data to create reports/etc that could go into MS Excel.  64bit.

A similar tool, Clementine, cost someone doing the same thing as we will $90K and it craps out on them.  They can only get a 32bit of Clementine and I suspect it runs out of memory (or at least the ability to address enough memory).

I'd prefer an open-source tool that runs on a 64bit linux OS.  We got some big hoss RHEL5 machines with beau coup RAM on which to run the beast in a cluster environment (LSF).

Any pointers would be appreciated.

[mtnbkr]

Perl.

That's what I used when I was doing that sort of thing. 

If the files are logs, you could use Splunk. 

Chris

[roo_ster]

Yep, I was thinking Perl as a backstop.

They are time-stamped logs, but not your usual IT-type logs.

[mtnbkr]

might still work with Splunk.  It's free to try and has a free non-business license as well.

[RevDisk]

I'd prefer an open-source tool that runs on a 64bit linux OS.  We got some big hoss RHEL5 machines with beau coup RAM on which to run the beast in a cluster environment (LSF).

Any pointers would be appreciated.

You want KNIME.

[CNYCacher]

look into awk and sed as well as grep

[roo_ster]

look into awk and sed as well as grep

awk, sed, & grep are my home boys; but are likely not up to the task for this.

Perl has all the awk, sed, grep functionality with fewer limitations. If it comes to scripting, Perl will be the hammer.

[CNYCacher]

awk, sed, & grep are my home boys; but are likely not up to the task for this.

Perl has all the awk, sed, grep functionality with fewer limitations. If it comes to scripting, Perl will be the hammer.

Have you considered transforming those rows of logs into SQL tables?  Lots of ways you could then build reports.  If you set the proper indexes the report generation would be fast fast fast as well.

[roo_ster]

Have you considered transforming those rows of logs into SQL tables?  Lots of ways you could then build reports.  If you set the proper indexes the report generation would be fast fast fast as well.

If  text-based data mining tool does not do the trick, that is likely the next step.

[tyme]

If you want to do sql queries on the data, and you're not going to concurrently write to the database, SQLite is fast, simple, and speaks reasonably good sql.  If that won't work, there's mysql and postgresql of course.  Even Access and MSSQL Server Express wouldn't be bad for this if it weren't for their database size limits -- apparently 2GB and 4GB.

It looks like Clementine, now called IBM SPSS Modeller, is more of a predictive analysis program which will be more statistical in focus.  SQL doesn't have the kind of statistical functions that you'd want.  If that's the kind of data mining you're looking for, I suggest R, which is a free, open-source, multi-platform, widely used clone of S+.

[GigaBuist]

Even Access and MSSQL Server Express wouldn't be bad for this if it weren't for their database size limits -- apparently 2GB and 4GB.

The Express version of MSSQL 2008R2 has a DB limit of 10GB now, up from the 4GB limit in 2008.