Crashy linux help

[zahc]

I have a linux desktop at work that, until 2 days ago, was completely stable. 200+ days of uptime and no problems at all. Running ubuntu 10.04, if memory serves. I mostly use it for piddly perl stuff and administering a website via ssh.

All of a sudden, it locked up and I had to restart. Ever since, the system will only stay up for a few hours at most. Often, the display goes weird with strange blocking and tearing, while I can still kind-of use the gui. The system will do weird things, like the GUI will work, but I can't launch any programs...even simple things like nautilus. Sometimes the simplest programs like top and du instantly die saying 'killed'. Or they start but then segfault. One time I switched to a virtual terminal to try to restart the xserver, but I couldn't log in...when I entered my credentials I got a screen of debug information including weird memory addresses and a message saying BUG: unable to handle kernel paging request. Since I can't log in, I had to power-cycle. Or the system just locks up hard.

I ran 3 passes of memtest this morning and no errors.

Are there any log files I can look at that might tell me what's wrong? I can try a clean reinstall, but my home directory is encrypted and I'm worried that re-mounting it to a new install might be interesting.

[mtnbkr]

untrue.  Linux doesn't crash.

In all seriousness, what are you seeing in /var/log/alllog?  Maybe tail -f alllog while you try to do something?  Try booting to the CLI and perusing the logs in /var/log.

Chris

[lee n. field]

My first thought is one you already covered -- RAM diags.  RAM problems, in my experience, can make a system generally flakey like that.

Also do HD diagnostics.   Ultimate Boot CD has a bunch, look for either Seatools GUI or the appropriate manufacturer's diagnostic.  Look for hard disk errors in, uhhhh, I think /var/log/kern.log.

And just for grins, scan your motherboard for bowed up capacitors.  I have seen that in relatively modern systems.

but my home directory is encrypted and I'm worried that re-mounting it to a new install might be interesting.

Well, that'll be a learning experience.

[mtnbkr]

How much memory?  Are you using swapfile?  If so, is it mounted?  What do you get when you run df -h? 

Chris

[zahc]

1 gig of memory. I don't know about swap; the system is churning memtest right now. I had a theory that since the system crashes after a couple hours, that I should run memtest for at least a couple hours in case the problem is intermittent.

I will check some of those error log paths to see what I find, after I reboot.

And just for grins, scan your motherboard for bowed up capacitors.  I have seen that in relatively modern systems.

You mean look at them optically?

[lee n. field]

You mean look at them optically?

yes

[zahc]

Wow, this is really interesting.

So I have my box sitting there running memtest86. I needed the flash drive that was plugged into the front panel USB port, so I pulled it out. BAM, instantly there was a red entry on the memtest screen, like the exact moment I pulled the flash drive out. I figured that's pretty weird, but it must be a coincidence, because how can a flash drive have anything to do with anything? It's running off a liveCD and not even mounting anything. Probably not even scanning the USB ports for devices.

Well, a couple minutes later it's still bothering me, so I took a USB SD card reader and stuck it in the same USB port--with no SD card in it. Pulled it back out. BAM, another red row on the memtest screen the instant I pulled it out. Tried again. Nothing. Tried over and over...nothing. Tried wiggling the SD card reader around. Nothing. Tried leaving it in for a few minutes then pulling it out. Nothing. Tst 6 and 7 had errors, 5 pass, 2 errors.

Edit: there's definitely something weird with putting the flash drive in and out. I don't know what it means, but by doing it several times I have managed to fill up the memtest screen with errors; now there are 10 errors.

[vaskidmark]

Have you tried turning it over, holding it over your head, and shaking it hard?

Always resolves computer problems for me.

stay safe.

[lee n. field]

Pull the hard disk, install into another box with similar specs.  Kernel hardware support is broad enough that it shouldn't have a problem unlike a certain popular but problematic OS.

See if problem follows hard disk.

I like brute force, sometimes.

[mtnbkr]

Pull the hard disk, install into another box with similar specs.  Kernel hardware support is broad enough that it shouldn't have a problem unlike a certain popular but problematic OS.

I guess I should expect my PC to crash any day now considering I pulled the drive from my old PC, plugged it into my new one (that was a completely different CPU, new Mobo, etc), rebooted a few times, and went on about my life.  Windows 7 isn't smart enough to know it's supposed to crash constantly.

Chris

[zahc]

I tried taking out one memory module at a time and running memtest. They both test ok unless I plug things into the front USB ports. My theory is that the PSU or mobo caps are bad and plugging something in blips the power rail just enough to cause memory errors. I rebooted the system with only a keyboard and monitor plugged in, and will see how long it stays up. A mouse would be nice, though.

[RocketMan]

Zahc, if the mobo caps are indeed toast, and if you have a compelling reason keep that particular hardware running, let me know.   I'd be happy to change the caps out for you at cost. (Good quality low ESR caps are dirt cheap.)  I have the proper desoldering tools for that kind of work and have done it many times.  I even have a good mobo test bench to do post-repair burn in.

[Gewehr98]

Kernel hardware support is broad enough that it shouldn't have a problem unlike a certain popular but problematic OS.

I swap Windows XP/Vista/7 HDs between systems all the time with no problems.  Maybe you're doing it wrong?

[Nick1911]

I swap Windows XP/Vista/7 HDs between systems all the time with no problems.  Maybe you're doing it wrong?

At least xp, I know for a fact, is compiled at setup time for a particular architecture.  Ive had to do "repair-install" too many times because of it way back when.

[mtnbkr]

At least xp, I know for a fact, is compiled at setup time for a particular architecture.  Ive had to do "repair-install" too many times because of it way back when.

Even Win95 would reconfigure itself if you swapped the drive into a different system.  I've done it before.  It wasn't as stable as my Win7 swap, but it worked well enough to let the user work for a few weeks until the new system came in.  Win7 does it so well, I haven't been compelled to reinstall Windows after taking my drive from my dead Intel-based Shuttle PC and stuffing it into my shiny new AMD-based system. 

At the moment, the only thing that'll get me to rebuild the system is an inexpensive SSD.

Chris

[zahc]

Just to further rule out a software problem I tried to install to a new hard drive, but it locked up in a similar fashion. This is what the screen does sometimes:

[Nick1911]

Even Win95 would reconfigure itself if you swapped the drive into a different system.  I've done it before.  It wasn't as stable as my Win7 swap, but it worked well enough to let the user work for a few weeks until the new system came in.  Win7 does it so well, I haven't been compelled to reinstall Windows after taking my drive from my dead Intel-based Shuttle PC and stuffing it into my shiny new AMD-based system. 

At the moment, the only thing that'll get me to rebuild the system is an inexpensive SSD.

Chris

Weird.  I never, not one time, got it to work without a repair install for win2k, win me or win xp.  Every single time, they would BSoD on boot after swapping a motherboard.

[GigaBuist]

I'm with Nick on this one.  XP was a pisser if you switched systems on it.  Hell, it was designed to lock up if too many things changed at one point.  Not sure if that remained into the end of XP and carried into Windows Vista and 7 or not.

At one point, when I was working for a jukebox company that ran XP on the jukes, we had one of the old embedded guys working on a customer bootloader that would swap the HAL.dll before booting based on which chipset was in the board.  We simply could not take hard drives running on an older mobo and switch them to the new one without it.

Not sure how that turned out.  I got canned a couple of weeks after the effort was started.

[GigaBuist]

Oh, and for zahc:  I had a PC do that before when hotplugging USB devices.  Darned thing would just crash if I yanked something out wrong.  I figured it was a short or something somewhere.  It was fairly old (5-6 years) when the problem surfaced.

[lee n. field]

I'm with Nick on this one.  XP was a pisser if you switched systems on it.  Hell, it was designed to lock up if too many things changed at one point.  Not sure if that remained into the end of XP and carried into Windows Vista and 7 or not.

Pretty much my experience.  Thus my comment above. 

Just to further rule out a software problem I tried to install to a new hard drive, but it locked up in a similar fashion. This is what the screen does sometimes:

So, sounds like it's hardware.

[Perd Hapley]

Just to further rule out a software problem I tried to install to a new hard drive, but it locked up in a similar fashion. This is what the screen does sometimes:

What is that, the orange and lavender screen of death?

[rcnixon]

Try the HD on another, similar system and if it's stable, the MOBO is probably bad.  Get a new MOBO, they're cheap enough.  With all due respect to the poster who offered board-level repairs, It's just not worth the time.  An extreme MOBO is about $150 - $170 without memory or a CPU.  A resonable one is $99 to $120.  Add memory and a CPU and you can get into a new, high-perf MOBO for $200 - $250.  I too have the tools and skills to do the job but my time is worth more than a bad board that may or may not continue to work.  My former company did board-level repairs but the boards were $1.5 million OC192 router interface cards.

Russ

[Perd Hapley]

Try the HD on another, similar system and if it's stable, the MOBO is probably bad.  Get a new MOBO, they're cheap enough.  With all due respect to the poster who offered board-level repairs, It's just not worth the time.  An extreme MOBO is about $150 - $170 without memory or a CPU.  A resonable one is $99 to $120.  Add memory and a CPU and you can get into a new, high-perf MOBO for $200 - $250.  I too have the tools and skills to do the job but my time is worth more than a bad board that may or may not continue to work.  My former company did board-level repairs but the boards were $1.5 million OC192 router interface cards.

Russ

Hey now. I like having Rocketman's offer on the table. It beats dropping hundreds of dollars on a new mobo, processor and RAM.

[Physics]

Only problems I've ever had that produced effects like that were GPU or heat related.