[zfs-discuss] Need help fixing... ???

UbuntuNewbie chessdoter at googlemail.com
Thu Oct 31 06:16:45 EDT 2013


Thank you, Durval, for assisting me with this one.

Am Donnerstag, 31. Oktober 2013 10:05:16 UTC+1 schrieb Durval Menezes:
>
> Hi UbuntuNewbie, 
>
> (...)
> From what I see below you have some kind of logical inconsistency in 
> filesystem metadata. I don't think there's ANY way rsync could be the 
> culprit (unless something really braindead happened, like you giving it the 
> /dev raw device as an argument to work with -- which I don't think you 
> would do). It's almost certainly hardware (most probable: bad memory, bad 
> controller, bad cabling, bad power supply, etc)  or something in the kernel 
> (buggy controller driver, etc), or perhaps ZFS (least probable of all).
>
3 years ago, there have been problems, that LOOKED like hardware related. I 
messed with my vendor a lot, only to find out 18 months later, that since 
an updated display driver from ATI was released, all problems disappeared. 
so for almost 2 years, all is well.

>
> What can you tell us regarding your hardware setup?
>
Well, what exactly do i need to include?
I am on a  Intel® Core™ i7-2600 CPU @ 3.40GHz × 8 with 16 GB RAM, 1 SSD 
(used as system + cache) + 3 disks (currently 2 * 2 TB + 1 * 1 TB) internal 
+ several external ones. All of them use ZFS.

>
> Looks mighty like metadata corruption to me. I've seen very similar things 
> in other filesystems being caused by hardware problems, but never in ZFS 
> (doesn't ZFS store 2 copies of all metadata exactly to make something like 
> that almost impossible?).
>
ok, makes sense to me too. so i will have to check it, as best as i can. 

> I would do a "tar", specifically excluding the directory containing the 
> messed-up files. 
>
I might be able to do that. But that would lose all the older snapshots 
(the backup history), right? 

>  Questions:
> my plan to destroy the filesystem and to replace it with an intact one 
> failed. How should i proceed instead to fix the current state?
>
> If I were in your shoes and I had time (ie, the issue is not critical 
> enough to demand correction *now*, and I could afford to stop using the 
> filesystem for anything in the mean time), I would *not* destroy the FS 
> before trying to understand what happened, specifically in regards to the 
> *cause*: it's my observation that simply ignoring those kinds of issues 
> almost always guarantee that they will come back to bite you in the rear...
>
I can afford keeping the filesystem by moving it to an area, where it will 
not be included in future backups, as soon as i can replace it (maybe even 
from backup - i would only lose the last 2 weeks of history). What i can 
not do is to investigate (because my understanding is not specific enough). 
So whats left is: move filesystem, recreate from backup, and hope... :-(

>  
>
>> my scripts (to create the OS backup & snapshots, and later back them up 
>> to external media) have been running since a long time. the latest change 
>> though was:
>> i included --inplace parameter to the rsync (as suggested in some other 
>> post on this list), and that did work fine for several weeks. What actions 
>> should i take to possibly prevent this issue from coming up again?
>>
>
> I don't think that "--inplace" could be responsible in any way for that 
> issue. 
>
This is useful to know. I came up with it only because it was the last 
change to the working scripts. But ofc, if hardware is involved - which i 
cannot exclude as of now - then i was looking at the wrong place.

>  
>
>> Any diagnosis suggested/recommended (please keep in mind, i am ONLY a 
>> user)
>>
>
> Some suggestions:
> 0) DO A BACKUP (see my suggestion above on skipping the problem area) if 
> you didn't already. Treat that filesystem as something very delicate that 
> could break completely at any moment.
>
I WILL replace it - as noted above. 

> 1) Do you log kernel messages? Look in the log file for anything strange 
> prior to the first time you noticed that error. 
>
7 days worth of syslog (in case that is what you were asking for) - quickly 
scanning through the crucial hours, i found this:

Oct 29 09:28:04 multi-os-host kernel: [  187.804043] Pid: 3850, comm: rsync 
Tainted: P        WC O 3.2.0-55-generic #85-Ubuntu
Oct 29 09:28:04 multi-os-host kernel: [  187.804045] Call Trace:
Oct 29 09:28:04 multi-os-host kernel: [  187.804051]  [<ffffffff810681df>] 
warn_slowpath_common+0x7f/0xc0
Oct 29 09:28:04 multi-os-host kernel: [  187.804055]  [<ffffffff8106823a>] 
warn_slowpath_null+0x1a/0x20
Oct 29 09:28:04 multi-os-host kernel: [  187.804058]  [<ffffffff81193fb6>] 
unlock_new_inode+0x76/0x80
Oct 29 09:28:04 multi-os-host kernel: [  187.804097]  [<ffffffffa01dbfd5>] 
zfs_znode_alloc+0x465/0x540 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804120]  [<ffffffffa014c569>] 
? dbuf_rele_and_unlock+0x169/0x210 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804141]  [<ffffffffa014c9bf>] 
? dmu_buf_rele+0x2f/0x40 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804146]  [<ffffffff8103ec59>] 
? default_spin_lock_flags+0x9/0x10
Oct 29 09:28:04 multi-os-host kernel: [  187.804169]  [<ffffffffa0154904>] 
? dmu_object_info_from_dnode+0x144/0x1b0 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804203]  [<ffffffffa01de1f8>] 
zfs_zget+0x168/0x200 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804236]  [<ffffffffa01b6481>] 
? zap_lookup_norm+0xd1/0x1c0 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804269]  [<ffffffffa01bd633>] 
zfs_dirent_lock+0x4c3/0x5d0 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804301]  [<ffffffffa01d68af>] 
zfs_create+0x15f/0x6d0 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804306]  [<ffffffff816611ce>] 
? _raw_spin_lock+0xe/0x20
Oct 29 09:28:04 multi-os-host kernel: [  187.804337]  [<ffffffffa01ed735>] 
zpl_create+0xa5/0x140 [zfs]
Oct 29 09:28:04 multi-os-host kernel: [  187.804342]  [<ffffffff81186db4>] 
vfs_create+0xb4/0x120
Oct 29 09:28:04 multi-os-host kernel: [  187.804346]  [<ffffffff81188989>] 
do_last+0x5c9/0x730
Oct 29 09:28:04 multi-os-host kernel: [  187.804349]  [<ffffffff81189e91>] 
path_openat+0xd1/0x3f0
Oct 29 09:28:04 multi-os-host kernel: [  187.804355]  [<ffffffff8152f6ad>] 
? sock_aio_read+0x2d/0x40
Oct 29 09:28:04 multi-os-host kernel: [  187.804359]  [<ffffffff8118a2d2>] 
do_filp_open+0x42/0xa0
Oct 29 09:28:04 multi-os-host kernel: [  187.804363]  [<ffffffff8131c751>] 
? strncpy_from_user+0x31/0x40
Oct 29 09:28:04 multi-os-host kernel: [  187.804366]  [<ffffffff8118561a>] 
? do_getname+0x10a/0x180
Oct 29 09:28:04 multi-os-host kernel: [  187.804370]  [<ffffffff816611ce>] 
? _raw_spin_lock+0xe/0x20
Oct 29 09:28:04 multi-os-host kernel: [  187.804374]  [<ffffffff811975f7>] 
? alloc_fd+0xf7/0x150
Oct 29 09:28:04 multi-os-host kernel: [  187.804378]  [<ffffffff81179848>] 
do_sys_open+0xf8/0x240
Oct 29 09:28:04 multi-os-host kernel: [  187.804382]  [<ffffffff811799b0>] 
sys_open+0x20/0x30
Oct 29 09:28:04 multi-os-host kernel: [  187.804386]  [<ffffffff81669802>] 
system_call_fastpath+0x16/0x1b
Oct 29 09:28:04 multi-os-host kernel: [  187.804389] ---[ end trace 
9a21758196ff2893 ]---

Interesting maybe: also the display driver in question complained by 50 
lines like so:
Oct 29 09:07:37 multi-os-host kernel: [95119.163535] 
[fglrx:firegl_apl_loadDatabase] *ERROR* APL: apl initialize fail.

2) If you don't what does "dmesg|tail -30"  shows when you do the "ls" 
> above and get those errors?
>
thx, i get this:
[ 3711.252959] Pid: 8766, comm: ls Tainted: P         C O 3.2.0-55-generic 
#85-Ubuntu
[ 3711.252961] Call Trace:
[ 3711.252965]  [<ffffffff810681df>] warn_slowpath_common+0x7f/0xc0
[ 3711.252967]  [<ffffffff8106823a>] warn_slowpath_null+0x1a/0x20
[ 3711.252969]  [<ffffffff81193fb6>] unlock_new_inode+0x76/0x80
[ 3711.253003]  [<ffffffffa01dbfd5>] zfs_znode_alloc+0x465/0x540 [zfs]
[ 3711.253018]  [<ffffffffa01e6f5d>] ? zio_wait+0x12d/0x1c0 [zfs]
[ 3711.253021]  [<ffffffff8165fcbd>] ? mutex_lock+0x1d/0x50
[ 3711.253023]  [<ffffffff8103ec59>] ? default_spin_lock_flags+0x9/0x10
[ 3711.253034]  [<ffffffffa0154904>] ? 
dmu_object_info_from_dnode+0x144/0x1b0 [zfs]
[ 3711.253049]  [<ffffffffa01de1f8>] zfs_zget+0x168/0x200 [zfs]
[ 3711.253064]  [<ffffffffa01b6481>] ? zap_lookup_norm+0xd1/0x1c0 [zfs]
[ 3711.253078]  [<ffffffffa01bd633>] zfs_dirent_lock+0x4c3/0x5d0 [zfs]
[ 3711.253091]  [<ffffffffa01bd7cb>] zfs_dirlook+0x8b/0x300 [zfs]
[ 3711.253105]  [<ffffffffa01ba32d>] ? zfs_zaccess+0x9d/0x430 [zfs]
[ 3711.253112]  [<ffffffffa008f39a>] ? kmem_alloc_debug+0x14a/0x3d0 [spl]
[ 3711.253126]  [<ffffffffa01d7221>] zfs_lookup+0x2e1/0x330 [zfs]
[ 3711.253139]  [<ffffffffa01ece98>] zpl_lookup+0x78/0xf0 [zfs]
[ 3711.253142]  [<ffffffff816611ce>] ? _raw_spin_lock+0xe/0x20
[ 3711.253145]  [<ffffffff811851f5>] d_alloc_and_lookup+0x45/0x90
[ 3711.253147]  [<ffffffff81192975>] ? d_lookup+0x35/0x60
[ 3711.253148]  [<ffffffff81187382>] do_lookup+0x202/0x310
[ 3711.253150]  [<ffffffff811890bc>] path_lookupat+0x11c/0x750
[ 3711.253152]  [<ffffffff816611ce>] ? _raw_spin_lock+0xe/0x20
[ 3711.253154]  [<ffffffff81197e1e>] ? vfsmount_lock_local_unlock+0x1e/0x30
[ 3711.253157]  [<ffffffff8131c6e7>] ? __strncpy_from_user+0x27/0x60
[ 3711.253159]  [<ffffffff81189721>] do_path_lookup+0x31/0xc0
[ 3711.253161]  [<ffffffff8118a229>] user_path_at_empty+0x59/0xa0
[ 3711.253162]  [<ffffffff811857d5>] ? putname+0x35/0x50
[ 3711.253164]  [<ffffffff8118a233>] ? user_path_at_empty+0x63/0xa0
[ 3711.253166]  [<ffffffff8118a281>] user_path_at+0x11/0x20
[ 3711.253168]  [<ffffffff8117f21a>] vfs_fstatat+0x3a/0x70
[ 3711.253170]  [<ffffffff816611ce>] ? _raw_spin_lock+0xe/0x20
[ 3711.253172]  [<ffffffff8117f26e>] vfs_lstat+0x1e/0x20
[ 3711.253173]  [<ffffffff8117f40a>] sys_newlstat+0x1a/0x40
[ 3711.253175]  [<ffffffff81199d0f>] ? mntput+0x1f/0x30
[ 3711.253177]  [<ffffffff81185102>] ? path_put+0x22/0x30
[ 3711.253179]  [<ffffffff8119e75b>] ? sys_lgetxattr+0x5b/0x70
[ 3711.253181]  [<ffffffff81669802>] system_call_fastpath+0x16/0x1b
[ 3711.253183] ---[ end trace 31c5bf1c13168200 ]---

> 3) What does "smartctl -a" on the underlying disks show?
>
Hm, 14K of output. all tests passed, but all disks show old_age all over 
the place, sometimes pre_fail

> 4) What does a "zpool status" report after a  "zpool scrub"?
>
  pool: RAID
 state: ONLINE
  scan: scrub repaired 0 in 5h0m with 0 errors on Wed Oct 30 08:11:52 2013
config:

    NAME                                                STATE     READ 
WRITE CKSUM
    RAID                                                ONLINE       0     
0     0
      raidz1-0                                          ONLINE       0     
0     0
        ata-WDC_WD20EARX-00PASB0_WD-WMAZA8018132-part1  ONLINE       0     
0     0
        ata-ST2000DL003-9VT166_5YD2S8T2-part5           ONLINE       0     
0     0
        ata-WDC_WD10EACS-22D6B0_WD-WCAU43256035         ONLINE       0     
0     0
    cache
      ata-Solidata_SSD_IDLX-YATOP-000000008-part2       ONLINE       0     
0     0

errors: No known data errors

>
> Whatever happens, please keep us posted.
>
> Cheers, and good luck, 
> -- 
>    Durval.
>

So on my todo list, i have: replace the filesystem (and keep the bogus 
one), try to check memory, watch out for drive failures (i have a 
replacement at hand) 

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss+unsubscribe at zfsonlinux.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20131031/9eb6cf6e/attachment.html>


More information about the zfs-discuss mailing list