[zfs-discuss] Kernel panic when operating on ZFS Datasets

Martin Ritchie ritchiem at apache.org
Tue Dec 12 23:30:00 EST 2017


Hey all,

I'm looking for a little help with an issue that I can't seem to identify
in the lists or issues.

I've just freshly installed a new zfs pool on a fresh Ubuntu 17.10 install.

I created a fresh zpool raidz2 with 6 x 6tb drives. I did first create it
with 5+a sparse 6tb file and then force it to be degraded to copy the data
off the final 6tb drive before then adding the original data drive into the
pool to replace the sparse file. Created with ashift=12. The data sets were
created with compression=lz4 and no atime at the root.

4.13.0-19-generic #22-Ubuntu SMP Mon Dec 4 11:58:07 UTC 2017 x86_64 x86_64
x86_64 GNU/Linux
SPL: Loaded module v0.6.5.11-1ubuntu1
ZFS: Loaded module v0.6.5.11-1ubuntu3, ZFS pool version 5000, ZFS
filesystem version 5

Everything was looking good.. I created a number of datasets:
ritchiem at earth:~$ sudo zfs list
NAME                USED  AVAIL  REFER  MOUNTPOINT
earth              7.92T  13.1T  1.55T  /earth
earth/devops       9.65M  13.1T  9.65M  /earth/devops
earth/docker       4.02G  13.1T   192K  /earth/docker
earth/docker/vm  4.02G  13.1T  4.02G  /earth/docker/vm
earth/docker/scripts     288K  13.1T   288K  /earth/docker/scripts
earth/vm        4.51T  13.1T  4.51T  /earth/vm
earth/vm-config         10.8G  13.1T  10.8G  /earth/vm-config
earth/remote       1.65T  13.1T  1.12T  /earth/remote
earth/remote/TM     542G  13.1T   542G  /earth/remote/TM
earth/tmp           206G  13.1T   206G  /earth/tmp

As I started to rsync data to their new resting datasets I had a number of
issues.

Data in the earth/tmp pool seemed to be lost. Data that was rsync appeared
to be replaced with a subsequent rsync. This one perhaps I ham fisted the
rsync but that was the first odd sign.

Now the real problem is that I can't do any filesystem operations on pools
earth, remote, remote/TM they all hang, processes blocked on IO a quick ps
grep:

root     10912  0.0  0.0   7536  2380 ?        D    06:25   0:09
/usr/bin/updatedb.mlocate
root     22771  0.0  0.0  22296  1324 ?        D    21:25   0:00 mount
ritchiem 23264  0.0  0.0  16448  1056 ?        D    22:02   0:00 ls
--color=auto /earth/remote
ritchiem 24032  0.0  0.0   7468   924 ?        D    22:40   0:00 df -h
root     24075  0.0  0.0  22296  1336 ?        D    22:40   0:00 mount
ritchiem 24264  0.0  0.0  22960  3800 ?        D    22:49   0:00 -bash
ritchiem 24311  0.0  0.0  22952  5744 ?        Ds   22:49   0:00 -bash


The only sign of what might be going wrong is a kernel panic stack trace in
dmesg:

Dec 11 21:35:40 earth kernel: [  140.258195] ------------[ cut here
]------------
Dec 11 21:35:40 earth kernel: [  140.258196] kernel BUG at
/build/linux-tt6jd0/linux-4.13.0/lib/string.c:985!
Dec 11 21:35:40 earth kernel: [  140.258240] invalid opcode: 0000 [#1] SMP
Dec 11 21:35:40 earth kernel: [  140.258264] Modules linked in: btusb btrtl
btbcm btintel zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O)
snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi intel_rapl
x86_pkg_temp_thermal snd_hda_intel intel_powerclamp ath3k snd_hda_codec
coretemp kvm_intel kvm bluetooth irqbypass eeepc_wmi asus_wmi snd_hda_core
intel_cstate sparse_keymap snd_hwdep intel_rapl_perf video wmi_bmof
ecdh_generic lpc_ich snd_pcm snd_timer snd mei_me soundcore mei shpchp
mac_hid ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs raid10 raid456
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq
libcrc32c raid1 raid0 multipath linear amdkfd amd_iommu_v2 crct10dif_pclmul
radeon crc32_pclmul i2c_algo_bit ghash_clmulni_intel
Dec 11 21:35:40 earth kernel: [  140.258673]  ttm pcbc e1000e mxm_wmi
drm_kms_helper firewire_ohci aesni_intel syscopyarea sysfillrect sysimgblt
aes_x86_64 firewire_core fb_sys_fops crypto_simd ahci ptp glue_helper
cryptd drm crc_itu_t libahci pps_core wmi
Dec 11 21:35:40 earth kernel: [  140.258790] CPU: 3 PID: 1965 Comm: zpool
Tainted: P           O    4.13.0-19-generic #22-Ubuntu
Dec 11 21:35:40 earth kernel: [  140.258836] Hardware name: System
manufacturer System Product Name/P8P67 PRO, BIOS 1502 03/02/2011
Dec 11 21:35:40 earth kernel: [  140.258884] task: ffff99a7993c5d00
task.stack: ffffb6dac2acc000
Dec 11 21:35:40 earth kernel: [  140.258923] RIP:
0010:fortify_panic+0x13/0x22
Dec 11 21:35:40 earth kernel: [  140.258948] RSP: 0018:ffffb6dac2acf8b0
EFLAGS: 00010282
Dec 11 21:35:40 earth kernel: [  140.258979] RAX: 0000000000000023 RBX:
ffffb6dac2acf9c0 RCX: 0000000000000000
Dec 11 21:35:40 earth kernel: [  140.259018] RDX: 0000000000000000 RSI:
ffff99a7af4cdc78 RDI: ffff99a7af4cdc78
Dec 11 21:35:40 earth kernel: [  140.259056] RBP: ffffb6dac2acf8b0 R08:
0000000000000000 R09: 00000000000003f7
Dec 11 21:35:40 earth kernel: [  140.259095] R10: 0000000000000000 R11:
00000000ffffffff R12: 0000000000000001
Dec 11 21:35:40 earth kernel: [  140.259134] R13: ffff99a783ad2ad0 R14:
0000000000000054 R15: ffff99a7992c80c0
Dec 11 21:35:40 earth kernel: [  140.259173] FS:  00007fd5d0fef780(0000)
GS:ffff99a7af4c0000(0000) knlGS:0000000000000000
Dec 11 21:35:40 earth kernel: [  140.259217] CS:  0010 DS: 0000 ES: 0000
CR0: 0000000080050033
Dec 11 21:35:40 earth kernel: [  140.259252] CR2: 00007f1c15f8c2e0 CR3:
0000000418c5e000 CR4: 00000000000406e0
Dec 11 21:35:40 earth kernel: [  140.259291] Call Trace:
Dec 11 21:35:40 earth kernel: [  140.259363]
zfs_acl_node_read.constprop.16+0x31a/0x320 [zfs]
Dec 11 21:35:40 earth kernel: [  140.259436]
zfs_zaccess_aces_check+0x96/0x330 [zfs]
Dec 11 21:35:40 earth kernel: [  140.259468]  ?
update_sd_lb_stats+0x137/0x5d0
Dec 11 21:35:40 earth kernel: [  140.259529]  zfs_zaccess_common+0xc3/0x160
[zfs]
Dec 11 21:35:40 earth kernel: [  140.259590]  zfs_zaccess+0xa7/0x290 [zfs]
Dec 11 21:35:40 earth kernel: [  140.259653]  zfs_lookup+0x1a0/0x320 [zfs]
Dec 11 21:35:40 earth kernel: [  140.259713]  zpl_lookup+0xc9/0x210 [zfs]
Dec 11 21:35:40 earth kernel: [  140.259739]  ? lookup_fast+0x115/0x310
Dec 11 21:35:40 earth kernel: [  140.259764]  ? dput.part.23+0x2d/0x1e0
Dec 11 21:35:40 earth kernel: [  140.259788]  lookup_slow+0xa5/0x160
Dec 11 21:35:40 earth kernel: [  140.259810]  walk_component+0x1bd/0x340
Dec 11 21:35:40 earth kernel: [  140.259834]  ? path_init+0x177/0x2f0
Dec 11 21:35:40 earth kernel: [  140.259856]  path_lookupat+0x84/0x1f0
Dec 11 21:35:40 earth kernel: [  140.259880]  ? call_rcu_sched+0x17/0x20
Dec 11 21:35:40 earth kernel: [  140.259904]  filename_lookup+0xb6/0x190
Dec 11 21:35:40 earth kernel: [  140.259929]  ? remove_wait_queue+0x4d/0x60
Dec 11 21:35:40 earth kernel: [  140.259956]  ?
__check_object_size+0xaf/0x1b0
Dec 11 21:35:40 earth kernel: [  140.259983]  ? strncpy_from_user+0x4d/0x170
Dec 11 21:35:40 earth kernel: [  140.260009]  user_path_at_empty+0x36/0x40
Dec 11 21:35:40 earth kernel: [  140.260033]  ? user_path_at_empty+0x36/0x40
Dec 11 21:35:40 earth kernel: [  140.260059]  vfs_statx+0x76/0xe0
Dec 11 21:35:40 earth kernel: [  140.260083]  SYSC_newlstat+0x3d/0x70
Dec 11 21:35:40 earth kernel: [  140.260108]  ? _do_fork+0x153/0x3f0
Dec 11 21:35:40 earth kernel: [  140.260132]  ? ____fput+0xe/0x10
Dec 11 21:35:40 earth kernel: [  140.260155]  SyS_newlstat+0xe/0x10
Dec 11 21:35:40 earth kernel: [  140.260178]
entry_SYSCALL_64_fastpath+0x1e/0xa9
Dec 11 21:35:40 earth kernel: [  140.260206] RIP: 0033:0x7fd5cf999895
Dec 11 21:35:40 earth kernel: [  140.260228] RSP: 002b:00007ffc1de75f88
EFLAGS: 00000246 ORIG_RAX: 0000000000000006
Dec 11 21:35:40 earth kernel: [  140.260270] RAX: ffffffffffffffda RBX:
0000564eba1797b0 RCX: 00007fd5cf999895
Dec 11 21:35:40 earth kernel: [  140.260308] RDX: 00007ffc1de75fe0 RSI:
00007ffc1de75fe0 RDI: 00007ffc1de76070
Dec 11 21:35:40 earth kernel: [  140.260347] RBP: 00007ffc1de77070 R08:
00007ffc1de75f18 R09: 0000000000000010
Dec 11 21:35:40 earth kernel: [  140.260386] R10: 0000000000000000 R11:
0000000000000246 R12: 00007ffc1de76070
Dec 11 21:35:40 earth kernel: [  140.260424] R13: 0000564eba16c2b0 R14:
0000564eba250ef0 R15: 0000000000000000
Dec 11 21:35:40 earth kernel: [  140.260464] Code: e0 4c 89 e2 e8 b1 59 00
00 42 c6 04 20 00 48 89 d8 5b 41 5c 5d c3 0f 0b 55 48 89 fe 48 c7 c7 38 74
16 99 48 89 e5 e8 50 a4 7e ff <0f> 0b 90 90 90 90 90 90 90 90 90 90 90 90
90 55 31 c9 48 89 fa
Dec 11 21:35:40 earth kernel: [  140.260598] RIP: fortify_panic+0x13/0x22
RSP: ffffb6dac2acf8b0
Dec 11 21:35:40 earth kernel: [  140.260632] ---[ end trace
d7010bf9362f1191 ]---


Any help or pointers would be greatly appreciated.

Regards
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.zfsonlinux.org/pipermail/zfs-discuss/attachments/20171212/f6f3c325/attachment.html>


More information about the zfs-discuss mailing list