[zfs-discuss] Unlistable files

Vladimir Brik vladimir.brik at icecube.wisc.edu
Thu Apr 5 17:03:13 EDT 2018


> Are you using a ZVOL with a filesystem on top?
No.

> Can you share the output of "zpool status; zpool list; zfs list -t all"
> *after* issuing the copy which cause unlistable files?
Output below:

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 87h47m with 0 errors on Sat Mar 31 07:09:27
2018
config:

        NAME                        STATE     READ WRITE CKSUM
        tank                        ONLINE       0     0     0
          raidz1-0                  ONLINE       0     0     0
            wwn-0x5000c50085ac4c0f  ONLINE       0     0     0
            wwn-0x5000c50085acda77  ONLINE       0     0     0
            wwn-0x5000c500858db3d7  ONLINE       0     0     0
            wwn-0x5000c50085ac9887  ONLINE       0     0     0
            wwn-0x5000c50085aca6df  ONLINE       0     0     0
          raidz1-1                  ONLINE       0     0     0
            wwn-0x5000c500858db743  ONLINE       0     0     0
            wwn-0x5000c500858db347  ONLINE       0     0     0
            wwn-0x5000c500858db4a7  ONLINE       0     0     0
            wwn-0x5000c500858dbb0f  ONLINE       0     0     0
            wwn-0x5000c50085acaa97  ONLINE       0     0     0
          raidz1-2                  ONLINE       0     0     0
            wwn-0x5000c50085accb4b  ONLINE       0     0     0
            wwn-0x5000c50085acab9f  ONLINE       0     0     0
            wwn-0x5000c50085ace783  ONLINE       0     0     0
            wwn-0x5000c500858db67b  ONLINE       0     0     0
            wwn-0x5000c50085acb983  ONLINE       0     0     0
          raidz1-3                  ONLINE       0     0     0
            wwn-0x5000c50085ac4fd7  ONLINE       0     0     0
            wwn-0x5000c50085acb24b  ONLINE       0     0     0
            wwn-0x5000c50085ace13b  ONLINE       0     0     0
            wwn-0x5000c500858db43f  ONLINE       0     0     0
            wwn-0x5000c500858db61b  ONLINE       0     0     0
          raidz1-4                  ONLINE       0     0     0
            wwn-0x5000c500858dbbb7  ONLINE       0     0     0
            wwn-0x5000c50085acce7f  ONLINE       0     0     0
            wwn-0x5000c50085acd693  ONLINE       0     0     0
            wwn-0x5000c50085ac3d87  ONLINE       0     0     0
            wwn-0x5000c50085acc89b  ONLINE       0     0     0
          raidz1-5                  ONLINE       0     0     0
            wwn-0x5000c500858db28b  ONLINE       0     0     0
            wwn-0x5000c500858db68f  ONLINE       0     0     0
            wwn-0x5000c500858dbadf  ONLINE       0     0     0
            wwn-0x5000c500858db623  ONLINE       0     0     0
            wwn-0x5000c500858db48b  ONLINE       0     0     0
          raidz1-6                  ONLINE       0     0     0
            wwn-0x5000c500858db6ef  ONLINE       0     0     0
            wwn-0x5000c500858db39b  ONLINE       0     0     0
            wwn-0x5000c500858db47f  ONLINE       0     0     0
            wwn-0x5000c500858dbb23  ONLINE       0     0     0
            wwn-0x5000c500858db803  ONLINE       0     0     0
        logs
          zfs-slog                  ONLINE       0     0     0
        spares
          wwn-0x5000c500858db463    AVAIL

errors: No known data errors
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
tank   254T   168T  85.6T         -    30%    66%  1.00x  ONLINE  -
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank           134T  62.0T  18.7T  /mnt/tank
tank/jade      661G  62.0T   661G  /mnt/tank/jade
tank/simprod   115T  15.0T   115T  /mnt/tank/simprod



On 04/05/2018 03:50 PM, Gionatan Danti wrote:
> Il 05-04-2018 22:34 Vladimir Brik via zfs-discuss ha scritto:
>> Hello.
>>
>> I have run into a strange issue where files don't show up in directory
>> listing but can be accessed by path directly. I wonder if somebody knows
>> what might have caused this.
>>
>> # find dst/a/foo
>> dst/a/foo
>> (as expected)
>>
>> # find dst/a/ -name foo
>> (no output)
>>
>> # ls -l dst/a/foo
>> -rw-r--r-- 1 xxx xxx 5991051 Feb 22 13:35 dst/a/foo
>> (as expected)
>>
>> # ls -l dst/a/ | grep foo
>> (no output)
>>
>> # cp dst/a/foo bar
>> (this works; bar is created and can be listed)
>>
>> The problem occurs when I run something like "cp -r src dst", where src
>> is a directory with 12 sub-directories with 6999 files each, about 84K
>> files total, 2.9TB. After copy finishes, dst is missing several thousand
>> files according to find. (Similar thing happened when I tarred src and
>> then unpacked it in a different location; according to tar --list the
>> tarball contained all files.)
>>
>> The cp command reported "No space left on device" for a couple of files.
>> The filesystem has about 80TB free (zpool is about 50% full). The files
>> for which "No space left on device" error was generated just weren't
>> created, it seems, but other missing files are accessible by their full
>> path but did not show up in directory listings (as shown above).
>>
>> ls is reporting some sub-directories of dst have 7000 hard links instead
>> of 7001 that the sub-directories in src have. All missing files seem to
>> be from such sub-directories.
>>
>> After rebooting the server, the missing were no longer accessible by
>> full path.
>>
>> It seems the problem is reproducible. Missing files are not always the
>> same.
>>
>> I am running ZFS 0.7.7, Scientific Linux release 6.8. No ZFS snapshots.
>>
>> If anybody can shed light on this, I would really appreciate it :)
>>
> 
> Are you using a ZVOL with a filesystem on top?
> Can you share the output of "zpool status; zpool list; zfs list -t all"
> *after* issuing the copy which cause unlistable files?
> 
> Regards.
> 


More information about the zfs-discuss mailing list