[zfs-discuss] ZPOOL double import and host ownership question

Phil Harman phil.harman at gmail.com
Thu Feb 11 03:36:38 EST 2016


And if you just happen to clone your servers after /etc/hostid has been generated?

Building clusters from the ground up is expert business. ZFS won't protect you from yourself - e.g. it doesn't use any disk reservation or STONITH mechanisms. This is why commercial cluster solutions like RSF/1 use SCSI reservations and the "failfast" facility on illumos / solaris platforms.

Without a standard for hardware serial numbers on all platforms, hostid is always going to be an issue. The systems from the
company that invented ZFS and hostid, all had hardware serial numbers (of course).

Yes, the zpool manpage could be more explicit about the use and limitations of hostid, but ZFS doesn't claim to be cluster ready.

Anyone can build a cluster of sorts with shared storage, but not all clusters are equal. Just because you can, doesn't mean you should. I've been doing it for more than 25 years, and it still gives me the shivers ;)


On 11 Feb 2016, at 07:32, Veniamin Konoplev via zfs-discuss <zfs-discuss at list.zfsonlinux.org> wrote:

>> IMHO, since ZFS depends on the hostid for safety, the default behaviour should be if "zpool" is creating the first pool on a system to check for /etc/hostid, and if it isn't present it should create a new one with a random hostid, or alternately report an error to the user that says the /etc/hostid file should be created if it doesn't yet exist and add an option to override this.
> 
> Yes you are right, unfortunately current ZOL implementation is not safe, at least on RH/Centos 6, since two clean installed hosts will import a shared pool twice without special precautions.
> 
> It should be considered as a CRITICAL bug. Note that previous ZOL versions works correctly in similar cases. So, I a little bit frustrated.
> 
> 
> 
> IMHO, if "hostid" is not reliable method on Linux then random qookie should be used for pairing current host to zfs pool. This is similar to your "random hostid" approach but a little bit more correct since ZFS should not implicitly affect such fundamental objects as "hostid".
> 
> That is exactly I initially assumed when removed "zpool.cache" from the second host after cloning.
> 
> 
> 11.02.2016 0:32, Andreas Dilger wrote:
>>> On Feb 10, 2016, at 7:45 AM, Veniamin Konoplev via zfs-discuss <zfs-discuss at list.zfsonlinux.org> wrote:
>>> 
>>> Setting the /etc/hostid explicitly solved the problem.
>>> 
>>> Thank you very much.
>> 
>> But this is a danger to anyone that doesn't create an /etc/hostid file, see
>> https://github.com/zfsonlinux/zfs/issues/1429 for similar reports.
>> 
>> Since double-import of a zpool is an extremely dangerous, and likely fatally corrupting, action ZFS should be "safe by default" in this respect.  Even having /etc/hostid is not safe, since this will be cloned to the backup server if the backup server starts off as a copy of the original server.
>> 
>> IMHO, since ZFS depends on the hostid for safety, the default behaviour should be if "zpool" is creating the first pool on a system to check for /etc/hostid, and if it isn't present it should create a new one with a random hostid, or alternately report an error to the user that says the /etc/hostid file should be created if it doesn't yet exist and add an option to override this.
>> 
>> There is a more robust feature, Multi-modifier Protection (MMP), discussed at https://github.com/zfsonlinux/zfs/issues/745 and http://permalink.gmane.org/gmane.os.illumos.zfs/302 but this is only at the design stage and won't help current users.
>> 
>> Cheers, Andreas
>> 
>>> 10.02.2016 16:48, OviOS Linux пишет:
>>>> SPL is looking for a /etc/hostid file
>>>> this must be 4 bytes, otherwise it will ignore it
>>>> here, an example
>>>> 
>>>> [    1.411155] SPL: Loaded module v0.6.5.3-1
>>>> [    4.403032] SPL: Ignoring the /etc/hostid file because it is 0 bytes;
>>>> expecting 4 bytes instead.
>>>> [    4.403037] SPL: using hostid 0x00000000
>>>> 
>>>> 
>>>> On Wed, Feb 10, 2016 at 8:06 AM, Veniamin Konoplev <v.konoplev at cosmos.ru
>>>> <mailto:v.konoplev at cosmos.ru>> wrote:
>>>> 
>>>>    I am afraid hostid was deprecated:
>>>> 
>>>>    [root at nas1 ~]# dmesg | grep -i spl
>>>>    SPL: Loaded module v0.6.5.4-1
>>>>    SPL: using hostid 0x00000000
>>>> 
>>>> 
>>>> 
>>>>    10.02.2016 15:54, OviOS Linux пишет:
>>>> 
>>>>        Also, in dmesg you can see what hostid SPL is using. It's
>>>>        something like
>>>>        SPL using hostid xxxxxx
>>>> 
>>>>        On 10 Feb 2016 7:25 a.m., "Veniamin Konoplev"
>>>>        <v.konoplev at cosmos.ru <mailto:v.konoplev at cosmos.ru>
>>>>        <mailto:v.konoplev at cosmos.ru <mailto:v.konoplev at cosmos.ru>>> wrote:
>>>> 
>>>>             None, I supplied almost exact console output.
>>>>             zfs version is: zfs-0.6.5.4-1.el6.x86_64
>>>> 
>>>>             10.02.2016 15:19, OviOS Linux пишет:
>>>> 
>>>>                 After you import on node1, if you do zpool import does
>>>>        it warn
>>>>                 you the
>>>>                 pool was last imported on a different system?
>>>> 
>>>>                 On 10 Feb 2016 7:13 a.m., "Veniamin Konoplev"
>>>>                 <v.konoplev at cosmos.ru <mailto:v.konoplev at cosmos.ru>
>>>>        <mailto:v.konoplev at cosmos.ru <mailto:v.konoplev at cosmos.ru>>
>>>>                 <mailto:v.konoplev at cosmos.ru
>>>>        <mailto:v.konoplev at cosmos.ru> <mailto:v.konoplev at cosmos.ru
>>>>        <mailto:v.konoplev at cosmos.ru>>>> wrote:
>>>> 
>>>>                      It would be too simple :)
>>>>                      I did not use -f of course.
>>>> 
>>>>                      [root at nas1 ~]# zpool list
>>>>                      no pools available
>>>> 
>>>>                      [root at nas2 ~]# zpool list
>>>>                      no pools available
>>>> 
>>>>                      [root at nas1 ~]# zpool import test
>>>>                      [root at nas1 ~]# zpool list
>>>>                      NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP
>>>>        DEDUP
>>>>                 HEALTH
>>>>                      ALTROOT
>>>>                      test  7.25T   444K  7.25T         -     0%     0%
>>>>        1.00x
>>>>                 ONLINE  -
>>>> 
>>>>                      [root at nas2 ~]# zpool import test
>>>>                      [root at nas2 ~]# zpool list
>>>>                      NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP
>>>>        DEDUP
>>>>                 HEALTH
>>>>                      ALTROOT
>>>>                      test  7.25T   444K  7.25T         -     0%     0%
>>>>        1.00x
>>>>                 ONLINE  -
>>>> 
>>>>                      [root at nas1 ~]# hostid
>>>>                      0a0a0b03
>>>> 
>>>>                      [root at nas2 ~]# hostid
>>>>                      0a0a0c03
>>>> 
>>>>                      [root at nas1 ~]# cat /etc/issue
>>>>                      CentOS release 6.7 (Final)
>>>>                      Kernel \r on an \m
>>>> 
>>>> 
>>>>                      [root at nas1 ~]# uname  -r
>>>>                      2.6.32-573.el6.x86_64
>>>> 
>>>>                      [root at nas1 ~]# rpm -aq | grep zfs
>>>>                      zfs-release-1-4.el6.noarch
>>>>                      libzfs2-0.6.5.4-1.el6.x86_64
>>>>                      zfs-dkms-0.6.5.4-1.el6.noarch
>>>>                      zfs-0.6.5.4-1.el6.x86_64
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>                      Kind regards,
>>>>                      Veniamin Konoplev
>>>>                      Space Research Institute
>>>>                      Networking department
>>>> 
>>>>                      10.02.2016 14:41, OviOS Linux пишет:
>>>> 
>>>>                          Hey, you don't mention if you imported on the
>>>>        second
>>>>                 node using
>>>>                          '-f'.
>>>>                          If you did, that is expected.
>>>> 
>>>>                          On 10 Feb 2016 5:29 a.m., "Veniamin Konoplev via
>>>>                 zfs-discuss"
>>>>                          <zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>
>>>>                          <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>>
>>>>                          <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>
>>>>                          <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>>>> wrote:
>>>> 
>>>>                               Hi all,
>>>> 
>>>>                               I recently encountered a strange problem.
>>>>                               I have a zpool on a shared disk jbod. It is
>>>>                 connected to
>>>>                          two servers.
>>>>                               The second server is created by cloning
>>>>        the first.
>>>> 
>>>>                               I preliminary deleted zpool.cache on the
>>>>        second
>>>>                 server but
>>>>                               nevertheless I able to import zpool twice
>>>>                 simultaneously on
>>>>                          two servers.
>>>> 
>>>>                               Hostid's on the servers are different.
>>>> 
>>>>                               I propose that some important server
>>>>                 identification was
>>>>                          migrated
>>>>                               during the server cloning but can not
>>>>        find which one.
>>>> 
>>>>                               Can anybody help me: what is wrong? and how
>>>>                 current host
>>>>                          ownership
>>>>                               is designed in ZPOOL?
>>>> 
>>>>                               --
>>>>                               Kind regards,
>>>>                               Veniamin Konoplev
>>>>                               Space Research Institute
>>>>                               Networking department
>>>> 
>>>>        _______________________________________________
>>>>                               zfs-discuss mailing list
>>>>        zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>
>>>>                          <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>>
>>>>                          <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>
>>>>                          <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>
>>>>                 <mailto:zfs-discuss at list.zfsonlinux.org
>>>>        <mailto:zfs-discuss at list.zfsonlinux.org>>>>
>>>>        http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Ovi Chis
>>>> -------------------------
>>>> Storage Engineer
>>>> 
>>>> Netapp and VMware Certified
>>>> 
>>>> Dev @ OviOS Linux
>>>> www.ovios.org <http://www.ovios.org>
>>>> 
>>>> Some Linux / VMWare / Storage tutorials
>>>> 
>>>> linuksovi.blogspot.ca <http://linuksovi.blogspot.ca>
>>> _______________________________________________
>>> zfs-discuss mailing list
>>> zfs-discuss at list.zfsonlinux.org
>>> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss
>> 
>> 
>> Cheers, Andreas
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at list.zfsonlinux.org
> http://list.zfsonlinux.org/cgi-bin/mailman/listinfo/zfs-discuss


More information about the zfs-discuss mailing list