[PATCH v2 2/2] secretmem/gup: don't check if page is secretmem without reference