Y2038 September 2017

y2038@lists.linaro.org

1 participants
2 discussions

[PATCH v3] generic/390: Add tests for inode timestamp policy

by Deepa Dinamani

The test helps to validate clamping and mount behaviors according to supported file system timestamp ranges. Note that the test can fail on 32-bit systems for a few file systems. This will be corrected when vfs is transitioned to use 64-bit timestamps. Signed-off-by: Deepa Dinamani <deepa.kernel(a)gmail.com> --- The branch of the kernel tree can be located at https://github.com/deepa-hub/vfs refs/heads/vfs_timestamp_policy The xfs_io patch to add utimes is at https://www.spinics.net/lists/linux-xfs/msg02952.html Changes since v2: * Refactored notrun handling * Updated comments Changes since v1: * Use xfs_io utimes command * Updated error handling * Reorganized code according to review comments common/rc | 48 +++++++++++++ tests/generic/390 | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++ tests/generic/390.out | 2 + tests/generic/group | 1 + 4 files changed, 243 insertions(+) create mode 100755 tests/generic/390 create mode 100644 tests/generic/390.out diff --git a/common/rc b/common/rc index e3b54ec..17f025e 100644 --- a/common/rc +++ b/common/rc @@ -1960,6 +1960,51 @@ _run_aiodio() return $status } +# this test requires y2038 sysfs switch and filesystem +# timestamp ranges support. +_require_y2038() +{ + local device=${1:-$TEST_DEV} + local sysfsdir=/proc/sys/fs/fs-timestamp-check-on + + if [ ! -e $sysfsdir ]; then + _notrun "no kernel support for y2038 sysfs switch" + fi + + local tsmin tsmax + read tsmin tsmax <<<$(_filesystem_timestamp_range $device) + if [ $tsmin -eq -1 -a $tsmax -eq -1 ]; then + _notrun "filesystem $FSTYP timestamp bounds are unknown" + fi +} + +_filesystem_timestamp_range() +{ + device=${1:-$TEST_DEV} + case $FSTYP in + ext4) + if [ $(dumpe2fs -h $device 2>/dev/null | grep "Inode size:" | cut -d: -f2) -gt 128 ]; then + echo "-2147483648 15032385535" + else + echo "-2147483648 2147483647" + fi + ;; + + xfs) + echo "-2147483648 2147483647" + ;; + jfs) + echo "0 4294967295" + ;; + f2fs) + echo "-2147483648 2147483647" + ;; + *) + echo "-1 -1" + ;; + esac +} + # indicate whether YP/NIS is active or not # _yp_active() @@ -2070,6 +2115,9 @@ _require_xfs_io_command() echo $testio | egrep -q "Inappropriate ioctl" && \ _notrun "xfs_io $command support is missing" ;; + "utimes" ) + testio=`$XFS_IO_PROG -f -c "utimes" 0 0 0 0 $testfile 2>&1` + ;; *) testio=`$XFS_IO_PROG -c "$command help" 2>&1` esac diff --git a/tests/generic/390 b/tests/generic/390 new file mode 100755 index 0000000..f68b931 --- /dev/null +++ b/tests/generic/390 @@ -0,0 +1,192 @@ +#! /bin/bash +# FS QA Test 390 +# +# Tests to verify policy for filesystem timestamps for +# supported ranges: +# 1. Verify filesystem rw mount according to sysctl +# timestamp_supported. +# 2. Verify timestamp clamping for timestamps beyond max +# timestamp supported. +# +# Exit status 1: either or both tests above fail. +# Exit status 0: both the above tests pass. +# +#----------------------------------------------------------------------- +# Copyright (c) 2016 Deepa Dinamani. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License as +# published by the Free Software Foundation. +# +# This program is distributed in the hope that it would be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write the Free Software Foundation, +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +#----------------------------------------------------------------------- +# + +seq=`basename $0` +seqres=$RESULT_DIR/$seq +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! +trap "exit \$status" 0 1 2 3 15 + +# Get standard environment, filters and checks. +. ./common/rc +. ./common/filter +. ./common/attr + +# remove previous $seqres.full before test +rm -f $seqres.full + +# Prerequisites for the test run. +_supported_fs generic +_supported_os Linux +_require_scratch +_require_xfs_io_command utimes + +# Compare file timestamps obtained from stat +# with a given timestamp. +check_stat() +{ + file=$1 + timestamp=$2 + + stat_timestamp=`stat -c"%X;%Y" $file` + + prev_timestamp="$timestamp;$timestamp" + if [ $prev_timestamp != $stat_timestamp ]; then + echo "$prev_timestamp != $stat_timestamp" | tee -a $seqres.full + fi +} + +run_test_individual() +{ + file=$1 + timestamp=$2 + update_time=$3 + + #check if the time needs update + if [ $update_time -eq 1 ]; then + echo "Updating file: $file to timestamp `date -d @$timestamp`" >> $seqres.full + $XFS_IO_PROG -f -c "utimes $timestamp 0 $timestamp 0" $file + if [ $? -ne 0 ]; then + echo "Failed to update times on $file" | tee -a $seqres.full + fi + fi + + tsclamp=$(($timestamp>$tsmax?$tsmax:$timestamp)) + echo "Checking file: $file Updated timestamp is `date -d @$tsclamp`" >> $seqres.full + check_stat $file $tsclamp +} + +run_test() +{ + update_time=$1 + + n=1 + + for TIME in "${TIMESTAMPS[@]}" + do + #Run the test + run_test_individual ${SCRATCH_MNT}/test_$n $TIME $update_time + + #update iterator + ((n++)) + done +} + +_scratch_mkfs &>> $seqres.full 2>&1 || _fail "mkfs failed" +_require_y2038 $SCRATCH_DEV + +read tsmin tsmax <<<$(_filesystem_timestamp_range $SCRATCH_DEV) +echo min supported timestamp $tsmin $(date --date=@$tsmin) >> $seqres.full +echo max supported timestamp $tsmax $(date --date=@$tsmax) >> $seqres.full + +# Test timestamps array + +declare -a TIMESTAMPS=( + $tsmin + 0 + $tsmax + $((tsmax+1)) + 4294967295 + 8589934591 + 34359738367 +) + +# Max timestamp is hardcoded to Mon Jan 18 19:14:07 PST 2038 +sys_tsmax=2147483647 +echo "max timestamp that needs to be supported by fs for rw mount is" \ + "$((sys_tsmax+1)) $(date --date=@$((sys_tsmax+1)))" >> $seqres.full + +read ts_check <<<$(cat /proc/sys/fs/fs-timestamp-check-on) + +_scratch_mount +result=$? + +if [ $ts_check -ne 0 ]; then + echo "sysctl filesystem timestamp check is on" >> $seqres.full + # check for mount failure if the minimum requirement for max timestamp + # supported is not met. + if [ $sys_tsmax -ge $tsmax ]; then + if [ $result -eq 0 ]; then + echo "mount test failed" | tee -a $seqres.full + exit + fi + else + if [ $result -ne 0 ]; then + echo "failed to mount $SCRATCH_DEV" | tee -a $seqres.full + exit + fi + fi +else + # if sysctl switch is off then mount should succeed always. + echo "sysctl filesystem timestamp check is off" >> $seqres.full + if [ $result -ne 0 ]; then + echo "failed to mount $SCRATCH_DEV and timestamp check is off" >> $seqres.full + exit + fi +fi + +# Begin test case 1 +echo "In memory timestamps update test start" >> $seqres.full + +# update time on the file +update_time=1 + +run_test $update_time + +echo "In memory timestamps update complete" >> $seqres.full + +echo "Unmounting and mounting scratch $SCRATCH_MNT" >> $seqres.full + +# unmount and remount $SCRATCH_DEV +_scratch_cycle_mount + +# Begin test case 2 + +n=1 + +# Do not update time on the file this time, just read from disk +update_time=0 + +echo "On disk timestamps update test start" >> $seqres.full + +# Re-run test +run_test $update_time + +echo "On disk timestamps update test complete" >> $seqres.full + +echo "y2038 inode timestamp tests completed successfully" + +# success, all done +status=0 +exit diff --git a/tests/generic/390.out b/tests/generic/390.out new file mode 100644 index 0000000..82bd4eb --- /dev/null +++ b/tests/generic/390.out @@ -0,0 +1,2 @@ +QA output created by 390 +y2038 inode timestamp tests completed successfully diff --git a/tests/generic/group b/tests/generic/group index 08007d7..d137d01 100644 --- a/tests/generic/group +++ b/tests/generic/group @@ -392,3 +392,4 @@ 387 auto clone 388 auto log metadata 389 auto quick acl +390 auto quick rw -- 2.7.4

4 years, 11 months

FWD: Summary of the 2038 BoF at DebConf 17

by Steve McIntyre

Hi folks, As discussed with Arnd, I ran a BoF session about Y2038 at DebConf last month, giving an overview of the problem and starting some discussion. Here's the write-up, and there's a link to the video of the session below. ----- Forwarded message from Steve McIntyre <steve(a)einval.com> ----- From: Steve McIntyre <steve(a)einval.com> To: debian-devel(a)lists.debian.org Date: Sat, 2 Sep 2017 00:58:54 +0100 Subject: Summary of the 2038 BoF at DC17 Hi folks, As promised, here's a quick summary of what was discussed at the 2038 BoF session I ran in Montréal. Thanks to the awesome efforts of our video team, the session is already online [1]. I've taken a copy of the Gobby notes too, alongside my small set of slides for the session. [2] We had a conversation about the coming End of The World, the 2038 problem. What's the problem? ------------------- UNIX time_t is 31 bits (signed), counting seconds since Jan 1, 1970. It's going to wrap. It's used *everywhere* in UNIX-based systems. Imagine the effects of Y2K, but worse. What could go wrong? -------------------- All kinds if disasters! We're not trying to exaggerate this *too* much, but it's likely to be a significat problem. For most of the things that needed fixing for Y2K, they were on big obvious computers, typically mainframes. Y2K was solved by people doing a lot of work; to many outside of the direct work, it almost came to be an anti-climax. In 20 years' time, the systems that we will be trying to fix for the 2038 problem are likely to be much harder to track. They're typically not even going to look like computers - look at the IoT devices available today, and extrapolate. Imagine all kinds of devices with embedded computers that we won't know how to talk to, let alone verify their software. When does it happen? -------------------- Pick the example from the date(1) man page: $ date --date=@$((2**31-1)) Tue 19 Jan 03:14:07 GMT 2038 At that point, lots of software will believe it's suddenly 1902... What needs doing? ----------------- Lots of fixes are going to be needed all the way up the stack. Data formats are often not 2038-safe. The filesystems in use today are typically not ready. Modern ext4 *is*, using 34 bits for seconds and 30 bits for nanoseconds. btrfs uses a 64-bit second counter. But data on older filesystems will need to be migrated. There are many places in the Linux kernel where 32-bit time_t is used. This is being worked on, and so are the interfaces that expose 32-bit time_t. Lots of libraries use 32-bit time_t, even in places where it might not be obvious. Finally, applications will need fixing. Linux kernel ------------ There's project underway to fix Linux time-handling, led by Deepa Dinamani and Arnd Bergmann. There's a web site describing the efforts at https://kernelnewbies.org/y2038 and a mailing list at y2038(a)lists.linaro.org. They are using the y2038 project as a good way to get new developers involved in the Linux kernel, and those people are working on fixing things in a number of areas: adding core 64-bit time support, fixing drivers, and adding new versions of interfaces (syscalls, ioctls). We can't just replace all the existing interfaces with 64-bit versions, of course - we need to continue suporting the existing interfaces for existing code. There are lots of tasks here where people can join in and help. Glibc ----- Glibc is the next obvious piece of the puzzle - almost everything depends on it. Planning is ongoing at https://sourceware.org/glibc/wiki/Y2038ProofnessDesign to provide 64-bit time_t support without breaking the existing 32-bit code. There's more coverage in LWN at https://lwn.net/Articles/664800/ The glibc developers are obviously going to need the kernel to provide 64-bit interfaces to make much progress. Again, there's lot of work to be done here and help will be welcome. Elsewhere? ---------- If you're working further up the stack, it's hard to make many fixes when the lower levels are not yet done. Kernels other than Linux are also going to have the same problems to solve - not really looked at them in much detail. As the old time_t interfaces are POSIX-specified, hopefully we'll get equivalent new 64-bit interfaces that will be standard across systems. Massive numbers of libraries are going to need updates, possibly more than people realise. Anything embedding a time_t will obviously need changing. However, many more structures will embed a timeval or timespec and they're also broken. Almost anything that embeds libc-exposed timing functions will need updating. We're going to need mass rebuilds to find things that break with new interfaces, and to ensure that old interfaces still work. An obvious thing to do here also is automated scanning for ABI compliance as things change. Things to do now ---------------- Firstly: developers trying to be *too* clever are likely to only make things worse - don't do it! Whatever you do in your code, don't bodge around the 32-bit time_t problem. *Don't* store time values in weird formats, and don't assume things about it to "avoid" porting problems. These are all going to cause pain in the future as we try to fix problems. For the time being in your code, *use* time_t and expect an ABI break down the road. This is the best plan *for now*. In terms of things like on-disk data structures, don't try to second-guess future interfaces by simply adding padding space for a 64-bit time_t or equivalent. The final solution for time handling may not be what you expect and you might just make things worse. Dive in and help the kernel and glibc folks if you can! Next, check your code and the dependencies of your code to see if there are any bodges or breakages that you *can* fix now. Discussion ---------- It's a shame to spoil the future pensions of people by trying to fix this problem early! :-) There are various license checkers already in use today - could the same technology help finding time junk? Similarly, are any of the static analysis tools likely to help. It's believed that Coverity (for example) may be looking into the static analysis component of this. There's plenty of scope for developers to help work in these areas too. Do we need to worry about *unsigned* time_t usage too (good to 2106)? As an example, OpenPGP packets use that. This gives us a little bit longer, but will still need to be considered. The main point to consider is fixing things *properly*, don't just hack around things by moving the epoch or something similarly short-term. It's important that we work on fixing issues *now* to stop people building broken things that will bite us. We all expect that our own computer systems will be fine by 2038; Debian systems will be fixed and working! We'll have rebuilt the world with new interfaces and found the issues. The issues are going to be in the IoT, with systems that we won't be able to simply rebuild/verify/test - they'll fail. We need to get the underlying systems right ASAP for those systems. 2038 is the problem we're looking at now, but we're going to start seeing issues well before then - think repeating calendar entries. Libraries often don't need to expose any time_t style information, but it's something to be careful about. If people have worked things out well, changing the internal implementation of a delay function should not pollute up the stack. But it's easy to pick up changes without realising - think about select() in the event loop, for example. Statically linked things (e.g. the Go ecosystem) are likely to bite - we need to make sure that the libraries that they embed are fixed early, before we can rebuild that stack upwards. How can we enforec the ability to upgrade and get support for IoT products so that they don't just brick themselves in future? GPL violations play into this because the sources are unavailable - i.e., no way to rebuild and upgrade. Ancient vendor kernels are a major PITA, and only make things more urgent. If you're designing your own data format without reference to current or upcoming standards, then of course consider the need for better time handling. Conversions will be needed anyway. Main takeaways: * This is a real problem * We need to fix the problem *early* * People are working on this already, and there's plenty of tasks to help with [1] http://meetings-archive.debian.net/pub/debian-meetings/2017/debconf17/it-s-… [2] https://www.einval.com/~steve/talks/Debconf17-eotw-2038/ -- Steve McIntyre, Cambridge, UK. steve(a)einval.com "...In the UNIX world, people tend to interpret `non-technical user' as meaning someone who's only ever written one device driver." -- Daniel Pead ----- End forwarded message ----- -- Steve McIntyre, Cambridge, UK. steve(a)einval.com Getting a SCSI chain working is perfectly simple if you remember that there must be exactly three terminations: one on one end of the cable, one on the far end, and the goat, terminated over the SCSI chain with a silver-handled knife whilst burning *black* candles. --- Anthony DeBoer

6 years, 7 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Y2038 September 2017