The import was working around the fact "tuple[T]" was used instead of
typing.Tuple[T].
Convert it to use typing.Tuple to be consistent with how the rest of the
code is annotated.
Signed-off-by: Daniel Latypov <dlatypov(a)google.com>
Reviewed-by: David Gow <davidgow(a)google.com>
Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com>
Tested-by: Brendan Higgins <brendanhiggins(a)google.com>
---
v1 -> v2: fix typos in commit message.
---
tools/testing/kunit/kunit_kernel.py | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index 90bc007f1f93..2c6f916ccbaf 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -6,15 +6,13 @@
# Author: Felix Guo <felixguoxiuping(a)gmail.com>
# Author: Brendan Higgins <brendanhiggins(a)google.com>
-from __future__ import annotations
import importlib.util
import logging
import subprocess
import os
import shutil
import signal
-from typing import Iterator
-from typing import Optional
+from typing import Iterator, Optional, Tuple
from contextlib import ExitStack
@@ -208,7 +206,7 @@ def get_source_tree_ops(arch: str, cross_compile: Optional[str]) -> LinuxSourceT
raise ConfigError(arch + ' is not a valid arch')
def get_source_tree_ops_from_qemu_config(config_path: str,
- cross_compile: Optional[str]) -> tuple[
+ cross_compile: Optional[str]) -> Tuple[
str, LinuxSourceTreeOperations]:
# The module name/path has very little to do with where the actual file
# exists (I learned this through experimentation and could not find it
base-commit: 1d71307a6f94df3750f8f884545a769e227172fe
--
2.32.0.93.g670b81a890-goog
And thus avoid a Python stacktrace:
~/linux/tools/testing/selftests/net$ ./devlink_port_split.py
Traceback (most recent call last):
File "/home/linux/tools/testing/selftests/net/./devlink_port_split.py",
line 277, in <module> main()
File "/home/linux/tools/testing/selftests/net/./devlink_port_split.py",
line 242, in main
dev = list(devs.keys())[0]
IndexError: list index out of range
Signed-off-by: Paolo Pisati <paolo.pisati(a)canonical.com>
---
tools/testing/selftests/net/devlink_port_split.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tools/testing/selftests/net/devlink_port_split.py b/tools/testing/selftests/net/devlink_port_split.py
index 834066d465fc..d162915311fd 100755
--- a/tools/testing/selftests/net/devlink_port_split.py
+++ b/tools/testing/selftests/net/devlink_port_split.py
@@ -239,6 +239,9 @@ def main(cmdline=None):
assert stderr == ""
devs = json.loads(stdout)['dev']
+ if len(devs.keys()) == 0:
+ print("no devlink device found")
+ sys.exit(1)
dev = list(devs.keys())[0]
cmd = "devlink dev show %s" % dev
--
2.30.2
Commit b6d5799b0b58 ("kunit: Add 'kunit_shutdown' option") changes KUnit
to call kernel_halt() by default when done testing.
This fixes the issue with not having .gcda files due to not calling
atexit() handlers, and therefore we can stop recommending people
manually tweak UML code.
The need to use older versions of GCC (<=6) remains however, due to
linktime issues, same as before. Note: There also might still be issues
with .gcda files as well in newer versions.
Signed-off-by: Daniel Latypov <dlatypov(a)google.com>
---
Documentation/dev-tools/kunit/running_tips.rst | 14 +-------------
1 file changed, 1 insertion(+), 13 deletions(-)
diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
index 7d99386cf94a..d1626d548fa5 100644
--- a/Documentation/dev-tools/kunit/running_tips.rst
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -86,19 +86,7 @@ Generating code coverage reports under UML
.. note::
TODO(brendanhiggins(a)google.com): There are various issues with UML and
versions of gcc 7 and up. You're likely to run into missing ``.gcda``
- files or compile errors. We know one `faulty GCC commit
- <https://github.com/gcc-mirror/gcc/commit/8c9434c2f9358b8b8bad2c1990edf10a21…>`_
- but not how we'd go about getting this fixed. The compile errors still
- need some investigation.
-
-.. note::
- TODO(brendanhiggins(a)google.com): for recent versions of Linux
- (5.10-5.12, maybe earlier), there's a bug with gcov counters not being
- flushed in UML. This translates to very low (<1%) reported coverage. This is
- related to the above issue and can be worked around by replacing the
- one call to ``uml_abort()`` (it's in ``os_dump_core()``) with a plain
- ``exit()``.
-
+ files or compile errors.
This is different from the "normal" way of getting coverage information that is
documented in Documentation/dev-tools/gcov.rst.
base-commit: 87c9c16317882dd6dbbc07e349bc3223e14f3244
--
2.32.0.93.g670b81a890-goog
The import was working around the fact "tuple[T]" was used instead of
typing.Tuple[T].
Convert it to use type.Tuple to be consistent with how the rest of the
code is anotated.
Signed-off-by: Daniel Latypov <dlatypov(a)google.com>
---
tools/testing/kunit/kunit_kernel.py | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index e1951fa60027..5987d5b1b874 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -6,15 +6,13 @@
# Author: Felix Guo <felixguoxiuping(a)gmail.com>
# Author: Brendan Higgins <brendanhiggins(a)google.com>
-from __future__ import annotations
import importlib.util
import logging
import subprocess
import os
import shutil
import signal
-from typing import Iterator
-from typing import Optional
+from typing import Iterator, Optional, Tuple
from contextlib import ExitStack
@@ -208,7 +206,7 @@ def get_source_tree_ops(arch: str, cross_compile: Optional[str]) -> LinuxSourceT
raise ConfigError(arch + ' is not a valid arch')
def get_source_tree_ops_from_qemu_config(config_path: str,
- cross_compile: Optional[str]) -> tuple[
+ cross_compile: Optional[str]) -> Tuple[
str, LinuxSourceTreeOperations]:
# The module name/path has very little to do with where the actual file
# exists (I learned this through experimentation and could not find it
base-commit: 87c9c16317882dd6dbbc07e349bc3223e14f3244
--
2.32.0.93.g670b81a890-goog
Introduction
Huawei Digest Lists is a simple in kernel database for storing file and
metadata digests and for reporting to its users (e.g. Integrity
Measurement Architecture or IMA) whether the digests are in the database
or not. The choice of placing it in the kernel and not in a user space
process is explained later in the Security Assumptions section.
The database is populated by directly uploading the so called digest
lists, a set of digests concatenated together and preceded by a header
including information about them (e.g. whether the file or metadata with
a given digest is immutable or not). Digest lists are stored in the
kernel memory as they are, the kernel just builds indexes to easily
lookup digests.
The kernel supports only one digest list format called compact. However,
alternative formats (e.g. the RPM header) can be supported through a
user space parser that, by invoking the appropriate library (more can be
added), converts the original digest list to the compact format and
uploads it to the kernel.
The database keeps track of whether digest lists have been processed in
some way (e.g. measured or appraised by IMA). This is important for
example for remote attestation, so that remote verifiers understand what
has been loaded in the database.
It is a transactional database, i.e. it has the ability to roll back to
the beginning of the transaction if an error occurred during the
addition of a digest list (the deletion operation always succeeds). This
capability has been tested with an ad-hoc fault injection mechanism
capable of simulating failures during the operations.
Finally, the database exposes to user space, through securityfs, the
digest lists currently loaded, the number of digests added, a query
interface and an interface to set digest list labels.
Binary Integrity
Integrity is a fundamental security property in information systems.
Integrity could be described as the condition in which a generic
component is just after it has been released by the entity that created
it.
One way to check whether a component is in this condition (called binary
integrity) is to calculate its digest and to compare it with a reference
value (i.e. the digest calculated in controlled conditions, when the
component is released).
IMA, a software part of the integrity subsystem, can perform such
evaluation and execute different actions:
- store the digest in an integrity-protected measurement list, so that
it can be sent to a remote verifier for analysis;
- compare the calculated digest with a reference value (usually
protected with a signature) and deny operations if the file is found
corrupted;
- store the digest in the system log.
Contribution
Huawei Digest Lists facilitates the provisioning of reference values for
system and application files from software vendors, to the kernel.
Possible sources for digest lists are:
- RPM headers;
- Debian repository metadata.
These sources are already usable without any modification required for
Linux vendors.
If digest list sources are signed (usually they are, like the ones
above), remote verifiers could identify their provenance, or Linux
vendors could prevent the loading of unsigned ones or those signed with
an untrusted key.
Possible Usages
Provisioned reference values can be used (e.g. by IMA) to make
integrity-related decisions (allow list or deny list).
Possible usages for IMA are:
- avoid recording measurement of files whose digest is found in the
pre-provisioned reference values:
- reduces the number of TPM operations (PCR extend);
- could make a TPM PCR predictable, as the PCR would not be
affected by the temporal sequence of executions if binaries are
known (no measurement);
- exclusively grant access to files whose digest is found in the
pre-provisioned reference values:
- faster verification time (fewer signature verifications);
- no need to generate a signature for every file.
Security Assumptions
Since digest lists are stored in the kernel memory, they are no
susceptible to attacks by user space processes.
A locked-down kernel, that accepts only verified kernel modules, will
allow digest lists to be added or deleted only though a well-defined and
monitored interface. In this situation, the root user is assumed to be
untrusted, i.e. it cannot subvert without being detected the mandatory
policy stating which files are accessible by the system.
Adoption
A former version of Huawei Digest Lists is used in the following OSes:
- openEuler 20.09
https://github.com/openeuler-mirror/kernel/tree/openEuler-20.09
- openEuler 21.03
https://github.com/openeuler-mirror/kernel/tree/openEuler-21.03
Originally, Huawei Digest Lists was part of IMA. In this version, it has
been redesigned as a standalone module with an API that makes its
functionality accessible by IMA and, eventually, other subsystems.
User Space Support
Digest lists can be generated and managed with digest-list-tools:
https://github.com/openeuler-mirror/digest-list-tools
It includes two main applications:
- gen_digest_lists: generates digest lists from files in the
filesystem or from the RPM database (more digest list sources can be
supported);
- manage_digest_lists: converts and uploads digest lists to the
kernel.
Simple Usage Example
1. Digest list generation (RPM headers and their signature are copied
to the specified directory):
# mkdir /etc/digest_lists
# gen_digest_lists -t file -f rpm+db -d /etc/digest_lists -o add
2. Digest list upload with the user space parser:
# manage_digest_lists -p add-digest -d /etc/digest_lists
3. First digest list query:
# echo sha256-$(sha256sum /bin/cat) > \
/sys/kernel/security/integrity/digest_lists/digest_query
# cat /sys/kernel/security/integrity/digest_lists/digest_query
sha256-[...]-0-file_list-rpm-coreutils-8.32-18.fc33.x86_64
(actions: 0): version: 1, algo: sha256, type: 2, modifiers: 1,
count: 106, datalen: 3392
4. Second digest list query:
# echo sha256-$(sha256sum /bin/zip) > \
/sys/kernel/security/integrity/digest_lists/digest_query
# cat /sys/kernel/security/integrity/digest_lists/digest_query
sha256-[...]-0-file_list-rpm-zip-3.0-27.fc33.x86_64 (actions: 0):
version: 1, algo: sha256, type: 2, modifiers: 1, count: 4,
datalen: 128
Architecture
This section introduces the high level architecture.
5. add/delete from hash table and add refs to digest list
+---------------------------------------------+
| +-----+ +-------------+ +--+
| | key |-->| digest refs |-->...-->| |
V +-----+ +-------------+ +--+
+-------------+ +-----+ +-------------+
| digest list | | key |-->| digest refs |
| (compact) | +-----+ +-------------+
+-------------+ +-----+ +-------------+
^ 4. copy to | key |-->| digest refs |
| kernel memory +-----+ +-------------+ kernel space
--------------------------------------------------------------------------
^ ^ user space
|<----------------+ 3b. upload |
+-------------+ +------------+ | 6. query digest
| digest list | | user space | 2b. convert
| (compact) | | parser |
+-------------+ +------------+
1a. upload ^ 1b. read
|
+------------+
| RPM header |
+------------+
As mentioned before, digest lists can be uploaded directly if they are
in the compact format (step 1a) or can be uploaded indirectly by the
user space parser if they are in an alternative format (steps 1b-3b).
During upload, the kernel makes a copy of the digest list to the kernel
memory (step 4), and creates the necessary structures to index the
digests (hash table and an array of digest list references to locate the
digests in the digest list) (step 5).
Finally, digests can be searched from user space through a securityfs
file (step 6) or by the kernel itself.
Implementation
This section describes the implementation of Huawei Digest Lists.
Basic Definitions
This section introduces the basic definitions required to use digest
lists.
Compact Digest List Format
Compact digest lists consist of one or multiple headers defined as:
struct compact_list_hdr {
__u8 version;
__u8 _reserved;
__le16 type;
__le16 modifiers;
__le16 algo;
__le32 count;
__le32 datalen;
} __packed;
which characterize the subsequent block of concatenated digests.
The algo field specifies the algorithm used to calculate the digest.
The count field specifies how many digests are stored in the subsequent
block of digests.
The datalen field specifies the length of the subsequent block of
digests (it is redundant, it is the same as
hash_digest_size[algo] * count).
Compact Types
Digests can be of different types:
- COMPACT_PARSER: digests of executables which are given the ability
to parse digest lists not in the compact format and to upload to the
kernel the digest list converted to the compact format;
- COMPACT_FILE: digests of regular files;
- COMPACT_METADATA: digests of file metadata (e.g. the digest
calculated by EVM to verify a portable signature);
- COMPACT_DIGEST_LIST: digests of digest lists (only used internally
by the kernel).
Different users of Huawei Digest Lists might query digests with
different compact types. For example, IMA would be interested in
COMPACT_FILE, as it deals with regular files, while EVM would be
interested in COMPACT_METADATA, as it verifies file metadata.
Compact Modifiers
Digests can also have specific attributes called modifiers (bit
position):
- COMPACT_MOD_IMMUTABLE: file content or metadata should not be
modifiable.
IMA might use this information to deny open for writing, or EVM to deny
setxattr operations.
Actions
This section defines a set of possible actions that have been executed
on the digest lists (bit position):
- COMPACT_ACTION_IMA_MEASURED: the digest list has been measured by
IMA;
- COMPACT_ACTION_IMA_APPRAISED: the digest list has been successfully
appraised by IMA;
- COMPACT_ACTION_IMA_APPRAISED_DIGSIG: the digest list has been
successfully appraised by IMA by verifying a digital signature.
This information might help users of Huawei Digest Lists to decide
whether to use the result of a queried digest.
For example, if a digest belongs to a digest list that was not measured
before, IMA should ignore the result of the query as the measurement
list sent to remote verifiers would lack how the database was populated.
Compact Digest List Example
version: 1, type: 2, modifiers: 0 algo: 4, count: 3, datalen: 96
<SHA256 digest1><SHA256 digest2><SHA256 digest3>
version: 1, type: 3, modifiers: 1 algo: 6, count: 2, datalen: 128
<SHA512 digest1><SHA512 digest2>
This digest list consists of two blocks. The first block contains three
SHA256 digests of regular files. The second block contains two SHA512
digests of immutable metadata.
Compact Digest List Operations
Finally, this section defines the possible operations that can be
performed with digest lists:
- DIGEST_LIST_ADD: the digest list is being added;
- DIGEST_LIST_DEL: the digest list is being deleted.
Objects
This section defines the objects to manage digest lists:
- digest_list_item: represents a digest list;
- digest_list_item_ref: represents a reference to a digest list, i.e.
the location at which a digest within a digest list can be accessed;
- digest_item: represents a unique digest.
They are represented in the following class diagram:
digest_offset,-----------+
hdr_offset |
|
+------------------+ | +----------------------+
| digest_list_item |--- N:1 ---| digest_list_item_ref |
+------------------+ +----------------------+
|
1:N
|
+-------------+
| digest_item |
+-------------+
A digest_list_item is associated to one or multiple
digest_list_item_ref, one for each digest it contains. However, a
digest_list_item_ref is associated to only one digest_list_item, as it
represents a single location within a specific digest list.
Given that a digest_list_item_ref represents a single location, it is
associated to only one digest_item. However, a digest_item can have
multiple references (as it might appears multiple times within the same
digest list or in different digest lists, if it is duplicated).
A digest_list_item is defined as:
struct digest_list_item {
loff_t size;
u8 *buf;
u8 actions;
u8 digest[64];
enum hash_algo algo;
const char *label;
};
- size: size of the digest list buffer;
- buf: digest list buffer;
- actions: actions performed on the digest list;
- digest: digest of the digest list;
- algo: digest algorithm;
- label: label used to identify the digest list (e.g. file name).
A digest_list_item_ref is defined as:
struct digest_list_item_ref {
struct digest_list_item *digest_list;
loff_t digest_offset;
loff_t hdr_offset;
};
- digest_list: pointer to a digest_list_item structure;
- digest_offset: offset of the digest related to the digest list
buffer;
- hdr_offset: offset of the header of the digest block containing the
digest.
A digest_item is defined as:
struct digest_item {
struct hlist_node hnext;
struct digest_list_item_ref *refs;
};
- hnext: pointers of the hash table;
- refs: array of digest_list_item_ref structures including a
terminator (protected by RCU).
All digest list references are stored for a given digest, so that a
query result can include the OR of the modifiers and actions of each
referenced digest list.
The relationship between the described objects can be graphically
represented as:
Hash table +-------------+ +-------------+
PARSER +-----+ | digest_item | | digest_item |
FILE | key |-->| |-->...-->| |
METADATA +-----+ |ref0|...|refN| |ref0|...|refN|
+-------------+ +-------------+
ref0: | | refN:
digest_offset | +-----------------------------+ digest_offset
hdr_offset | | hdr_offset
V V
+--------------------+
| digest_list_item |
| |
| size, buf, actions |
+--------------------+
^
|
Hash table +-------------+ +-------------+
DIGEST_LIST +-----+ |ref0 | |ref0 |
| key |-->| |-->...-->| |
+-----+ | digest_item | | digest_item |
+-------------+ +-------------+
The reference for the digest of the digest list differs from the
references for the other digest types. digest_offset and hdr_offset are
set to zero, so that the digest of the digest list is retrieved from the
digest_list_item structure directly (see get_digest() below).
Finally, this section defines useful helpers to access a digest or the
header the digest belongs to. For example:
static inline struct compact_list_hdr *get_hdr(
struct digest_list_item *digest_list,
loff_t hdr_offset)
{
return (struct compact_list_hdr *)(digest_list->buf + hdr_offset);
}
the header can be obtained by summing the address of the digest list
buffer in the digest_list_item structure with hdr_offset.
Similarly:
static inline u8 *get_digest(struct digest_list_item *digest_list,
loff_t digest_offset, loff_t hdr_offset)
{
/* Digest list digest is stored in a different place. */
if (!digest_offset)
return digest_list->digest;
return digest_list->buf + digest_offset;
}
the digest can be obtained by summing the address of the digest list
buffer with digest_offset (except for the digest lists, where the digest
is stored in the digest field of the digest_list_item structure).
Methods
This section introduces the methods requires to manage the three objects
defined.
digest_item Methods
digest_add()
digest_add() first checks in the hash table for the passed type if a
digest_item for the same digest already exists. If not, it creates a new
one. Then, digest_add() calls digest_list_ref_add() to add a new
reference of the digest list being added to the found or new
digest_item.
digest_del()
digest_del() also searches the digest_item in the hash table. It should
be always found, as digest lists can be deleted only if they were added
before. Then, digest_del() calls digest_list_ref_del() to delete a
reference of the digest list being deleted from the found digest_item.
digest_lookup()
digest_lookup() searches the passed digest in the hash table. Then, it
returns immediately a digest_item (or NULL if the digest is not found)
if the modifiers and actions information are not requested by the
caller, or iterates over all the valid references of the digest and
calculates the OR for both of them. Iteration in the array of references
ends when the digest list pointer in a reference is set to NULL. Access
to the refs array is protected by RCU to avoid access to digest lists
being added or deleted (update is serialized by the securityfs
interfaces).
digest_lookup() is not exposed to the rest of the kernel, because access
to the returned digest_item outside RCU would be illegal.
digest_get_info()
digest_get_info() is the public version of digest_lookup(), which does
not return a digest_item but just the resulting modifiers and actions
from the OR of the modifiers and actions from the referenced digest
lists.
digest_list_item_ref Methods
digest_list_ref_add()
digest_list_ref_add() adds a new reference at the end of the refs array
(also keeps a terminator as the last element). It does not search for
duplicates, as a duplicate reference simply means that the digest
appears multiple times in the digest list. digest_list_ref_add() does
not add the new element in place, but first creates a copy of the
current refs array and uses RCU to replace it with the new one.
digest_list_ref_del()
digest_list_ref_del() first searches in the refs array a reference to a
given digest list. Then, it invalidates the found reference so that it
is skipped by the reader. Afterwards, it tries to allocate a smaller
refs array (with enough slots to store the valid references, except the
one being deleted). If memory allocation succeeds, digest_list_ref_del()
copies the valid references to the copy of refs and uses RCU to replace
the original refs. Otherwise, it keeps the original refs with the
invalidated reference.
digest_list_item Methods
digest_list_add()
digest_list_add() first searches the digest of the digest list in the
hash table for the COMPACT_DIGEST_LIST type. Addition can be done if the
digest list is not found (it is pointless to load the same digest list
again). digest_list_add() then creates a new digest_item, representing
the digest of the digest list, a special digest_list_item_ref with
digest_offset and hdr_offset set to zero, and a new digest_list_item.
digest_list_del()
digest_list_del() also searches the digest of the digest list in the
hash table for the COMPACT_DIGEST_LIST type. Deletion can be done only
if the digest list is found. digest_list_del() then deletes the
digest_list_item, the special digest_list_item_ref and the digest_item.
Parser
This section introduces the necessary functions to parse a digest list
and to execute the requested operation.
The main function is digest_list_parse(), which coordinates the various
steps required to add or delete a digest list, and has the logic to roll
back when one of the steps fails.
1. Calls digest_list_validate() to validate the passed buffer
containing the digest list to ensure that the format is correct.
2. Calls get_digest_list() to create a new digest_list_item for the add
operation, or to retrieve the existing one for the delete operation.
get_digest_list() refuses to add digest lists that were previously
added and to delete digest lists that weren't previously added.
Also, get_digest_list() refuses to delete digest lists that are not
processed in the same way as when they were added (it would
guarantee that also deletion is notified to remote verifiers).
3. Calls _digest_list_parse() which takes the created/retrieved
digest_list_item and adds or delete the digests included in the
digest list.
4. If an error occurred, performs a rollback to the previous state, by
calling _digest_list_parse() with the opposite operation and the
buffer size at the time the error occurred.
5. digest_list_parse() deletes the digest_list_item on unsuccessful add
or successful delete.
Interfaces
This section introduces the interfaces in
<securityfs>/integrity/digest_lists necessary to interact with Huawei
Digest Lists.
digest_list_add
digest_list_add can be used to upload a digest list and add the digests
to the hash table; passed data are interpreted as file path if the first
byte is / or as the digest list itself otherwise.
ima_measure_critical_data() is called to calculate the digest of the
digest list and eventually, if an appropriate rule is set in the IMA
policy, to measure it.
digest_list_del
digest_list_del can be used to upload a digest list and delete the
digests from the hash table; data are interpreted in the same way as
described for digest_list_add.
digest_lists_loaded
digest_lists_loaded is a directory containing two files for each loaded
digest list: one shows the digest list in binary format, and the other
(with .ascii prefix) shows the digest list in ASCII format.
Files are added and removed at the same time digest lists are added and
removed.
digest_label
digest_label can be used to set a label to be applied to the next digest
list (buffer) loaded through digest_list_add.
digest_query
digest_query: allows to write a query in the format <algo>-<digest> and
to obtain all digest lists that include that digest.
digests_count
digests_count shows the current number of digests stored in the hash
table by type.
Testing
This section introduces a number of tests to ensure that Huawei Digest
Lists works as expected:
- digest_list_add_del_test_file_upload;
- digest_list_add_del_test_file_upload_fault;
- digest_list_add_del_test_buffer_upload;
- digest_list_add_del_test_buffer_upload_fault;
- digest_list_fuzzing_test.
The tests are in tools/testing/selftests/digest_lists/selftest.c.
The first four tests randomly perform add, delete and query of digest
lists. They internally keep track at any time of the digest lists that
are currently uploaded to the kernel.
Also, digest lists are generated randomly by selecting an arbitrary
digest algorithm and an arbitrary the number of digests. To ensure a
good number of collisions, digests are a sequence of zeros, except for
the first four bytes that are set with a random number within a defined
range.
When a query operation is selected, a digest is chosen by getting
another random number within the same range. Then, the tests count how
many times the digest is found in the internally stored digest lists and
in the query result obtained from the kernel. The tests are successful
if the obtained numbers are the same.
The file_upload variant creates a temporary file from a generated digest
list and sends its path to the kernel, so that the file is uploaded. The
digest_upload variant directly sends the digest list buffer to the
kernel (it will be done by the user space parser after it converts a
digest list not in the compact format).
The fault variant performs the test by enabling the ad-hoc fault
injection mechanism in the kernel (accessible through
<debugfs>/fail_digest_lists). The fault injection mechanism randomly
injects errors during the addition and deletion of digest lists. When an
error occurs, the rollback mechanism performs the reverse operation
until the point the error occurred, so that the kernel is left in the
same state as when the requested operation began. Since the kernel
returns the error to user space, the tests also know that the operation
didn't succeed and behave accordingly (they also revert the internal
state).
Lastly, the fuzzing test simply sends randomly generated digest lists to
the kernel, to ensure that the parser is robust enough to handle
malformed data.
Roberto Sassu (12):
ima: Add digest, algo, measured parameters to
ima_measure_critical_data()
digest_lists: Overview
digest_lists: Basic definitions
digest_lists: Objects
digest_lists: Methods
digest_lists: Parser
digest_lists: Interfaces - digest_list_add, digest_list_del
digest_lists: Interfaces - digest_lists_loaded
digest_lists: Interfaces - digest_label
digest_lists: Interfaces - digest_query
digest_lists: Interfaces - digests_count
digest_lists: Tests
Documentation/security/digest_lists.rst | 756 +++++++++++
Documentation/security/index.rst | 1 +
MAINTAINERS | 14 +
include/linux/digest_lists.h | 24 +
include/linux/ima.h | 8 +-
include/linux/kernel_read_file.h | 1 +
include/uapi/linux/digest_lists.h | 43 +
security/integrity/Kconfig | 1 +
security/integrity/Makefile | 1 +
security/integrity/digest_lists/Kconfig | 11 +
security/integrity/digest_lists/Makefile | 8 +
.../integrity/digest_lists/digest_lists.h | 155 +++
security/integrity/digest_lists/fs.c | 719 ++++++++++
security/integrity/digest_lists/methods.c | 548 ++++++++
security/integrity/digest_lists/parser.c | 270 ++++
security/integrity/ima/ima.h | 3 +-
security/integrity/ima/ima_appraise.c | 3 +-
security/integrity/ima/ima_asymmetric_keys.c | 3 +-
security/integrity/ima/ima_init.c | 3 +-
security/integrity/ima/ima_main.c | 32 +-
security/integrity/ima/ima_queue_keys.c | 2 +-
security/integrity/integrity.h | 4 +
security/selinux/ima.c | 5 +-
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/digest_lists/Makefile | 6 +
tools/testing/selftests/digest_lists/common.c | 109 ++
tools/testing/selftests/digest_lists/common.h | 37 +
tools/testing/selftests/digest_lists/config | 3 +
.../testing/selftests/digest_lists/selftest.c | 1169 +++++++++++++++++
29 files changed, 3925 insertions(+), 15 deletions(-)
create mode 100644 Documentation/security/digest_lists.rst
create mode 100644 include/linux/digest_lists.h
create mode 100644 include/uapi/linux/digest_lists.h
create mode 100644 security/integrity/digest_lists/Kconfig
create mode 100644 security/integrity/digest_lists/Makefile
create mode 100644 security/integrity/digest_lists/digest_lists.h
create mode 100644 security/integrity/digest_lists/fs.c
create mode 100644 security/integrity/digest_lists/methods.c
create mode 100644 security/integrity/digest_lists/parser.c
create mode 100644 tools/testing/selftests/digest_lists/Makefile
create mode 100644 tools/testing/selftests/digest_lists/common.c
create mode 100644 tools/testing/selftests/digest_lists/common.h
create mode 100644 tools/testing/selftests/digest_lists/config
create mode 100644 tools/testing/selftests/digest_lists/selftest.c
--
2.25.1
(Maintainers bcc, to avoid too many recipient troubles)
As discussed at:
https://lore.kernel.org/linux-doc/871r9k6rmy.fsf@meer.lwn.net/
It is better to avoid using :doc:`foo` to refer to Documentation/foo.rst, as the
automarkup.py extension should handle it automatically, on most cases.
There are a couple of exceptions to this rule:
1. when :doc: tag is used to point to a kernel-doc DOC: markup;
2. when it is used with a named tag, e. g. :doc:`some name <foo>`;
On this series:
Patch 1 manually adjust the references inside driver-api/pm/devices.rst,
as there it uses :file:`foo` to refer to some Documentation/ files;
Patch 2 converts a table at Documentation/dev-tools/kunit/api/index.rst
into a list, carefully avoiding the
The remaining patches convert the other occurrences via a replace script.
They were manually edited, in order to honour 80-columns where possible.
This series based on docs-next branch. In order to avoid merge conflicts,
I rebased it internally against yesterday's linux-next, dropping a patch
and a hunk that would have caused conflicts there.
I'll re-send the remaining patch (plus another patch) with conflicting
changes, together with any other doc:`filename` reference that might
still be upstream by 5.14-rc1.
---
v2:
- dropped media patches (as I merged via my own tree);
- removed one patch that would conflict at linux-next (adm1177.rst);
- removed one hunk fron kunit patch that would also conflict at
linux-next.
Mauro Carvalho Chehab (29):
docs: devices.rst: better reference documentation docs
docs: dev-tools: kunit: don't use a table for docs name
docs: admin-guide: pm: avoid using ReST :doc:`foo` markup
docs: admin-guide: hw-vuln: avoid using ReST :doc:`foo` markup
docs: admin-guide: sysctl: avoid using ReST :doc:`foo` markup
docs: block: biodoc.rst: avoid using ReST :doc:`foo` markup
docs: bpf: bpf_lsm.rst: avoid using ReST :doc:`foo` markup
docs: core-api: avoid using ReST :doc:`foo` markup
docs: dev-tools: testing-overview.rst: avoid using ReST :doc:`foo`
markup
docs: dev-tools: kunit: avoid using ReST :doc:`foo` markup
docs: devicetree: bindings: submitting-patches.rst: avoid using ReST
:doc:`foo` markup
docs: doc-guide: avoid using ReST :doc:`foo` markup
docs: driver-api: avoid using ReST :doc:`foo` markup
docs: driver-api: gpio: using-gpio.rst: avoid using ReST :doc:`foo`
markup
docs: driver-api: surface_aggregator: avoid using ReST :doc:`foo`
markup
docs: driver-api: usb: avoid using ReST :doc:`foo` markup
docs: firmware-guide: acpi: avoid using ReST :doc:`foo` markup
docs: i2c: avoid using ReST :doc:`foo` markup
docs: kernel-hacking: hacking.rst: avoid using ReST :doc:`foo` markup
docs: networking: devlink: avoid using ReST :doc:`foo` markup
docs: PCI: endpoint: pci-endpoint-cfs.rst: avoid using ReST :doc:`foo`
markup
docs: PCI: pci.rst: avoid using ReST :doc:`foo` markup
docs: process: submitting-patches.rst: avoid using ReST :doc:`foo`
markup
docs: security: landlock.rst: avoid using ReST :doc:`foo` markup
docs: trace: coresight: coresight.rst: avoid using ReST :doc:`foo`
markup
docs: trace: ftrace.rst: avoid using ReST :doc:`foo` markup
docs: userspace-api: landlock.rst: avoid using ReST :doc:`foo` markup
docs: virt: kvm: s390-pv-boot.rst: avoid using ReST :doc:`foo` markup
docs: x86: avoid using ReST :doc:`foo` markup
.../PCI/endpoint/pci-endpoint-cfs.rst | 2 +-
Documentation/PCI/pci.rst | 6 +--
.../special-register-buffer-data-sampling.rst | 3 +-
Documentation/admin-guide/pm/intel_idle.rst | 16 +++++---
Documentation/admin-guide/pm/intel_pstate.rst | 9 +++--
Documentation/admin-guide/sysctl/abi.rst | 2 +-
Documentation/admin-guide/sysctl/kernel.rst | 37 ++++++++++---------
Documentation/block/biodoc.rst | 2 +-
Documentation/bpf/bpf_lsm.rst | 13 ++++---
.../core-api/bus-virt-phys-mapping.rst | 2 +-
Documentation/core-api/dma-api.rst | 5 ++-
Documentation/core-api/dma-isa-lpc.rst | 2 +-
Documentation/core-api/index.rst | 4 +-
Documentation/dev-tools/kunit/api/index.rst | 8 ++--
Documentation/dev-tools/kunit/faq.rst | 2 +-
Documentation/dev-tools/kunit/index.rst | 14 +++----
Documentation/dev-tools/kunit/start.rst | 4 +-
Documentation/dev-tools/kunit/tips.rst | 5 ++-
Documentation/dev-tools/kunit/usage.rst | 8 ++--
Documentation/dev-tools/testing-overview.rst | 16 ++++----
.../bindings/submitting-patches.rst | 11 +++---
Documentation/doc-guide/contributing.rst | 8 ++--
Documentation/driver-api/gpio/using-gpio.rst | 4 +-
Documentation/driver-api/ioctl.rst | 2 +-
Documentation/driver-api/pm/devices.rst | 8 ++--
.../surface_aggregator/clients/index.rst | 3 +-
.../surface_aggregator/internal.rst | 15 ++++----
.../surface_aggregator/overview.rst | 6 ++-
Documentation/driver-api/usb/dma.rst | 6 +--
.../acpi/dsd/data-node-references.rst | 3 +-
.../firmware-guide/acpi/dsd/graph.rst | 2 +-
.../firmware-guide/acpi/enumeration.rst | 7 ++--
Documentation/i2c/instantiating-devices.rst | 2 +-
Documentation/i2c/old-module-parameters.rst | 3 +-
Documentation/i2c/smbus-protocol.rst | 4 +-
Documentation/kernel-hacking/hacking.rst | 4 +-
.../networking/devlink/devlink-region.rst | 2 +-
.../networking/devlink/devlink-trap.rst | 4 +-
Documentation/process/submitting-patches.rst | 32 ++++++++--------
Documentation/security/landlock.rst | 3 +-
Documentation/trace/coresight/coresight.rst | 8 ++--
Documentation/trace/ftrace.rst | 2 +-
Documentation/userspace-api/landlock.rst | 11 +++---
Documentation/virt/kvm/s390-pv-boot.rst | 2 +-
Documentation/x86/boot.rst | 4 +-
Documentation/x86/mtrr.rst | 2 +-
46 files changed, 171 insertions(+), 147 deletions(-)
--
2.31.1
Current PTP driver exposes one PTP device to user which binds network
interface/interfaces to provide timestamping. Actually we have a way
utilizing timecounter/cyclecounter to virtualize any number of PTP
clocks based on a same free running physical clock for using.
The purpose of having multiple PTP virtual clocks is for user space
to directly/easily use them for multiple domains synchronization.
user
space: ^ ^
| SO_TIMESTAMPING new flag: | Packets with
| SOF_TIMESTAMPING_BIND_PHC | TX/RX HW timestamps
v v
+--------------------------------------------+
sock: | sock (new member sk_bind_phc) |
+--------------------------------------------+
^ ^
| ethtool_op_get_phc_vclocks | Convert HW timestamps
| | to sk_bind_phc
v v
+--------------+--------------+--------------+
vclock: | ptp1 | ptp2 | ptpN |
+--------------+--------------+--------------+
pclock: | ptp0 free running |
+--------------------------------------------+
The block diagram may explain how it works. Besides the PTP virtual
clocks, the packet HW timestamp converting to the bound PHC is also
done in sock driver. For user space, PTP virtual clocks can be
created via sysfs, and extended SO_TIMESTAMPING API (new flag
SOF_TIMESTAMPING_BIND_PHC) can be used to bind one PTP virtual clock
for timestamping.
The test tool timestamping.c (together with linuxptp phc_ctl tool) can
be used to verify:
# echo 4 > /sys/class/ptp/ptp0/n_vclocks
[ 129.399472] ptp ptp0: new virtual clock ptp2
[ 129.404234] ptp ptp0: new virtual clock ptp3
[ 129.409532] ptp ptp0: new virtual clock ptp4
[ 129.413942] ptp ptp0: new virtual clock ptp5
[ 129.418257] ptp ptp0: guarantee physical clock free running
#
# phc_ctl /dev/ptp2 set 10000
# phc_ctl /dev/ptp3 set 20000
#
# timestamping eno0 2 SOF_TIMESTAMPING_TX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE SOF_TIMESTAMPING_BIND_PHC
# timestamping eno0 2 SOF_TIMESTAMPING_RX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE SOF_TIMESTAMPING_BIND_PHC
# timestamping eno0 3 SOF_TIMESTAMPING_TX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE SOF_TIMESTAMPING_BIND_PHC
# timestamping eno0 3 SOF_TIMESTAMPING_RX_HARDWARE SOF_TIMESTAMPING_RAW_HARDWARE SOF_TIMESTAMPING_BIND_PHC
Changes for v2:
- Converted to num_vclocks for creating virtual clocks.
- Guranteed physical clock free running when using virtual
clocks.
- Fixed build warning.
- Updated copyright.
Changes for v3:
- Supported PTP virtual clock in default in PTP driver.
- Protected concurrency of ptp->num_vclocks accessing.
- Supported PHC vclocks query via ethtool.
- Extended SO_TIMESTAMPING API for PHC binding.
- Converted HW timestamps to PHC bound, instead of previous
binding domain value to PHC idea.
- Other minor fixes.
Yangbo Lu (10):
ptp: add ptp virtual clock driver framework
ptp: support ptp physical/virtual clocks conversion
ptp: track available ptp vclocks information
ptp: add kernel API ptp_get_vclocks_index()
ethtool: add a new command for getting PHC virtual clocks
ptp: add kernel API ptp_convert_timestamp()
net: sock: extend SO_TIMESTAMPING for PHC binding
net: socket: support hardware timestamp conversion to PHC bound
selftests/net: timestamping: support binding PHC
MAINTAINERS: add entry for PTP virtual clock driver
Documentation/ABI/testing/sysfs-ptp | 13 ++
MAINTAINERS | 7 +
drivers/ptp/Makefile | 2 +-
drivers/ptp/ptp_clock.c | 27 ++-
drivers/ptp/ptp_private.h | 34 ++++
drivers/ptp/ptp_sysfs.c | 95 +++++++++
drivers/ptp/ptp_vclock.c | 212 +++++++++++++++++++++
include/linux/ethtool.h | 2 +
include/linux/ptp_clock_kernel.h | 29 ++-
include/net/sock.h | 8 +-
include/uapi/linux/ethtool.h | 14 ++
include/uapi/linux/ethtool_netlink.h | 15 ++
include/uapi/linux/net_tstamp.h | 17 +-
include/uapi/linux/ptp_clock.h | 5 +
net/core/sock.c | 65 ++++++-
net/ethtool/Makefile | 2 +-
net/ethtool/common.c | 24 +++
net/ethtool/common.h | 2 +
net/ethtool/ioctl.c | 27 +++
net/ethtool/netlink.c | 10 +
net/ethtool/netlink.h | 2 +
net/ethtool/phc_vclocks.c | 86 +++++++++
net/mptcp/sockopt.c | 10 +-
net/socket.c | 19 +-
tools/testing/selftests/net/timestamping.c | 62 ++++--
25 files changed, 750 insertions(+), 39 deletions(-)
create mode 100644 drivers/ptp/ptp_vclock.c
create mode 100644 net/ethtool/phc_vclocks.c
base-commit: 89212e160b81e778f829b89743570665810e3b13
--
2.25.1
Shuah,
Do kselftests require to be backward-compatible?
I see Documentation/dev-tools/kselftest.rst does not require this, but
maybe it's assumed like in other test suites (or in perf).
| In general, the rules for selftests are
|
| * Do as much as you can if you're not root;
|
| * Don't take too long;
|
| * Don't break the build on any architecture, and
|
| * Don't cause the top-level "make run_tests" to fail if your feature is
| unconfigured.
For example LTP says:
| LTP test should be as backward compatible as possible. [...]
|
| Therefore LTP test for more current features should be able to cope with older
| systems.
Also, (it's said[1]) perf, even though in kernel tree, is supposed to work
properly on any (older/newer) version of Linux.
Can you clarify this point in kselftest.rst?
I think, this would be useful for future kselftests developers, users,
and packagers. (Currently, I package for ALT Linux kselftests (and perf)
from the latest mainline branch, so people could test even older kernels
with the latest kselftests.
If there is policy to be backward-compatible kselftests in the future
could reach a state where users would run them in all pass mode (without
selecting only working tests). This, in turn, would increase [ease of]
usability of tests and thus frequency of their run and consequentially
quality kernel testing overall.
Thanks,
[1] https://lkml.org/lkml/2020/7/29/677
Some environments do not set $SHELL when running tests. There's no need
to use $SHELL here anyway, so just replace it with hard-coded path
instead. Additionally avoid using bash-isms in the command, so that
regular /bin/sh can be used.
Suggested-by: Guillaume Tucker <guillaume.tucker(a)collabora.com>
Fixes: 46d1a0f03d66 ("selftests/lkdtm: Add tests for LKDTM targets")
Cc: stable(a)vger.kernel.org
Signed-off-by: Kees Cook <keescook(a)chromium.org>
---
tools/testing/selftests/lkdtm/run.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/lkdtm/run.sh b/tools/testing/selftests/lkdtm/run.sh
index bb7a1775307b..0f9f22ac004b 100755
--- a/tools/testing/selftests/lkdtm/run.sh
+++ b/tools/testing/selftests/lkdtm/run.sh
@@ -78,8 +78,9 @@ dmesg > "$DMESG"
# Most shells yell about signals and we're expecting the "cat" process
# to usually be killed by the kernel. So we have to run it in a sub-shell
-# and silence errors.
-($SHELL -c 'cat <(echo '"$test"') >'"$TRIGGER" 2>/dev/null) || true
+# to avoid terminating this script. Leave stderr alone, just in case
+# something _else_ happens.
+(/bin/sh -c '(echo '"$test"') | cat >'"$TRIGGER") || true
# Record and dump the results
dmesg | comm --nocheck-order -13 "$DMESG" - > "$LOG" || true
--
2.25.1
The SGX selftests can fail for a bunch of non-obvious reasons
like 'noexec' permissions on /dev (which is the default *EVERYWHERE*
it seems).
A new test mistakenly also looked for +x permission on the
/dev/sgx_enclave. File execute permissions really only apply to
the ability of execve() to work on a file, *NOT* on the ability
for an application to map the file with PROT_EXEC. SGX needs to
mmap(PROT_EXEC), but doesn't need to execve() the device file.
Remove the check.
Fixes: 4284f7acb78b ("selftests/sgx: Improve error detection and messages")
Reported-by: Tim Gardner <tim.gardner(a)canonical.com>
Cc: Jarkko Sakkinen <jarkko(a)kernel.org>
Cc: Reinette Chatre <reinette.chatre(a)intel.com>
Cc: Dave Hansen <dave.hansen(a)linux.intel.com>
Cc: Shuah Khan <shuah(a)kernel.org>
Cc: linux-sgx(a)vger.kernel.org
Cc: linux-kselftest(a)vger.kernel.org
Cc: linux-kernel(a)vger.kernel.org
---
b/tools/testing/selftests/sgx/load.c | 16 +++-------------
1 file changed, 3 insertions(+), 13 deletions(-)
diff -puN tools/testing/selftests/sgx/load.c~sgx-no-file-exec tools/testing/selftests/sgx/load.c
--- a/tools/testing/selftests/sgx/load.c~sgx-no-file-exec 2021-06-21 11:48:25.226294281 -0700
+++ b/tools/testing/selftests/sgx/load.c 2021-06-21 12:03:28.023292029 -0700
@@ -150,16 +150,6 @@ bool encl_load(const char *path, struct
goto err;
}
- /*
- * This just checks if the /dev file has these permission
- * bits set. It does not check that the current user is
- * the owner or in the owning group.
- */
- if (!(sb.st_mode & (S_IXUSR | S_IXGRP | S_IXOTH))) {
- fprintf(stderr, "no execute permissions on device file %s\n", device_path);
- goto err;
- }
-
ptr = mmap(NULL, PAGE_SIZE, PROT_READ, MAP_SHARED, fd, 0);
if (ptr == (void *)-1) {
perror("mmap for read");
@@ -169,13 +159,13 @@ bool encl_load(const char *path, struct
#define ERR_MSG \
"mmap() succeeded for PROT_READ, but failed for PROT_EXEC.\n" \
-" Check that current user has execute permissions on %s and \n" \
-" that /dev does not have noexec set: mount | grep \"/dev .*noexec\"\n" \
+" Check that /dev does not have noexec set:\n" \
+" \tmount | grep \"/dev .*noexec\"\n" \
" If so, remount it executable: mount -o remount,exec /dev\n\n"
ptr = mmap(NULL, PAGE_SIZE, PROT_EXEC, MAP_SHARED, fd, 0);
if (ptr == (void *)-1) {
- fprintf(stderr, ERR_MSG, device_path);
+ fprintf(stderr, ERR_MSG);
goto err;
}
munmap(ptr, PAGE_SIZE);
_
According to a comment in commit 99513cfa16c6 ("selftest: Fixes for
icmp_redirect test") the test "IPv6: mtu exception plus redirect" is
expected to fail, because of a bug in the IPv6 logic that hasn't been
fixed yet apparently.
We should probably consider this failure as an "expected failure",
therefore change the script to return XFAIL for that particular test and
also report the total amount of expected failures at the end of the run.
Signed-off-by: Andrea Righi <andrea.righi(a)canonical.com>
---
tools/testing/selftests/net/icmp_redirect.sh | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/icmp_redirect.sh b/tools/testing/selftests/net/icmp_redirect.sh
index bf361f30d6ef..c19ecc6a8614 100755
--- a/tools/testing/selftests/net/icmp_redirect.sh
+++ b/tools/testing/selftests/net/icmp_redirect.sh
@@ -63,10 +63,14 @@ log_test()
local rc=$1
local expected=$2
local msg="$3"
+ local xfail=$4
if [ ${rc} -eq ${expected} ]; then
printf "TEST: %-60s [ OK ]\n" "${msg}"
nsuccess=$((nsuccess+1))
+ elif [ ${rc} -eq ${xfail} ]; then
+ printf "TEST: %-60s [XFAIL]\n" "${msg}"
+ nxfail=$((nxfail+1))
else
ret=1
nfail=$((nfail+1))
@@ -322,7 +326,7 @@ check_exception()
ip -netns h1 -6 ro get ${H1_VRF_ARG} ${H2_N2_IP6} | \
grep -v "mtu" | grep -q "${R1_LLADDR}"
fi
- log_test $? 0 "IPv6: ${desc}"
+ log_test $? 0 "IPv6: ${desc}" 1
}
run_ping()
@@ -488,6 +492,7 @@ which ping6 > /dev/null 2>&1 && ping6=$(which ping6) || ping6=$(which ping)
ret=0
nsuccess=0
nfail=0
+nxfail=0
while getopts :pv o
do
@@ -532,5 +537,6 @@ fi
printf "\nTests passed: %3d\n" ${nsuccess}
printf "Tests failed: %3d\n" ${nfail}
+printf "Tests xfailed: %3d\n" ${nxfail}
exit $ret
--
2.31.1
When running event-no-pid test on small machines (e.g. cloud 1-core
instance), other events might not happen:
+ cat trace
+ cnt=0
+ [ 0 -eq 0 ]
+ fail No other events were recorded
[15] event tracing - restricts events based on pid notrace filtering [FAIL]
Schedule a simple sleep task to be sure that some other process events
get recorded.
Fixes: ebed9628f5c2 ("selftests/ftrace: Add test to test new set_event_notrace_pid file")
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski(a)canonical.com>
Acked-by: Steven Rostedt (VMware) <rostedt(a)goodmis.org>
---
Changes since v1:
1. Correct spelling in commit msg.
2. Add Steven's ack.
---
.../testing/selftests/ftrace/test.d/event/event-no-pid.tc | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc b/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc
index e6eb78f0b954..9933ed24f901 100644
--- a/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc
+++ b/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc
@@ -57,6 +57,10 @@ enable_events() {
echo 1 > tracing_on
}
+other_task() {
+ sleep .001 || usleep 1 || sleep 1
+}
+
echo 0 > options/event-fork
do_reset
@@ -94,6 +98,9 @@ child=$!
echo "child = $child"
wait $child
+# Be sure some other events will happen for small systems (e.g. 1 core)
+other_task
+
echo 0 > tracing_on
cnt=`count_pid $mypid`
--
2.27.0
Dear linux-kselftest
The famous brand John Lewis Partnership, is UK's largest multi-
channel retailer with over 126 shops and multiple expansion in
Africa furnished by European/Asian/American products. We are
sourcing new products to attract new customers and also retain
our existing ones, create new partnerships with companies dealing
with different kinds of goods globally.
Your company's products are of interest to our market as we have
an amazing market for your products.
Provide us your current catalog through email to review more. We
hope to be able to order with you and start a long-term friendly,
respectable and solid business partnership. Please we would
appreciate it if you could send us your stock availability via
email if any.
Our payment terms are 15 days net in Europe, 30 days Net in UK
and 30 days net in Asia/USA as we operate with over 5297
suppliers around the globe for the past 50 years now. For
immediate response Send your reply to robert_turner@johnlewis-
trading.com for us to be able to
treat with care and urgency.
Best Regards
Rob Turner
Head Of Procurement Operations
John Lewis & Partners.
robert_turner(a)johnlewis-trading.com
Tel: +44-7451-274090
WhatsApp: +447497483925
www.johnlewis.com
REGISTERED OFFICE: 171 VICTORIA STREET, LONDON SW1E 5NN
On Wed, Jun 23, 2021 at 10:35:50PM +0800, kernel test robot wrote:
>
>
> Greeting,
>
> FYI, we noticed the following commit (built with gcc-9):
>
> commit: 84d8cf25b0f80da0ac229214864654a7662ec7e4 ("[PATCH v2] selftests/lkdtm: Use /bin/sh not $SHELL")
> url: https://github.com/0day-ci/linux/commits/Kees-Cook/selftests-lkdtm-Use-bin-…
> base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
>
> in testcase: kernel-selftests
> version: kernel-selftests-x86_64-f8879e85-1_20210618
> with following parameters:
>
> group: lkdtm
> ucode: 0xe2
Heh. Yes, this is working as intended. :) Most of the lkdtm tests will
trigger Oopses, and this is by design: it is checking that the kernel
catches bad conditions and freaks out appropriately.
-Kees
--
Kees Cook
When running event-no-pid test on a small machines (e.g. cloud 1-core
instance), other events might not happen:
+ cat trace
+ cnt=0
+ [ 0 -eq 0 ]
+ fail No other events were recorded
[15] event tracing - restricts events based on pid notrace filtering [FAIL]
Schedule a simple sleep task to be sure that some other process events
get recorder.
Fixes: ebed9628f5c2 ("selftests/ftrace: Add test to test new set_event_notrace_pid file")
Signed-off-by: Krzysztof Kozlowski <krzysztof.kozlowski(a)canonical.com>
---
.../testing/selftests/ftrace/test.d/event/event-no-pid.tc | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc b/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc
index e6eb78f0b954..9933ed24f901 100644
--- a/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc
+++ b/tools/testing/selftests/ftrace/test.d/event/event-no-pid.tc
@@ -57,6 +57,10 @@ enable_events() {
echo 1 > tracing_on
}
+other_task() {
+ sleep .001 || usleep 1 || sleep 1
+}
+
echo 0 > options/event-fork
do_reset
@@ -94,6 +98,9 @@ child=$!
echo "child = $child"
wait $child
+# Be sure some other events will happen for small systems (e.g. 1 core)
+other_task
+
echo 0 > tracing_on
cnt=`count_pid $mypid`
--
2.27.0
Commit 22f232d134e1 ("KVM: selftests: x86: Set supported CPUIDs on
default VM") moved vcpu_set_cpuid into vm_create_with_vcpus, but
dirty_log_test doesn't use it to create vm. So vcpu's CPUIDs is
not set, the guest's pa_bits in kvm would be smaller than the
value queried by userspace.
However, the dirty track memory slot is in the highest GPA, the
reserved bits in gpte would be set with wrong pa_bits.
For shadowpaing, page fault would fail in permission_fault and
be injected into guest. Since guest doesn't have idt, it finally
leads to vm_exit for triple fault.
Move vcpu_set_cpuid into vm_vcpu_add_default to set supported
CPUIDs on default vcpu, since almost all tests need it.
Fixes: 22f232d134e1 ("KVM: selftests: x86: Set supported CPUIDs on default VM")
Signed-off-by: Hou Wenlong <houwenlong93(a)linux.alibaba.com>
---
tools/testing/selftests/kvm/lib/kvm_util.c | 4 ----
tools/testing/selftests/kvm/lib/x86_64/processor.c | 3 +++
tools/testing/selftests/kvm/steal_time.c | 2 --
3 files changed, 3 insertions(+), 6 deletions(-)
diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c
index a2b732cf96ea..8ea854d7822d 100644
--- a/tools/testing/selftests/kvm/lib/kvm_util.c
+++ b/tools/testing/selftests/kvm/lib/kvm_util.c
@@ -375,10 +375,6 @@ struct kvm_vm *vm_create_with_vcpus(enum vm_guest_mode mode, uint32_t nr_vcpus,
uint32_t vcpuid = vcpuids ? vcpuids[i] : i;
vm_vcpu_add_default(vm, vcpuid, guest_code);
-
-#ifdef __x86_64__
- vcpu_set_cpuid(vm, vcpuid, kvm_get_supported_cpuid());
-#endif
}
return vm;
diff --git a/tools/testing/selftests/kvm/lib/x86_64/processor.c b/tools/testing/selftests/kvm/lib/x86_64/processor.c
index efe235044421..595322b24e4c 100644
--- a/tools/testing/selftests/kvm/lib/x86_64/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86_64/processor.c
@@ -600,6 +600,9 @@ void vm_vcpu_add_default(struct kvm_vm *vm, uint32_t vcpuid, void *guest_code)
/* Setup the MP state */
mp_state.mp_state = 0;
vcpu_set_mp_state(vm, vcpuid, &mp_state);
+
+ /* Setup supported CPUIDs */
+ vcpu_set_cpuid(vm, vcpuid, kvm_get_supported_cpuid());
}
/*
diff --git a/tools/testing/selftests/kvm/steal_time.c b/tools/testing/selftests/kvm/steal_time.c
index fcc840088c91..a6fe75cb9a6e 100644
--- a/tools/testing/selftests/kvm/steal_time.c
+++ b/tools/testing/selftests/kvm/steal_time.c
@@ -73,8 +73,6 @@ static void steal_time_init(struct kvm_vm *vm)
for (i = 0; i < NR_VCPUS; ++i) {
int ret;
- vcpu_set_cpuid(vm, i, kvm_get_supported_cpuid());
-
/* ST_GPA_BASE is identity mapped */
st_gva[i] = (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE);
sync_global_to_guest(vm, st_gva[i]);
--
2.31.1
This introduces signal->unsafe_execve_in_progress,
which is used to fix the case when at least one of the
sibling threads is traced, and therefore the trace
process may dead-lock in ptrace_attach, but de_thread
will need to wait for the tracer to continue execution.
The solution is to detect this situation and allow
ptrace_attach to continue, while de_thread() is still
waiting for traced zombies to be eventually released.
When the current thread changed the ptrace status from
non-traced to traced, we can simply abort the whole
execve and restart it by returning -ERESTARTSYS.
This needs to be done before changing the thread leader,
because the PTRACE_EVENT_EXEC needs to know the old
thread pid.
Although it is technically after the point of no return,
we just have to reset bprm->point_of_no_return here,
since at this time only the other threads have received
a fatal signal, not the current thread.
>From the user's point of view the whole execve was
simply delayed until after the ptrace_attach.
Other threads die quickly since the cred_guard_mutex
is released, but a deadly signal is already pending.
In case the mutex_lock_killable misses the signal,
->unsafe_execve_in_progress makes sure they release
the mutex immediately and return with -ERESTARTNOINTR.
This means there is no API change, unlike the previous
version of this patch which was discussed here:
https://lore.kernel.org/lkml/b6537ae6-31b1-5c50-f32b-8b8332ace882@hotmail.d…
See tools/testing/selftests/ptrace/vmaccess.c
for a test case that gets fixed by this change.
Note that since the test case was originally designed to
test the ptrace_attach returning an error in this situation,
the test expectation needed to be adjusted, to allow the
API to succeed at the first attempt.
Signed-off-by: Bernd Edlinger <bernd.edlinger(a)hotmail.de>
---
fs/exec.c | 45 ++++++++++++++++++++++++++-----
fs/proc/base.c | 6 +++++
include/linux/sched/signal.h | 13 +++++++++
kernel/ptrace.c | 9 +++++++
kernel/seccomp.c | 12 ++++++---
tools/testing/selftests/ptrace/vmaccess.c | 25 +++++++++++------
6 files changed, 92 insertions(+), 18 deletions(-)
diff --git a/fs/exec.c b/fs/exec.c
index 8344fba..ac3fec1 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1040,6 +1040,8 @@ static int de_thread(struct task_struct *tsk)
struct signal_struct *sig = tsk->signal;
struct sighand_struct *oldsighand = tsk->sighand;
spinlock_t *lock = &oldsighand->siglock;
+ unsigned int prev_ptrace = tsk->ptrace;
+ struct task_struct *t = tsk;
if (thread_group_empty(tsk))
goto no_thread_group;
@@ -1057,20 +1059,40 @@ static int de_thread(struct task_struct *tsk)
return -EAGAIN;
}
+ while_each_thread(tsk, t) {
+ if (unlikely(t->ptrace) && t != tsk->group_leader)
+ sig->unsafe_execve_in_progress = true;
+ }
+
sig->group_exit_task = tsk;
sig->notify_count = zap_other_threads(tsk);
if (!thread_group_leader(tsk))
sig->notify_count--;
+ spin_unlock_irq(lock);
- while (sig->notify_count) {
- __set_current_state(TASK_KILLABLE);
- spin_unlock_irq(lock);
+ if (unlikely(sig->unsafe_execve_in_progress))
+ mutex_unlock(&sig->cred_guard_mutex);
+
+ for (;;) {
+ set_current_state(TASK_KILLABLE);
+ if (!sig->notify_count)
+ break;
schedule();
if (__fatal_signal_pending(tsk))
goto killed;
- spin_lock_irq(lock);
}
- spin_unlock_irq(lock);
+ __set_current_state(TASK_RUNNING);
+
+ if (unlikely(sig->unsafe_execve_in_progress)) {
+ if (mutex_lock_killable(&sig->cred_guard_mutex))
+ goto killed;
+ sig->unsafe_execve_in_progress = false;
+ if (!prev_ptrace && tsk->ptrace) {
+ sig->group_exit_task = NULL;
+ sig->notify_count = 0;
+ return -ERESTARTSYS;
+ }
+ }
/*
* At this point all other threads have exited, all we have to
@@ -1255,8 +1277,11 @@ int begin_new_exec(struct linux_binprm * bprm)
* Make this the only thread in the thread group.
*/
retval = de_thread(me);
- if (retval)
+ if (retval) {
+ if (retval == -ERESTARTSYS)
+ bprm->point_of_no_return = false;
goto out;
+ }
/*
* Cancel any io_uring activity across execve
@@ -1466,6 +1491,11 @@ static int prepare_bprm_creds(struct linux_binprm *bprm)
if (mutex_lock_interruptible(¤t->signal->cred_guard_mutex))
return -ERESTARTNOINTR;
+ if (unlikely(current->signal->unsafe_execve_in_progress)) {
+ mutex_unlock(¤t->signal->cred_guard_mutex);
+ return -ERESTARTNOINTR;
+ }
+
bprm->cred = prepare_exec_creds();
if (likely(bprm->cred))
return 0;
@@ -1482,7 +1512,8 @@ static void free_bprm(struct linux_binprm *bprm)
}
free_arg_pages(bprm);
if (bprm->cred) {
- mutex_unlock(¤t->signal->cred_guard_mutex);
+ if (!current->signal->unsafe_execve_in_progress)
+ mutex_unlock(¤t->signal->cred_guard_mutex);
abort_creds(bprm->cred);
}
if (bprm->file) {
diff --git a/fs/proc/base.c b/fs/proc/base.c
index 3851bfc..3b2a55c 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -2739,6 +2739,12 @@ static ssize_t proc_pid_attr_write(struct file * file, const char __user * buf,
if (rv < 0)
goto out_free;
+ if (unlikely(current->signal->unsafe_execve_in_progress)) {
+ mutex_unlock(¤t->signal->cred_guard_mutex);
+ rv = -ERESTARTNOINTR;
+ goto out_free;
+ }
+
rv = security_setprocattr(PROC_I(inode)->op.lsm,
file->f_path.dentry->d_name.name, page,
count);
diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h
index 3f6a0fc..220a083 100644
--- a/include/linux/sched/signal.h
+++ b/include/linux/sched/signal.h
@@ -214,6 +214,17 @@ struct signal_struct {
#endif
/*
+ * Set while execve is executing but is *not* holding
+ * cred_guard_mutex to avoid possible dead-locks.
+ * The cred_guard_mutex is released *after* de_thread() has
+ * called zap_other_threads(), therefore a fatal signal is
+ * guaranteed to be already pending in the unlikely event, that
+ * current->signal->unsafe_execve_in_progress happens to be
+ * true after the cred_guard_mutex was acquired.
+ */
+ bool unsafe_execve_in_progress;
+
+ /*
* Thread is the potential origin of an oom condition; kill first on
* oom
*/
@@ -227,6 +238,8 @@ struct signal_struct {
struct mutex cred_guard_mutex; /* guard against foreign influences on
* credential calculations
* (notably. ptrace)
+ * Held while execve runs, except when
+ * a sibling thread is being traced.
* Deprecated do not use in new code.
* Use exec_update_lock instead.
*/
diff --git a/kernel/ptrace.c b/kernel/ptrace.c
index 61db50f..0cbc1eb 100644
--- a/kernel/ptrace.c
+++ b/kernel/ptrace.c
@@ -468,6 +468,14 @@ static int ptrace_traceme(void)
{
int ret = -EPERM;
+ if (mutex_lock_interruptible(¤t->signal->cred_guard_mutex))
+ return -ERESTARTNOINTR;
+
+ if (unlikely(current->signal->unsafe_execve_in_progress)) {
+ mutex_unlock(¤t->signal->cred_guard_mutex);
+ return -ERESTARTNOINTR;
+ }
+
write_lock_irq(&tasklist_lock);
/* Are we already being traced? */
if (!current->ptrace) {
@@ -483,6 +491,7 @@ static int ptrace_traceme(void)
}
}
write_unlock_irq(&tasklist_lock);
+ mutex_unlock(¤t->signal->cred_guard_mutex);
return ret;
}
diff --git a/kernel/seccomp.c b/kernel/seccomp.c
index 1d60fc2..b1389ee 100644
--- a/kernel/seccomp.c
+++ b/kernel/seccomp.c
@@ -1824,9 +1824,15 @@ static long seccomp_set_mode_filter(unsigned int flags,
* Make sure we cannot change seccomp or nnp state via TSYNC
* while another thread is in the middle of calling exec.
*/
- if (flags & SECCOMP_FILTER_FLAG_TSYNC &&
- mutex_lock_killable(¤t->signal->cred_guard_mutex))
- goto out_put_fd;
+ if (flags & SECCOMP_FILTER_FLAG_TSYNC) {
+ if (mutex_lock_killable(¤t->signal->cred_guard_mutex))
+ goto out_put_fd;
+
+ if (unlikely(current->signal->unsafe_execve_in_progress)) {
+ mutex_unlock(¤t->signal->cred_guard_mutex);
+ goto out_put_fd;
+ }
+ }
spin_lock_irq(¤t->sighand->siglock);
diff --git a/tools/testing/selftests/ptrace/vmaccess.c b/tools/testing/selftests/ptrace/vmaccess.c
index 4db327b..c7c2242 100644
--- a/tools/testing/selftests/ptrace/vmaccess.c
+++ b/tools/testing/selftests/ptrace/vmaccess.c
@@ -39,8 +39,15 @@ static void *thread(void *arg)
f = open(mm, O_RDONLY);
ASSERT_GE(f, 0);
close(f);
- f = kill(pid, SIGCONT);
- ASSERT_EQ(f, 0);
+ f = waitpid(-1, NULL, 0);
+ ASSERT_NE(f, -1);
+ ASSERT_NE(f, 0);
+ ASSERT_NE(f, pid);
+ f = waitpid(-1, NULL, 0);
+ ASSERT_EQ(f, pid);
+ f = waitpid(-1, NULL, 0);
+ ASSERT_EQ(f, -1);
+ ASSERT_EQ(errno, ECHILD);
}
TEST(attach)
@@ -57,22 +64,24 @@ static void *thread(void *arg)
sleep(1);
k = ptrace(PTRACE_ATTACH, pid, 0L, 0L);
- ASSERT_EQ(errno, EAGAIN);
- ASSERT_EQ(k, -1);
+ ASSERT_EQ(k, 0);
k = waitpid(-1, &s, WNOHANG);
ASSERT_NE(k, -1);
ASSERT_NE(k, 0);
ASSERT_NE(k, pid);
ASSERT_EQ(WIFEXITED(s), 1);
ASSERT_EQ(WEXITSTATUS(s), 0);
- sleep(1);
- k = ptrace(PTRACE_ATTACH, pid, 0L, 0L);
- ASSERT_EQ(k, 0);
k = waitpid(-1, &s, 0);
ASSERT_EQ(k, pid);
ASSERT_EQ(WIFSTOPPED(s), 1);
ASSERT_EQ(WSTOPSIG(s), SIGSTOP);
- k = ptrace(PTRACE_DETACH, pid, 0L, 0L);
+ k = ptrace(PTRACE_CONT, pid, 0L, 0L);
+ ASSERT_EQ(k, 0);
+ k = waitpid(-1, &s, 0);
+ ASSERT_EQ(k, pid);
+ ASSERT_EQ(WIFSTOPPED(s), 1);
+ ASSERT_EQ(WSTOPSIG(s), SIGTRAP);
+ k = ptrace(PTRACE_CONT, pid, 0L, 0L);
ASSERT_EQ(k, 0);
k = waitpid(-1, &s, 0);
ASSERT_EQ(k, pid);
--
1.9.1
This patchset provides a file descriptor for every VM and VCPU to read
KVM statistics data in binary format.
It is meant to provide a lightweight, flexible, scalable and efficient
lock-free solution for user space telemetry applications to pull the
statistics data periodically for large scale systems. The pulling
frequency could be as high as a few times per second.
In this patchset, every statistics data are treated to have some
attributes as below:
* architecture dependent or generic
* VM statistics data or VCPU statistics data
* type: cumulative, instantaneous, peak
* unit: none for simple counter, nanosecond, microsecond,
millisecond, second, Byte, KiByte, MiByte, GiByte, Clock Cycles
Since no lock/synchronization is used, the consistency between all
the statistics data is not guaranteed. That means not all statistics
data are read out at the exact same time, since the statistics data
are still being updated by KVM subsystems while they are read out.
---
* v11 -> v12
- Revised the structure kvm_stats_header and corresponding code by
Paolo's suggestion. Move the id string out of header.
- Define stats header and stats descriptors as const.
- Update some comments by Greg's review.
* v10 -> v11
- Rebase to kvm/queue, commit f1b832550832
(KVM: x86/mmu: Fix TDP MMU page table level)
- Separate binary stats implementation commit
- Use flexible length array member field in API structure instead of
zero-length array member field
- Move major binary stats reading function in a separate source file
- Move stats id string into vm/vcpu structures
- Add some detailed comments and update commit messages
- Addressed some other review comments from Greg K.H. and Paolo.
* v9 -> v10
- Relocate vcpu stat in vcpu's slab's usercopy region
- Fix test issue for capability checking
- Update commit message to explain why/how we need to add this new
API for KVM statistics
* v8 -> v9
- Rebase to commit 8331a2bc0898
(KVM: X86: Introduce KVM_HC_MAP_GPA_RANGE hypercall)
- Reduce code duplication between binary and debugfs interface
- Add field "offset" in stats descriptor to let us define stats
descriptors in any order (not necessary in the order of stats
defined in vm/vcpu stats structures)
- Add static check to make sure the number of stats descriptors
is the same as the number of stats defined in vm/vcpu stats
structures
- Fix missing/mismatched stats descriptor definition caused by
rebase
* v7 -> v8
- Rebase to kvm/queue, commit c1dc20e254b4 ("KVM: switch per-VM
stats to u64")
- Revise code to reflect the per-VM stats type from ulong to u64
- Addressed some other nits
* v6 -> v7
- Improve file descriptor allocation function by Krish suggestion
- Use "generic stats" instead of "common stats" as Krish suggested
- Addressed some other nits from Krish and David Matlack
* v5 -> v6
- Use designated initializers for STATS_DESC
- Change KVM_STATS_SCALE... to KVM_STATS_BASE...
- Use a common function for kvm_[vm|vcpu]_stats_read
- Fix some documentation errors/missings
- Use TEST_ASSERT in selftest
- Use a common function for [vm|vcpu]_stats_test in selftest
* v4 -> v5
- Rebase to kvm/queue, commit a4345a7cecfb ("Merge tag
'kvmarm-fixes-5.13-1'")
- Change maximum stats name length to 48
- Replace VM_STATS_COMMON/VCPU_STATS_COMMON macros with stats
descriptor definition macros.
- Fixed some errors/warnings reported by checkpatch.pl
* v3 -> v4
- Rebase to kvm/queue, commit 9f242010c3b4 ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Use C-stype comments in the whole patch
- Fix wrong count for x86 VCPU stats descriptors
- Fix KVM stats data size counting and validity check in selftest
* v2 -> v3
- Rebase to kvm/queue, commit edf408f5257b ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Resolve some nitpicks about format
* v1 -> v2
- Use ARRAY_SIZE to count the number of stats descriptors
- Fix missing `size` field initialization in macro STATS_DESC
[1] https://lore.kernel.org/kvm/20210402224359.2297157-1-jingzhangos@google.com
[2] https://lore.kernel.org/kvm/20210415151741.1607806-1-jingzhangos@google.com
[3] https://lore.kernel.org/kvm/20210423181727.596466-1-jingzhangos@google.com
[4] https://lore.kernel.org/kvm/20210429203740.1935629-1-jingzhangos@google.com
[5] https://lore.kernel.org/kvm/20210517145314.157626-1-jingzhangos@google.com
[6] https://lore.kernel.org/kvm/20210524151828.4113777-1-jingzhangos@google.com
[7] https://lore.kernel.org/kvm/20210603211426.790093-1-jingzhangos@google.com
[8] https://lore.kernel.org/kvm/20210611124624.1404010-1-jingzhangos@google.com
[9] https://lore.kernel.org/kvm/20210614212155.1670777-1-jingzhangos@google.com
[10] https://lore.kernel.org/kvm/20210617044146.2667540-1-jingzhangos@google.com
[11] https://lore.kernel.org/kvm/20210618044819.3690166-1-jingzhangos@google.com
---
Jing Zhang (7):
KVM: stats: Separate generic stats from architecture specific ones
KVM: stats: Add fd-based API to read binary stats data
KVM: stats: Support binary stats retrieval for a VM
KVM: stats: Support binary stats retrieval for a VCPU
KVM: stats: Add documentation for binary statistics interface
KVM: selftests: Add selftest for KVM statistics data binary interface
KVM: stats: Remove code duplication for binary and debugfs stats
Documentation/virt/kvm/api.rst | 198 ++++++++++++++-
arch/arm64/include/asm/kvm_host.h | 9 +-
arch/arm64/kvm/Makefile | 2 +-
arch/arm64/kvm/guest.c | 48 ++--
arch/mips/include/asm/kvm_host.h | 9 +-
arch/mips/kvm/Makefile | 2 +-
arch/mips/kvm/mips.c | 90 ++++---
arch/powerpc/include/asm/kvm_host.h | 9 +-
arch/powerpc/kvm/Makefile | 2 +-
arch/powerpc/kvm/book3s.c | 91 ++++---
arch/powerpc/kvm/book3s_hv.c | 12 +-
arch/powerpc/kvm/book3s_pr.c | 2 +-
arch/powerpc/kvm/book3s_pr_papr.c | 2 +-
arch/powerpc/kvm/booke.c | 76 ++++--
arch/s390/include/asm/kvm_host.h | 9 +-
arch/s390/kvm/Makefile | 3 +-
arch/s390/kvm/kvm-s390.c | 232 +++++++++--------
arch/x86/include/asm/kvm_host.h | 9 +-
arch/x86/kvm/Makefile | 2 +-
arch/x86/kvm/x86.c | 109 ++++----
include/linux/kvm_host.h | 182 ++++++++++++--
include/linux/kvm_types.h | 14 ++
include/uapi/linux/kvm.h | 73 ++++++
tools/testing/selftests/kvm/.gitignore | 1 +
tools/testing/selftests/kvm/Makefile | 3 +
.../testing/selftests/kvm/include/kvm_util.h | 3 +
.../selftests/kvm/kvm_binary_stats_test.c | 234 ++++++++++++++++++
tools/testing/selftests/kvm/lib/kvm_util.c | 12 +
virt/kvm/binary_stats.c | 144 +++++++++++
virt/kvm/kvm_main.c | 218 +++++++++++++---
30 files changed, 1443 insertions(+), 357 deletions(-)
create mode 100644 tools/testing/selftests/kvm/kvm_binary_stats_test.c
create mode 100644 virt/kvm/binary_stats.c
base-commit: f1b8325508327a302f1d5cd8a4bf51e2c9c72fa9
--
2.32.0.288.g62a8d224e6-goog
Hi,
This patch converts existing UUID runtime test to use KUnit framework.
Below, there's a comparison between the old output format and the new
one. Keep in mind that even if KUnit seems very verbose, this is the
corner case where _every_ test has failed.
* This is how the current output looks like in success:
test_uuid: all 18 tests passed
* And when it fails:
test_uuid: conversion test #1 failed on LE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
test_uuid: cmp test #2 failed on LE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
test_uuid: cmp test #2 actual data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
test_uuid: conversion test #3 failed on BE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
test_uuid: cmp test #4 failed on BE data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
test_uuid: cmp test #4 actual data: 'c33f4995-3701-450e-9fbf-206a2e98e576'
test_uuid: conversion test #5 failed on LE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
test_uuid: cmp test #6 failed on LE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
test_uuid: cmp test #6 actual data: '64b4371c-77c1-48f9-8221-29f054fc023b'
test_uuid: conversion test #7 failed on BE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
test_uuid: cmp test #8 failed on BE data: '64b4371c-77c1-48f9-8221-29f054fc023b'
test_uuid: cmp test #8 actual data: '64b4371c-77c1-48f9-8221-29f054fc023b'
test_uuid: conversion test #9 failed on LE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
test_uuid: cmp test #10 failed on LE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
test_uuid: cmp test #10 actual data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
test_uuid: conversion test #11 failed on BE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
test_uuid: cmp test #12 failed on BE data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
test_uuid: cmp test #12 actual data: '0cb4ddff-a545-4401-9d06-688af53e7f84'
test_uuid: negative test #13 passed on wrong LE data: 'c33f4995-3701-450e-9fbf206a2e98e576 '
test_uuid: negative test #14 passed on wrong BE data: 'c33f4995-3701-450e-9fbf206a2e98e576 '
test_uuid: negative test #15 passed on wrong LE data: '64b4371c-77c1-48f9-8221-29f054XX023b'
test_uuid: negative test #16 passed on wrong BE data: '64b4371c-77c1-48f9-8221-29f054XX023b'
test_uuid: negative test #17 passed on wrong LE data: '0cb4ddff-a545-4401-9d06-688af53e'
test_uuid: negative test #18 passed on wrong BE data: '0cb4ddff-a545-4401-9d06-688af53e'
test_uuid: failed 18 out of 18 tests
* Now, here's how it looks like with KUnit:
======== [PASSED] uuid ========
[PASSED] uuid_correct_be
[PASSED] uuid_correct_le
[PASSED] uuid_wrong_be
[PASSED] uuid_wrong_le
* And if every test fail with KUnit:
======== [FAILED] uuid ========
[FAILED] uuid_correct_be
# uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57
Expected uuid_parse(data->uuid, &be) == 1, but
uuid_parse(data->uuid, &be) == 0
failed to parse 'c33f4995-3701-450e-9fbf-206a2e98e576'
# uuid_correct_be: not ok 1 - c33f4995-3701-450e-9fbf-206a2e98e576
# uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57
Expected uuid_parse(data->uuid, &be) == 1, but
uuid_parse(data->uuid, &be) == 0
failed to parse '64b4371c-77c1-48f9-8221-29f054fc023b'
# uuid_correct_be: not ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b
# uuid_correct_be: ASSERTION FAILED at lib/test_uuid.c:57
Expected uuid_parse(data->uuid, &be) == 1, but
uuid_parse(data->uuid, &be) == 0
failed to parse '0cb4ddff-a545-4401-9d06-688af53e7f84'
# uuid_correct_be: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84
not ok 1 - uuid_correct_be
[FAILED] uuid_correct_le
# uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46
Expected guid_parse(data->uuid, &le) == 1, but
guid_parse(data->uuid, &le) == 0
failed to parse 'c33f4995-3701-450e-9fbf-206a2e98e576'
# uuid_correct_le: not ok 1 - c33f4995-3701-450e-9fbf-206a2e98e576
# uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46
Expected guid_parse(data->uuid, &le) == 1, but
guid_parse(data->uuid, &le) == 0
failed to parse '64b4371c-77c1-48f9-8221-29f054fc023b'
# uuid_correct_le: not ok 2 - 64b4371c-77c1-48f9-8221-29f054fc023b
# uuid_correct_le: ASSERTION FAILED at lib/test_uuid.c:46
Expected guid_parse(data->uuid, &le) == 1, but
guid_parse(data->uuid, &le) == 0
failed to parse '0cb4ddff-a545-4401-9d06-688af53e7f84'
# uuid_correct_le: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e7f84
not ok 2 - uuid_correct_le
[FAILED] uuid_wrong_be
# uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77
Expected uuid_parse(*data, &be) == 0, but
uuid_parse(*data, &be) == -22
parsing of 'c33f4995-3701-450e-9fbf206a2e98e576 ' should've failed
# uuid_wrong_be: not ok 1 - c33f4995-3701-450e-9fbf206a2e98e576
# uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77
Expected uuid_parse(*data, &be) == 0, but
uuid_parse(*data, &be) == -22
parsing of '64b4371c-77c1-48f9-8221-29f054XX023b' should've failed
# uuid_wrong_be: not ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b
# uuid_wrong_be: ASSERTION FAILED at lib/test_uuid.c:77
Expected uuid_parse(*data, &be) == 0, but
uuid_parse(*data, &be) == -22
parsing of '0cb4ddff-a545-4401-9d06-688af53e' should've failed
# uuid_wrong_be: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e
not ok 3 - uuid_wrong_be
[FAILED] uuid_wrong_le
# uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68
Expected guid_parse(*data, &le) == 0, but
guid_parse(*data, &le) == -22
parsing of 'c33f4995-3701-450e-9fbf206a2e98e576 ' should've failed
# uuid_wrong_le: not ok 1 - c33f4995-3701-450e-9fbf206a2e98e576
# uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68
Expected guid_parse(*data, &le) == 0, but
guid_parse(*data, &le) == -22
parsing of '64b4371c-77c1-48f9-8221-29f054XX023b' should've failed
# uuid_wrong_le: not ok 2 - 64b4371c-77c1-48f9-8221-29f054XX023b
# uuid_wrong_le: ASSERTION FAILED at lib/test_uuid.c:68
Expected guid_parse(*data, &le) == 0, but
guid_parse(*data, &le) == -22
parsing of '0cb4ddff-a545-4401-9d06-688af53e' should've failed
# uuid_wrong_le: not ok 3 - 0cb4ddff-a545-4401-9d06-688af53e
not ok 4 - uuid_wrong_le
Changes from v3:
- Drop unnecessary casts and braces.
- Simplify Kconfig entry
v3: https://lore.kernel.org/lkml/20210610163959.71634-1-andrealmeid@collabora.c…
Changes from v2:
- Clarify in commit message the new test cases setup
v2: https://lore.kernel.org/lkml/20210609233730.164082-1-andrealmeid@collabora.…
Changes from v1:
- Test suite name: uuid_test -> uuid
- Config name: TEST_UUID -> UUID_KUNIT_TEST
- Config entry in the Kconfig file left where it is
- Converted tests to use _MSG variant
v1: https://lore.kernel.org/lkml/20210605215215.171165-1-andrealmeid@collabora.…
André Almeida (1):
lib: Convert UUID runtime test to KUnit
lib/Kconfig.debug | 8 ++-
lib/Makefile | 2 +-
lib/test_uuid.c | 137 +++++++++++++++++++---------------------------
3 files changed, 64 insertions(+), 83 deletions(-)
--
2.31.1
Some environments (e.g. kerneci.org) do not set $SHELL for their test
environment. There's no need to use $SHELL here anyway, so just replace
it with hard-coded /bin/sh instead. Without this, the LKDTM tests would
never actually run on kerneci.org.
Fixes: 46d1a0f03d66 ("selftests/lkdtm: Add tests for LKDTM targets")
Cc: stable(a)vger.kernel.org
Signed-off-by: Kees Cook <keescook(a)chromium.org>
---
tools/testing/selftests/lkdtm/run.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/lkdtm/run.sh b/tools/testing/selftests/lkdtm/run.sh
index bb7a1775307b..968ff3cf5667 100755
--- a/tools/testing/selftests/lkdtm/run.sh
+++ b/tools/testing/selftests/lkdtm/run.sh
@@ -79,7 +79,7 @@ dmesg > "$DMESG"
# Most shells yell about signals and we're expecting the "cat" process
# to usually be killed by the kernel. So we have to run it in a sub-shell
# and silence errors.
-($SHELL -c 'cat <(echo '"$test"') >'"$TRIGGER" 2>/dev/null) || true
+(/bin/sh -c 'cat <(echo '"$test"') >'"$TRIGGER" 2>/dev/null) || true
# Record and dump the results
dmesg | comm --nocheck-order -13 "$DMESG" - > "$LOG" || true
--
2.25.1
The bidirectional test attempts to change the cipher to
TLS_CIPHER_AES_GCM_128. The test fixture setup will have already set
the cipher to be tested, and if it was different than the one set by
the bidir test setsockopt() will fail on account of having different
ciphers for rx and tx, causing the test to fail.
Forcing the use of GCM when testing ChaCha doesn't make sense anyway,
so just use the cipher configured by the test fixture setup.
Fixes: 4f336e88a870 ("selftests/tls: add CHACHA20-POLY1305 to tls selftests")
Signed-off-by: Seth Forshee <seth.forshee(a)canonical.com>
---
tools/testing/selftests/net/tls.c | 17 -----------------
1 file changed, 17 deletions(-)
diff --git a/tools/testing/selftests/net/tls.c b/tools/testing/selftests/net/tls.c
index 426d07875a48..9f4c87f4ce1e 100644
--- a/tools/testing/selftests/net/tls.c
+++ b/tools/testing/selftests/net/tls.c
@@ -831,23 +831,6 @@ TEST_F(tls, bidir)
char const *test_str = "test_read";
int send_len = 10;
char buf[10];
- int ret;
-
- if (!self->notls) {
- struct tls12_crypto_info_aes_gcm_128 tls12;
-
- memset(&tls12, 0, sizeof(tls12));
- tls12.info.version = variant->tls_version;
- tls12.info.cipher_type = TLS_CIPHER_AES_GCM_128;
-
- ret = setsockopt(self->fd, SOL_TLS, TLS_RX, &tls12,
- sizeof(tls12));
- ASSERT_EQ(ret, 0);
-
- ret = setsockopt(self->cfd, SOL_TLS, TLS_TX, &tls12,
- sizeof(tls12));
- ASSERT_EQ(ret, 0);
- }
ASSERT_EQ(strlen(test_str) + 1, send_len);
--
2.31.1
SRv6 End.DT46 Behavior is defined in the IETF RFC 8986 [1] along with SRv6
End.DT4 and End.DT6 Behaviors.
The proposed End.DT46 implementation is meant to support the decapsulation
of both IPv4 and IPv6 traffic coming from a *single* SRv6 tunnel.
The SRv6 End.DT46 Behavior greatly simplifies the setup and operations of
SRv6 VPNs in the Linux kernel.
- patch 1/2 is the core patch that adds support for the SRv6 End.DT46
Behavior;
- patch 2/2 adds the selftest for SRv6 End.DT46 Behavior.
The patch introducing the new SRv6 End.DT46 Behavior in iproute2 will
follow shortly.
Comments, suggestions and improvements are very welcome as always!
Thanks,
Andrea
RFC -> v1
patch 1/2, seg6: add support for SRv6 End.DT46 Behavior
- add Reviewed-by, thanks to David Ahern.
patch 2/2, selftests: seg6: add selftest for SRv6 End.DT46 Behavior
- add Acked-by, thanks to David Ahern.
[1] https://www.rfc-editor.org/rfc/rfc8986.html#name-enddt46-decapsulation-and-s
Andrea Mayer (2):
seg6: add support for SRv6 End.DT46 Behavior
selftests: seg6: add selftest for SRv6 End.DT46 Behavior
include/uapi/linux/seg6_local.h | 2 +
net/ipv6/seg6_local.c | 94 ++-
.../selftests/net/srv6_end_dt46_l3vpn_test.sh | 573 ++++++++++++++++++
3 files changed, 647 insertions(+), 22 deletions(-)
create mode 100755 tools/testing/selftests/net/srv6_end_dt46_l3vpn_test.sh
--
2.20.1
This patchset provides a file descriptor for every VM and VCPU to read
KVM statistics data in binary format.
It is meant to provide a lightweight, flexible, scalable and efficient
lock-free solution for user space telemetry applications to pull the
statistics data periodically for large scale systems. The pulling
frequency could be as high as a few times per second.
In this patchset, every statistics data are treated to have some
attributes as below:
* architecture dependent or generic
* VM statistics data or VCPU statistics data
* type: cumulative, instantaneous, peak
* unit: none for simple counter, nanosecond, microsecond,
millisecond, second, Byte, KiByte, MiByte, GiByte, Clock Cycles
Since no lock/synchronization is used, the consistency between all
the statistics data is not guaranteed. That means not all statistics
data are read out at the exact same time, since the statistics data
are still being updated by KVM subsystems while they are read out.
---
* v10 -> v11
- Rebase to kvm/queue, commit f1b832550832
(KVM: x86/mmu: Fix TDP MMU page table level)
- Separate binary stats implementation commit
- Use flexible length array member field in API structure instead of
zero-length array member field
- Move major binary stats reading function in a separate source file
- Move stats id string into vm/vcpu structures
- Add some detailed comments and update commit messages
- Addressed some other review comments from Greg K.H. and Paolo.
* v9 -> v10
- Relocate vcpu stat in vcpu's slab's usercopy region
- Fix test issue for capability checking
- Update commit message to explain why/how we need to add this new
API for KVM statistics
* v8 -> v9
- Rebase to commit 8331a2bc0898
(KVM: X86: Introduce KVM_HC_MAP_GPA_RANGE hypercall)
- Reduce code duplication between binary and debugfs interface
- Add field "offset" in stats descriptor to let us define stats
descriptors in any order (not necessary in the order of stats
defined in vm/vcpu stats structures)
- Add static check to make sure the number of stats descriptors
is the same as the number of stats defined in vm/vcpu stats
structures
- Fix missing/mismatched stats descriptor definition caused by
rebase
* v7 -> v8
- Rebase to kvm/queue, commit c1dc20e254b4 ("KVM: switch per-VM
stats to u64")
- Revise code to reflect the per-VM stats type from ulong to u64
- Addressed some other nits
* v6 -> v7
- Improve file descriptor allocation function by Krish suggestion
- Use "generic stats" instead of "common stats" as Krish suggested
- Addressed some other nits from Krish and David Matlack
* v5 -> v6
- Use designated initializers for STATS_DESC
- Change KVM_STATS_SCALE... to KVM_STATS_BASE...
- Use a common function for kvm_[vm|vcpu]_stats_read
- Fix some documentation errors/missings
- Use TEST_ASSERT in selftest
- Use a common function for [vm|vcpu]_stats_test in selftest
* v4 -> v5
- Rebase to kvm/queue, commit a4345a7cecfb ("Merge tag
'kvmarm-fixes-5.13-1'")
- Change maximum stats name length to 48
- Replace VM_STATS_COMMON/VCPU_STATS_COMMON macros with stats
descriptor definition macros.
- Fixed some errors/warnings reported by checkpatch.pl
* v3 -> v4
- Rebase to kvm/queue, commit 9f242010c3b4 ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Use C-stype comments in the whole patch
- Fix wrong count for x86 VCPU stats descriptors
- Fix KVM stats data size counting and validity check in selftest
* v2 -> v3
- Rebase to kvm/queue, commit edf408f5257b ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Resolve some nitpicks about format
* v1 -> v2
- Use ARRAY_SIZE to count the number of stats descriptors
- Fix missing `size` field initialization in macro STATS_DESC
[1] https://lore.kernel.org/kvm/20210402224359.2297157-1-jingzhangos@google.com
[2] https://lore.kernel.org/kvm/20210415151741.1607806-1-jingzhangos@google.com
[3] https://lore.kernel.org/kvm/20210423181727.596466-1-jingzhangos@google.com
[4] https://lore.kernel.org/kvm/20210429203740.1935629-1-jingzhangos@google.com
[5] https://lore.kernel.org/kvm/20210517145314.157626-1-jingzhangos@google.com
[6] https://lore.kernel.org/kvm/20210524151828.4113777-1-jingzhangos@google.com
[7] https://lore.kernel.org/kvm/20210603211426.790093-1-jingzhangos@google.com
[8] https://lore.kernel.org/kvm/20210611124624.1404010-1-jingzhangos@google.com
[9] https://lore.kernel.org/kvm/20210614212155.1670777-1-jingzhangos@google.com
[10] https://lore.kernel.org/kvm/20210617044146.2667540-1-jingzhangos@google.com
---
Jing Zhang (7):
KVM: stats: Separate generic stats from architecture specific ones
KVM: stats: Add fd-based API to read binary stats data
KVM: stats: Support binary stats retrieval for a VM
KVM: stats: Support binary stats retrieval for a VCPU
KVM: stats: Add documentation for binary statistics interface
KVM: selftests: Add selftest for KVM statistics data binary interface
KVM: stats: Remove code duplication for binary and debugfs stats
Documentation/virt/kvm/api.rst | 176 +++++++++++++-
arch/arm64/include/asm/kvm_host.h | 9 +-
arch/arm64/kvm/Makefile | 2 +-
arch/arm64/kvm/guest.c | 46 ++--
arch/mips/include/asm/kvm_host.h | 9 +-
arch/mips/kvm/Makefile | 2 +-
arch/mips/kvm/mips.c | 88 ++++---
arch/powerpc/include/asm/kvm_host.h | 9 +-
arch/powerpc/kvm/Makefile | 2 +-
arch/powerpc/kvm/book3s.c | 89 ++++---
arch/powerpc/kvm/book3s_hv.c | 12 +-
arch/powerpc/kvm/book3s_pr.c | 2 +-
arch/powerpc/kvm/book3s_pr_papr.c | 2 +-
arch/powerpc/kvm/booke.c | 74 ++++--
arch/s390/include/asm/kvm_host.h | 9 +-
arch/s390/kvm/Makefile | 3 +-
arch/s390/kvm/kvm-s390.c | 230 ++++++++++--------
arch/x86/include/asm/kvm_host.h | 9 +-
arch/x86/kvm/Makefile | 2 +-
arch/x86/kvm/x86.c | 107 ++++----
include/linux/kvm_host.h | 182 ++++++++++++--
include/linux/kvm_types.h | 12 +
include/uapi/linux/kvm.h | 44 ++++
tools/testing/selftests/kvm/.gitignore | 1 +
tools/testing/selftests/kvm/Makefile | 3 +
.../testing/selftests/kvm/include/kvm_util.h | 3 +
.../selftests/kvm/kvm_binary_stats_test.c | 225 +++++++++++++++++
tools/testing/selftests/kvm/lib/kvm_util.c | 12 +
virt/kvm/binary_stats.c | 130 ++++++++++
virt/kvm/kvm_main.c | 218 ++++++++++++++---
30 files changed, 1355 insertions(+), 357 deletions(-)
create mode 100644 tools/testing/selftests/kvm/kvm_binary_stats_test.c
create mode 100644 virt/kvm/binary_stats.c
base-commit: f1b8325508327a302f1d5cd8a4bf51e2c9c72fa9
--
2.32.0.288.g62a8d224e6-goog
I've observed that the tls multi_chunk_sendfile selftest hangs during
recv() and ultimately times out, and it seems to have done so even when
the test was first introduced. Reading through the commit message when
it was added (0e6fbe39bdf7 "net/tls(TLS_SW): Add selftest for 'chunked'
sendfile test") I get the impression that the test is meant to
demonstrate a problem with ktls, but there's no indication that the
problem has been fixed.
Am I right that the expectation is that this test will fail? If that's
the case, shouldn't this test be removed until the problem is fixed?
Thanks,
Seth
This patchset provides a file descriptor for every VM and VCPU to read
KVM statistics data in binary format.
It is meant to provide a lightweight, flexible, scalable and efficient
lock-free solution for user space telemetry applications to pull the
statistics data periodically for large scale systems. The pulling
frequency could be as high as a few times per second.
In this patchset, every statistics data are treated to have some
attributes as below:
* architecture dependent or generic
* VM statistics data or VCPU statistics data
* type: cumulative, instantaneous, peak
* unit: none for simple counter, nanosecond, microsecond,
millisecond, second, Byte, KiByte, MiByte, GiByte, Clock Cycles
Since no lock/synchronization is used, the consistency between all
the statistics data is not guaranteed. That means not all statistics
data are read out at the exact same time, since the statistics data
are still being updated by KVM subsystems while they are read out.
---
* v9 -> v10
- Relocate vcpu stat in vcpu's slab's usercopy region
- Fix test issue for capability checking
- Update commit message to explain why/how we need to add this new
API for KVM statistics
* v8 -> v9
- Rebase to commit 8331a2bc0898
(KVM: X86: Introduce KVM_HC_MAP_GPA_RANGE hypercall)
- Reduce code duplication between binary and debugfs interface
- Add field "offset" in stats descriptor to let us define stats
descriptors in any order (not necessary in the order of stats
defined in vm/vcpu stats structures)
- Add static check to make sure the number of stats descriptors
is the same as the number of stats defined in vm/vcpu stats
structures
- Fix missing/mismatched stats descriptor definition caused by
rebase
* v7 -> v8
- Rebase to kvm/queue, commit c1dc20e254b4 ("KVM: switch per-VM
stats to u64")
- Revise code to reflect the per-VM stats type from ulong to u64
- Addressed some other nits
* v6 -> v7
- Improve file descriptor allocation function by Krish suggestion
- Use "generic stats" instead of "common stats" as Krish suggested
- Addressed some other nits from Krish and David Matlack
* v5 -> v6
- Use designated initializers for STATS_DESC
- Change KVM_STATS_SCALE... to KVM_STATS_BASE...
- Use a common function for kvm_[vm|vcpu]_stats_read
- Fix some documentation errors/missings
- Use TEST_ASSERT in selftest
- Use a common function for [vm|vcpu]_stats_test in selftest
* v4 -> v5
- Rebase to kvm/queue, commit a4345a7cecfb ("Merge tag
'kvmarm-fixes-5.13-1'")
- Change maximum stats name length to 48
- Replace VM_STATS_COMMON/VCPU_STATS_COMMON macros with stats
descriptor definition macros.
- Fixed some errors/warnings reported by checkpatch.pl
* v3 -> v4
- Rebase to kvm/queue, commit 9f242010c3b4 ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Use C-stype comments in the whole patch
- Fix wrong count for x86 VCPU stats descriptors
- Fix KVM stats data size counting and validity check in selftest
* v2 -> v3
- Rebase to kvm/queue, commit edf408f5257b ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Resolve some nitpicks about format
* v1 -> v2
- Use ARRAY_SIZE to count the number of stats descriptors
- Fix missing `size` field initialization in macro STATS_DESC
[1] https://lore.kernel.org/kvm/20210402224359.2297157-1-jingzhangos@google.com
[2] https://lore.kernel.org/kvm/20210415151741.1607806-1-jingzhangos@google.com
[3] https://lore.kernel.org/kvm/20210423181727.596466-1-jingzhangos@google.com
[4] https://lore.kernel.org/kvm/20210429203740.1935629-1-jingzhangos@google.com
[5] https://lore.kernel.org/kvm/20210517145314.157626-1-jingzhangos@google.com
[6] https://lore.kernel.org/kvm/20210524151828.4113777-1-jingzhangos@google.com
[7] https://lore.kernel.org/kvm/20210603211426.790093-1-jingzhangos@google.com
[8] https://lore.kernel.org/kvm/20210611124624.1404010-1-jingzhangos@google.com
[9] https://lore.kernel.org/kvm/20210614212155.1670777-1-jingzhangos@google.com
---
Jing Zhang (5):
KVM: stats: Separate generic stats from architecture specific ones
KVM: stats: Add fd-based API to read binary stats data
KVM: stats: Add documentation for binary statistics interface
KVM: selftests: Add selftest for KVM statistics data binary interface
KVM: stats: Remove code duplication for binary and debugfs stats
Documentation/virt/kvm/api.rst | 177 +++++++++++-
arch/arm64/include/asm/kvm_host.h | 9 +-
arch/arm64/kvm/guest.c | 50 +++-
arch/mips/include/asm/kvm_host.h | 9 +-
arch/mips/kvm/mips.c | 92 +++---
arch/powerpc/include/asm/kvm_host.h | 9 +-
arch/powerpc/kvm/book3s.c | 93 ++++--
arch/powerpc/kvm/book3s_hv.c | 12 +-
arch/powerpc/kvm/book3s_pr.c | 2 +-
arch/powerpc/kvm/book3s_pr_papr.c | 2 +-
arch/powerpc/kvm/booke.c | 78 +++--
arch/s390/include/asm/kvm_host.h | 9 +-
arch/s390/kvm/kvm-s390.c | 234 ++++++++-------
arch/x86/include/asm/kvm_host.h | 9 +-
arch/x86/kvm/x86.c | 111 ++++---
include/linux/kvm_host.h | 180 +++++++++++-
include/linux/kvm_types.h | 12 +
include/uapi/linux/kvm.h | 48 ++++
tools/testing/selftests/kvm/.gitignore | 1 +
tools/testing/selftests/kvm/Makefile | 3 +
.../testing/selftests/kvm/include/kvm_util.h | 3 +
.../selftests/kvm/kvm_binary_stats_test.c | 225 +++++++++++++++
tools/testing/selftests/kvm/lib/kvm_util.c | 12 +
virt/kvm/kvm_main.c | 270 +++++++++++++++---
24 files changed, 1299 insertions(+), 351 deletions(-)
create mode 100644 tools/testing/selftests/kvm/kvm_binary_stats_test.c
base-commit: 8331a2bc089881d7fd2fc9a6658f39780817e4e0
--
2.32.0.272.g935e593368-goog
This patchset makes the following two major changes to the cpuset v2 code:
Patch 2: Add a new partition state "root-nolb" to create a partition
root with load balancing disabled. This is for handling intermitten
workloads that have a strict low latency requirement.
Patch 3: Allow partition roots that are not the top cpuset to distribute
all its cpus to child partitions as long as there is no task associated
with that partition root. This allows more flexibility for middleware
to manage multiple partitions.
Patch 4 updates the cgroup-v2.rst file accordingly. Patch 5 adds a test
to test the new cpuset partition code.
Waiman Long (5):
cgroup/cpuset: Don't call validate_change() for some flag changes
cgroup/cpuset: Add new cpus.partition type with no load balancing
cgroup/cpuset: Allow non-top parent partition root to distribute out
all CPUs
cgroup/cpuset: Update description of cpuset.cpus.partition in
cgroup-v2.rst
kselftest/cgroup: Add cpuset v2 partition root state test
Documentation/admin-guide/cgroup-v2.rst | 19 ++-
kernel/cgroup/cpuset.c | 124 +++++++++++----
tools/testing/selftests/cgroup/Makefile | 2 +-
.../selftests/cgroup/test_cpuset_prs.sh | 141 ++++++++++++++++++
4 files changed, 247 insertions(+), 39 deletions(-)
create mode 100755 tools/testing/selftests/cgroup/test_cpuset_prs.sh
--
2.18.1
This patchset provides a file descriptor for every VM and VCPU to read
KVM statistics data in binary format.
It is meant to provide a lightweight, flexible, scalable and efficient
lock-free solution for user space telemetry applications to pull the
statistics data periodically for large scale systems. The pulling
frequency could be as high as a few times per second.
In this patchset, every statistics data are treated to have some
attributes as below:
* architecture dependent or generic
* VM statistics data or VCPU statistics data
* type: cumulative, instantaneous, peak
* unit: none for simple counter, nanosecond, microsecond,
millisecond, second, Byte, KiByte, MiByte, GiByte. Clock Cycles
Since no lock/synchronization is used, the consistency between all
the statistics data is not guaranteed. That means not all statistics
data are read out at the exact same time, since the statistics date
are still being updated by KVM subsystems while they are read out.
---
* v8 -> v9
- Rebase to commit 8331a2bc0898
(KVM: X86: Introduce KVM_HC_MAP_GPA_RANGE hypercall)
- Reduce code duplication between binary and debugfs interface
- Add field "offset" in stats descriptor to let us define stats
descriptors in any order (not necessary in the order of stats
defined in vm/vcpu stats structures)
- Add static check to make sure the number of stats descriptors
is the same as the number of stats defined in vm/vcpu stats
structures
- Fix missing/mismatched stats descriptor definition caused by
rebase
* v7 -> v8
- Rebase to kvm/queue, commit c1dc20e254b4 ("KVM: switch per-VM
stats to u64")
- Revise code to reflect the per-VM stats type from ulong to u64
- Addressed some other nits
* v6 -> v7
- Improve file descriptor allocation function by Krish suggestion
- Use "generic stats" instead of "common stats" as Krish suggested
- Addressed some other nits from Krish and David Matlack
* v5 -> v6
- Use designated initializers for STATS_DESC
- Change KVM_STATS_SCALE... to KVM_STATS_BASE...
- Use a common function for kvm_[vm|vcpu]_stats_read
- Fix some documentation errors/missings
- Use TEST_ASSERT in selftest
- Use a common function for [vm|vcpu]_stats_test in selftest
* v4 -> v5
- Rebase to kvm/queue, commit a4345a7cecfb ("Merge tag
'kvmarm-fixes-5.13-1'")
- Change maximum stats name length to 48
- Replace VM_STATS_COMMON/VCPU_STATS_COMMON macros with stats
descriptor definition macros.
- Fixed some errors/warnings reported by checkpatch.pl
* v3 -> v4
- Rebase to kvm/queue, commit 9f242010c3b4 ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Use C-stype comments in the whole patch
- Fix wrong count for x86 VCPU stats descriptors
- Fix KVM stats data size counting and validity check in selftest
* v2 -> v3
- Rebase to kvm/queue, commit edf408f5257b ("KVM: avoid "deadlock"
between install_new_memslots and MMU notifier")
- Resolve some nitpicks about format
* v1 -> v2
- Use ARRAY_SIZE to count the number of stats descriptors
- Fix missing `size` field initialization in macro STATS_DESC
[1] https://lore.kernel.org/kvm/20210402224359.2297157-1-jingzhangos@google.com
[2] https://lore.kernel.org/kvm/20210415151741.1607806-1-jingzhangos@google.com
[3] https://lore.kernel.org/kvm/20210423181727.596466-1-jingzhangos@google.com
[4] https://lore.kernel.org/kvm/20210429203740.1935629-1-jingzhangos@google.com
[5] https://lore.kernel.org/kvm/20210517145314.157626-1-jingzhangos@google.com
[6] https://lore.kernel.org/kvm/20210524151828.4113777-1-jingzhangos@google.com
[7] https://lore.kernel.org/kvm/20210603211426.790093-1-jingzhangos@google.com
[8] https://lore.kernel.org/kvm/20210611124624.1404010-1-jingzhangos@google.com
---
Jing Zhang (5):
KVM: stats: Separate generic stats from architecture specific ones
KVM: stats: Add fd-based API to read binary stats data
KVM: stats: Add documentation for statistics data binary interface
KVM: selftests: Add selftest for KVM statistics data binary interface
KVM: stats: Remove code duplication for binary and debugfs stats
Documentation/virt/kvm/api.rst | 177 +++++++++++-
arch/arm64/include/asm/kvm_host.h | 9 +-
arch/arm64/kvm/guest.c | 50 +++-
arch/mips/include/asm/kvm_host.h | 9 +-
arch/mips/kvm/mips.c | 92 +++---
arch/powerpc/include/asm/kvm_host.h | 9 +-
arch/powerpc/kvm/book3s.c | 93 ++++---
arch/powerpc/kvm/book3s_hv.c | 12 +-
arch/powerpc/kvm/book3s_pr.c | 2 +-
arch/powerpc/kvm/book3s_pr_papr.c | 2 +-
arch/powerpc/kvm/booke.c | 78 ++++--
arch/s390/include/asm/kvm_host.h | 9 +-
arch/s390/kvm/kvm-s390.c | 234 +++++++++-------
arch/x86/include/asm/kvm_host.h | 9 +-
arch/x86/kvm/x86.c | 111 +++++---
include/linux/kvm_host.h | 178 +++++++++++-
include/linux/kvm_types.h | 12 +
include/uapi/linux/kvm.h | 48 ++++
tools/testing/selftests/kvm/.gitignore | 1 +
tools/testing/selftests/kvm/Makefile | 3 +
.../testing/selftests/kvm/include/kvm_util.h | 3 +
.../selftests/kvm/kvm_binary_stats_test.c | 225 +++++++++++++++
tools/testing/selftests/kvm/lib/kvm_util.c | 12 +
virt/kvm/kvm_main.c | 261 +++++++++++++++---
24 files changed, 1293 insertions(+), 346 deletions(-)
create mode 100644 tools/testing/selftests/kvm/kvm_binary_stats_test.c
base-commit: 8331a2bc089881d7fd2fc9a6658f39780817e4e0
--
2.32.0.272.g935e593368-goog
Note: this does not change the parser behavior at all (except for making
one error message more useful). This is just an internal refactor.
The TAP output parser currently operates over a List[str].
This works, but we only ever need to be able to "peek" at the current
line and the ability to "pop" it off.
Also, using a List means we need to wait for all the output before we
can start parsing. While this is not an issue for most tests which are
really lightweight, we do have some longer (~5 minutes) tests.
This patch introduces an LineStream wrapper class that
* Exposes a peek()/pop() interface instead of manipulating an array
* this allows us to more easily add debugging code [1]
* Can consume an input from a generator
* we can now parse results as tests are running (the parser code
currently doesn't print until the end, so no impact yet).
* Tracks the current line number to print better error messages
* Would allow us to add additional features more easily, e.g. storing
N previous lines so we can print out invalid lines in context, etc.
[1] The parsing logic is currently quite fragile.
E.g. it'll often say the kernel "CRASHED" if there's something slightly
wrong with the output format. When debugging a test that had some memory
corruption issues, it resulted in very misleading errors from the parser.
Now we could easily add this to trace all the lines consumed and why
+import inspect
...
def pop(self) -> str:
n = self._next
+ print(f'popping {n[0]}: {n[1].ljust(40, " ")}| caller={inspect.stack()[1].function}')
Example output:
popping 77: TAP version 14 | caller=parse_tap_header
popping 78: 1..1 | caller=parse_test_plan
popping 79: # Subtest: kunit_executor_test | caller=parse_subtest_header
popping 80: 1..2 | caller=parse_subtest_plan
popping 81: ok 1 - parse_filter_test | caller=parse_ok_not_ok_test_case
popping 82: ok 2 - filter_subsuite_test | caller=parse_ok_not_ok_test_case
popping 83: ok 1 - kunit_executor_test | caller=parse_ok_not_ok_test_suite
If we introduce an invalid line, we can see the parser go down the wrong path:
popping 77: TAP version 14 | caller=parse_tap_header
popping 78: 1..1 | caller=parse_test_plan
popping 79: # Subtest: kunit_executor_test | caller=parse_subtest_header
popping 80: 1..2 | caller=parse_subtest_plan
popping 81: 1..2 # this is invalid! | caller=parse_ok_not_ok_test_case
popping 82: ok 1 - parse_filter_test | caller=parse_ok_not_ok_test_case
popping 83: ok 2 - filter_subsuite_test | caller=parse_ok_not_ok_test_case
popping 84: ok 1 - kunit_executor_test | caller=parse_ok_not_ok_test_case
[ERROR] ran out of lines before end token
Signed-off-by: Daniel Latypov <dlatypov(a)google.com>
Reviewed-by: David Gow <davidgow(a)google.com>
---
v1 -> v2:
* class Input => class LineStream
* get_input() => extract_tap_lines()
---
tools/testing/kunit/kunit_parser.py | 136 ++++++++++++++++---------
tools/testing/kunit/kunit_tool_test.py | 18 ++--
2 files changed, 99 insertions(+), 55 deletions(-)
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index e8bcc139702e..370c0862cc1e 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -47,22 +47,63 @@ class TestStatus(Enum):
NO_TESTS = auto()
FAILURE_TO_PARSE_TESTS = auto()
+class LineStream:
+ """Provides a peek()/pop() interface over an iterator of (line#, text)."""
+ _lines: Iterator[Tuple[int, str]]
+ _next: Tuple[int, str]
+ _done: bool
+
+ def __init__(self, lines: Iterator[Tuple[int, str]]):
+ self._lines = lines
+ self._done = False
+ self._next = (0, '')
+ self._get_next()
+
+ def _get_next(self) -> None:
+ try:
+ self._next = next(self._lines)
+ except StopIteration:
+ self._done = True
+
+ def peek(self) -> str:
+ return self._next[1]
+
+ def pop(self) -> str:
+ n = self._next
+ self._get_next()
+ return n[1]
+
+ def __bool__(self) -> bool:
+ return not self._done
+
+ # Only used by kunit_tool_test.py.
+ def __iter__(self) -> Iterator[str]:
+ while bool(self):
+ yield self.pop()
+
+ def line_number(self) -> int:
+ return self._next[0]
+
kunit_start_re = re.compile(r'TAP version [0-9]+$')
kunit_end_re = re.compile('(List of all partitions:|'
'Kernel panic - not syncing: VFS:)')
-def isolate_kunit_output(kernel_output) -> Iterator[str]:
- started = False
- for line in kernel_output:
- line = line.rstrip() # line always has a trailing \n
- if kunit_start_re.search(line):
- prefix_len = len(line.split('TAP version')[0])
- started = True
- yield line[prefix_len:] if prefix_len > 0 else line
- elif kunit_end_re.search(line):
- break
- elif started:
- yield line[prefix_len:] if prefix_len > 0 else line
+def extract_tap_lines(kernel_output: Iterable[str]) -> LineStream:
+ def isolate_kunit_output(kernel_output: Iterable[str]) -> Iterator[Tuple[int, str]]:
+ line_num = 0
+ started = False
+ for line in kernel_output:
+ line_num += 1
+ line = line.rstrip() # line always has a trailing \n
+ if kunit_start_re.search(line):
+ prefix_len = len(line.split('TAP version')[0])
+ started = True
+ yield line_num, line[prefix_len:]
+ elif kunit_end_re.search(line):
+ break
+ elif started:
+ yield line_num, line[prefix_len:]
+ return LineStream(lines=isolate_kunit_output(kernel_output))
def raw_output(kernel_output) -> None:
for line in kernel_output:
@@ -97,14 +138,14 @@ def print_log(log) -> None:
TAP_ENTRIES = re.compile(r'^(TAP|[\s]*ok|[\s]*not ok|[\s]*[0-9]+\.\.[0-9]+|[\s]*#).*$')
-def consume_non_diagnostic(lines: List[str]) -> None:
- while lines and not TAP_ENTRIES.match(lines[0]):
- lines.pop(0)
+def consume_non_diagnostic(lines: LineStream) -> None:
+ while lines and not TAP_ENTRIES.match(lines.peek()):
+ lines.pop()
-def save_non_diagnostic(lines: List[str], test_case: TestCase) -> None:
- while lines and not TAP_ENTRIES.match(lines[0]):
- test_case.log.append(lines[0])
- lines.pop(0)
+def save_non_diagnostic(lines: LineStream, test_case: TestCase) -> None:
+ while lines and not TAP_ENTRIES.match(lines.peek()):
+ test_case.log.append(lines.peek())
+ lines.pop()
OkNotOkResult = namedtuple('OkNotOkResult', ['is_ok','description', 'text'])
@@ -112,18 +153,18 @@ OK_NOT_OK_SUBTEST = re.compile(r'^[\s]+(ok|not ok) [0-9]+ - (.*)$')
OK_NOT_OK_MODULE = re.compile(r'^(ok|not ok) ([0-9]+) - (.*)$')
-def parse_ok_not_ok_test_case(lines: List[str], test_case: TestCase) -> bool:
+def parse_ok_not_ok_test_case(lines: LineStream, test_case: TestCase) -> bool:
save_non_diagnostic(lines, test_case)
if not lines:
test_case.status = TestStatus.TEST_CRASHED
return True
- line = lines[0]
+ line = lines.peek()
match = OK_NOT_OK_SUBTEST.match(line)
while not match and lines:
- line = lines.pop(0)
+ line = lines.pop()
match = OK_NOT_OK_SUBTEST.match(line)
if match:
- test_case.log.append(lines.pop(0))
+ test_case.log.append(lines.pop())
test_case.name = match.group(2)
if test_case.status == TestStatus.TEST_CRASHED:
return True
@@ -138,14 +179,14 @@ def parse_ok_not_ok_test_case(lines: List[str], test_case: TestCase) -> bool:
SUBTEST_DIAGNOSTIC = re.compile(r'^[\s]+# (.*)$')
DIAGNOSTIC_CRASH_MESSAGE = re.compile(r'^[\s]+# .*?: kunit test case crashed!$')
-def parse_diagnostic(lines: List[str], test_case: TestCase) -> bool:
+def parse_diagnostic(lines: LineStream, test_case: TestCase) -> bool:
save_non_diagnostic(lines, test_case)
if not lines:
return False
- line = lines[0]
+ line = lines.peek()
match = SUBTEST_DIAGNOSTIC.match(line)
if match:
- test_case.log.append(lines.pop(0))
+ test_case.log.append(lines.pop())
crash_match = DIAGNOSTIC_CRASH_MESSAGE.match(line)
if crash_match:
test_case.status = TestStatus.TEST_CRASHED
@@ -153,7 +194,7 @@ def parse_diagnostic(lines: List[str], test_case: TestCase) -> bool:
else:
return False
-def parse_test_case(lines: List[str]) -> Optional[TestCase]:
+def parse_test_case(lines: LineStream) -> Optional[TestCase]:
test_case = TestCase()
save_non_diagnostic(lines, test_case)
while parse_diagnostic(lines, test_case):
@@ -165,24 +206,24 @@ def parse_test_case(lines: List[str]) -> Optional[TestCase]:
SUBTEST_HEADER = re.compile(r'^[\s]+# Subtest: (.*)$')
-def parse_subtest_header(lines: List[str]) -> Optional[str]:
+def parse_subtest_header(lines: LineStream) -> Optional[str]:
consume_non_diagnostic(lines)
if not lines:
return None
- match = SUBTEST_HEADER.match(lines[0])
+ match = SUBTEST_HEADER.match(lines.peek())
if match:
- lines.pop(0)
+ lines.pop()
return match.group(1)
else:
return None
SUBTEST_PLAN = re.compile(r'[\s]+[0-9]+\.\.([0-9]+)')
-def parse_subtest_plan(lines: List[str]) -> Optional[int]:
+def parse_subtest_plan(lines: LineStream) -> Optional[int]:
consume_non_diagnostic(lines)
- match = SUBTEST_PLAN.match(lines[0])
+ match = SUBTEST_PLAN.match(lines.peek())
if match:
- lines.pop(0)
+ lines.pop()
return int(match.group(1))
else:
return None
@@ -199,17 +240,17 @@ def max_status(left: TestStatus, right: TestStatus) -> TestStatus:
else:
return TestStatus.SUCCESS
-def parse_ok_not_ok_test_suite(lines: List[str],
+def parse_ok_not_ok_test_suite(lines: LineStream,
test_suite: TestSuite,
expected_suite_index: int) -> bool:
consume_non_diagnostic(lines)
if not lines:
test_suite.status = TestStatus.TEST_CRASHED
return False
- line = lines[0]
+ line = lines.peek()
match = OK_NOT_OK_MODULE.match(line)
if match:
- lines.pop(0)
+ lines.pop()
if match.group(1) == 'ok':
test_suite.status = TestStatus.SUCCESS
else:
@@ -231,7 +272,7 @@ def bubble_up_test_case_errors(test_suite: TestSuite) -> TestStatus:
max_test_case_status = bubble_up_errors(x.status for x in test_suite.cases)
return max_status(max_test_case_status, test_suite.status)
-def parse_test_suite(lines: List[str], expected_suite_index: int) -> Optional[TestSuite]:
+def parse_test_suite(lines: LineStream, expected_suite_index: int) -> Optional[TestSuite]:
if not lines:
return None
consume_non_diagnostic(lines)
@@ -257,26 +298,26 @@ def parse_test_suite(lines: List[str], expected_suite_index: int) -> Optional[Te
print_with_timestamp(red('[ERROR] ') + 'ran out of lines before end token')
return test_suite
else:
- print('failed to parse end of suite' + lines[0])
+ print(f'failed to parse end of suite "{name}", at line {lines.line_number()}: {lines.peek()}')
return None
TAP_HEADER = re.compile(r'^TAP version 14$')
-def parse_tap_header(lines: List[str]) -> bool:
+def parse_tap_header(lines: LineStream) -> bool:
consume_non_diagnostic(lines)
- if TAP_HEADER.match(lines[0]):
- lines.pop(0)
+ if TAP_HEADER.match(lines.peek()):
+ lines.pop()
return True
else:
return False
TEST_PLAN = re.compile(r'[0-9]+\.\.([0-9]+)')
-def parse_test_plan(lines: List[str]) -> Optional[int]:
+def parse_test_plan(lines: LineStream) -> Optional[int]:
consume_non_diagnostic(lines)
- match = TEST_PLAN.match(lines[0])
+ match = TEST_PLAN.match(lines.peek())
if match:
- lines.pop(0)
+ lines.pop()
return int(match.group(1))
else:
return None
@@ -284,7 +325,7 @@ def parse_test_plan(lines: List[str]) -> Optional[int]:
def bubble_up_suite_errors(test_suites: Iterable[TestSuite]) -> TestStatus:
return bubble_up_errors(x.status for x in test_suites)
-def parse_test_result(lines: List[str]) -> TestResult:
+def parse_test_result(lines: LineStream) -> TestResult:
consume_non_diagnostic(lines)
if not lines or not parse_tap_header(lines):
return TestResult(TestStatus.NO_TESTS, [], lines)
@@ -338,11 +379,12 @@ def print_and_count_results(test_result: TestResult) -> Tuple[int, int, int]:
print_with_timestamp('')
return total_tests, failed_tests, crashed_tests
-def parse_run_tests(kernel_output) -> TestResult:
+def parse_run_tests(kernel_output: Iterable[str]) -> TestResult:
total_tests = 0
failed_tests = 0
crashed_tests = 0
- test_result = parse_test_result(list(isolate_kunit_output(kernel_output)))
+ lines = extract_tap_lines(kernel_output)
+ test_result = parse_test_result(lines)
if test_result.status == TestStatus.NO_TESTS:
print(red('[ERROR] ') + yellow('no tests run!'))
elif test_result.status == TestStatus.FAILURE_TO_PARSE_TESTS:
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index 2e809dd956a7..433cd41d951c 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -11,6 +11,7 @@ from unittest import mock
import tempfile, shutil # Handling test_tmpdir
+import itertools
import json
import signal
import os
@@ -92,17 +93,18 @@ class KconfigTest(unittest.TestCase):
class KUnitParserTest(unittest.TestCase):
- def assertContains(self, needle, haystack):
- for line in haystack:
+ def assertContains(self, needle: str, haystack: kunit_parser.LineStream):
+ # Clone the iterator so we can print the contents on failure.
+ copy, backup = itertools.tee(haystack)
+ for line in copy:
if needle in line:
return
- raise AssertionError('"' +
- str(needle) + '" not found in "' + str(haystack) + '"!')
+ raise AssertionError(f'"{needle}" not found in {list(backup)}!')
def test_output_isolated_correctly(self):
log_path = test_data_path('test_output_isolated_correctly.log')
with open(log_path) as file:
- result = kunit_parser.isolate_kunit_output(file.readlines())
+ result = kunit_parser.extract_tap_lines(file.readlines())
self.assertContains('TAP version 14', result)
self.assertContains(' # Subtest: example', result)
self.assertContains(' 1..2', result)
@@ -113,7 +115,7 @@ class KUnitParserTest(unittest.TestCase):
def test_output_with_prefix_isolated_correctly(self):
log_path = test_data_path('test_pound_sign.log')
with open(log_path) as file:
- result = kunit_parser.isolate_kunit_output(file.readlines())
+ result = kunit_parser.extract_tap_lines(file.readlines())
self.assertContains('TAP version 14', result)
self.assertContains(' # Subtest: kunit-resource-test', result)
self.assertContains(' 1..5', result)
@@ -159,7 +161,7 @@ class KUnitParserTest(unittest.TestCase):
empty_log = test_data_path('test_is_test_passed-no_tests_run.log')
with open(empty_log) as file:
result = kunit_parser.parse_run_tests(
- kunit_parser.isolate_kunit_output(file.readlines()))
+ kunit_parser.extract_tap_lines(file.readlines()))
self.assertEqual(0, len(result.suites))
self.assertEqual(
kunit_parser.TestStatus.NO_TESTS,
@@ -170,7 +172,7 @@ class KUnitParserTest(unittest.TestCase):
print_mock = mock.patch('builtins.print').start()
with open(crash_log) as file:
result = kunit_parser.parse_run_tests(
- kunit_parser.isolate_kunit_output(file.readlines()))
+ kunit_parser.extract_tap_lines(file.readlines()))
print_mock.assert_any_call(StrContains('no tests run!'))
print_mock.stop()
file.close()
base-commit: c3d0e3fd41b7f0f5d5d5b6022ab7e813f04ea727
--
2.31.1.818.g46aad6cb9e-goog
Make the default .kunitconfig (specified in
arch/um/configs/kunit_defconfig) specify CONFIG_KUNIT_ALL_TESTS by
default. KUNIT_ALL_TESTS runs all tests which have satisfied
dependencies in the current .config (which would be the architecture
defconfig).
Currently, the default .kunitconfig enables only the example tests and
KUnit's own tests. While this does provide a good example of what a
.kunitconfig for running a few individual tests should look like, it
does mean that kunit_tool runs a pretty paltry collection of tests by
default.
The example tests' config entry (CONFIG_KUNIT_EXAMPLE_TEST=y) continues
to be included -- despite now being redundant -- to provide an example
of how tests are enabled when KUNIT_ALL_TESTS is disabled.
A default run of ./tools/testing/kunit/kunit.py run now runs 70 tests
instead of 14.
Signed-off-by: David Gow <davidgow(a)google.com>
Acked-by: Daniel Latypov <dlatypov(a)google.com>
Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com>
---
Changes since v1:
https://lore.kernel.org/linux-kselftest/20210518035825.1885357-1-davidgow@g…
- Keep the KUNIT_EXAMPLE_TEST entry as an example.
- Move (in patches 2,3) kunit_defconfig to tools/testing/kunit/configs
and replace all_tests.config.
arch/um/configs/kunit_defconfig | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/um/configs/kunit_defconfig b/arch/um/configs/kunit_defconfig
index 9235b7d42d38..e67af7b9f1bb 100644
--- a/arch/um/configs/kunit_defconfig
+++ b/arch/um/configs/kunit_defconfig
@@ -1,3 +1,3 @@
CONFIG_KUNIT=y
-CONFIG_KUNIT_TEST=y
CONFIG_KUNIT_EXAMPLE_TEST=y
+CONFIG_KUNIT_ALL_TESTS=y
--
2.31.1.818.g46aad6cb9e-goog
veth.sh is a shell script that uses /bin/sh; some distro (Ubuntu for
example) use dash as /bin/sh and in this case the test reports the
following error:
# ./veth.sh: 21: local: -r: bad variable name
# ./veth.sh: 21: local: -r: bad variable name
This happens because dash doesn't support the option "-r" with local.
Moreover, in case of missing bpf object, the script is exiting -1, that
is an illegal number for dash:
exit: Illegal number: -1
Change the script to be compatible both with bash and dash and prevent
the errors above.
Signed-off-by: Andrea Righi <andrea.righi(a)canonical.com>
---
tools/testing/selftests/net/veth.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/tools/testing/selftests/net/veth.sh b/tools/testing/selftests/net/veth.sh
index 2fedc0781ce8..11d7cdb898c0 100755
--- a/tools/testing/selftests/net/veth.sh
+++ b/tools/testing/selftests/net/veth.sh
@@ -18,7 +18,8 @@ ret=0
cleanup() {
local ns
- local -r jobs="$(jobs -p)"
+ local jobs
+ readonly jobs="$(jobs -p)"
[ -n "${jobs}" ] && kill -1 ${jobs} 2>/dev/null
rm -f $STATS
@@ -108,7 +109,7 @@ chk_gro() {
if [ ! -f ../bpf/xdp_dummy.o ]; then
echo "Missing xdp_dummy helper. Build bpf selftest first"
- exit -1
+ exit 1
fi
create_ns
--
2.31.1
udpgro_fwd.sh contains many bash specific operators ("[[", "local -r"),
but it's using /bin/sh; in some distro /bin/sh is mapped to /bin/dash,
that doesn't support such operators.
Force the test to use /bin/bash explicitly and prevent false positive
test failures.
Signed-off-by: Andrea Righi <andrea.righi(a)canonical.com>
---
tools/testing/selftests/net/udpgro_fwd.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tools/testing/selftests/net/udpgro_fwd.sh b/tools/testing/selftests/net/udpgro_fwd.sh
index a8fa64136282..7f26591f236b 100755
--- a/tools/testing/selftests/net/udpgro_fwd.sh
+++ b/tools/testing/selftests/net/udpgro_fwd.sh
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
# SPDX-License-Identifier: GPL-2.0
readonly BASE="ns-$(mktemp -u XXXXXX)"
--
2.31.1
The use of typecheck() in KUNIT_EXPECT_EQ() and friends is causing more
problems than I think it's worth. Things like enums need to have their
values explicitly cast, and literals all need to be very precisely
typed, else a large warning will be printed.
While typechecking does have its uses, the additional overhead of having
lots of needless casts -- combined with the awkward error messages which
don't mention which types are involved -- makes tests less readable and
more difficult to write.
By removing the typecheck() call, the two arguments still need to be of
compatible types, but don't need to be of exactly the same time, which
seems a less confusing and more useful compromise.
Signed-off-by: David Gow <davidgow(a)google.com>
Reviewed-by: Daniel Latypov <dlatypov(a)google.com>
Reviewed-by: Brendan Higgins <brendanhiggins(a)google.com>
---
Changes since v1:
https://lore.kernel.org/linux-kselftest/20210507050908.1008686-1-davidgow@g…
- Tidy up the patch description to note that a warning was being
produced, not an error.
- Add additional patches to remove many of the now unnecessary casts.
include/kunit/test.h | 1 -
1 file changed, 1 deletion(-)
diff --git a/include/kunit/test.h b/include/kunit/test.h
index 49601c4b98b8..4c56ffcb7403 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -775,7 +775,6 @@ void kunit_do_assertion(struct kunit *test,
do { \
typeof(left) __left = (left); \
typeof(right) __right = (right); \
- ((void)__typecheck(__left, __right)); \
\
KUNIT_ASSERTION(test, \
__left op __right, \
--
2.31.1.751.gd2f1c929bd-goog
As discussed at:
https://lore.kernel.org/linux-doc/871r9k6rmy.fsf@meer.lwn.net/
It is better to avoid using :doc:`foo` to refer to Documentation/foo.rst, as the
automarkup.py extension should handle it automatically, on most cases.
There are a couple of exceptions to this rule:
1. when :doc: tag is used to point to a kernel-doc DOC: markup;
2. when it is used with a named tag, e. g. :doc:`some name <foo>`;
It should also be noticed that automarkup.py has currently an issue:
if one use a markup like:
Documentation/dev-tools/kunit/api/test.rst
- documents all of the standard testing API excluding mocking
or mocking related features.
or, even:
Documentation/dev-tools/kunit/api/test.rst
documents all of the standard testing API excluding mocking
or mocking related features.
The automarkup.py will simply ignore it. Not sure why. This patch series
avoid the above patterns (which is present only on 4 files), but it would be
nice to have a followup patch fixing the issue at automarkup.py.
On this series:
Patch 1 manually adjust the references inside driver-api/pm/devices.rst,
as there it uses :file:`foo` to refer to some Documentation/ files;
Patch 2 converts a table at Documentation/dev-tools/kunit/api/index.rst
into a list, carefully avoiding the
Patch 3 converts the cross-references at the media documentation, also
avoiding the automarkup.py bug;
Patches 4-34 convert the other occurrences via a replace script. They were
manually edited, in order to honour 80-columns where possible.
I did a diff between the Sphinx 2.4.4 output before and after this patch
series in order to double-check that all converted Documentation/
references will produce <a href=<foo>.rst>foo title</a> tags.
Mauro Carvalho Chehab (34):
docs: devices.rst: better reference documentation docs
docs: dev-tools: kunit: don't use a table for docs name
media: docs: */media/index.rst: don't use ReST doc:`foo`
media: userspace-api: avoid using ReST :doc:`foo` markup
media: driver-api: drivers: avoid using ReST :doc:`foo` markup
media: admin-guide: avoid using ReST :doc:`foo` markup
docs: admin-guide: pm: avoid using ReSt :doc:`foo` markup
docs: admin-guide: hw-vuln: avoid using ReST :doc:`foo` markup
docs: admin-guide: sysctl: avoid using ReST :doc:`foo` markup
docs: block: biodoc.rst: avoid using ReSt :doc:`foo` markup
docs: bpf: bpf_lsm.rst: avoid using ReSt :doc:`foo` markup
docs: core-api: avoid using ReSt :doc:`foo` markup
docs: dev-tools: testing-overview.rst: avoid using ReSt :doc:`foo`
markup
docs: dev-tools: kunit: avoid using ReST :doc:`foo` markup
docs: devicetree: bindings: submitting-patches.rst: avoid using ReSt
:doc:`foo` markup
docs: doc-guide: avoid using ReSt :doc:`foo` markup
docs: driver-api: avoid using ReSt :doc:`foo` markup
docs: driver-api: gpio: using-gpio.rst: avoid using ReSt :doc:`foo`
markup
docs: driver-api: surface_aggregator: avoid using ReSt :doc:`foo`
markup
docs: driver-api: usb: avoid using ReSt :doc:`foo` markup
docs: firmware-guide: acpi: avoid using ReSt :doc:`foo` markup
docs: hwmon: adm1177.rst: avoid using ReSt :doc:`foo` markup
docs: i2c: avoid using ReSt :doc:`foo` markup
docs: kernel-hacking: hacking.rst: avoid using ReSt :doc:`foo` markup
docs: networking: devlink: avoid using ReSt :doc:`foo` markup
docs: PCI: endpoint: pci-endpoint-cfs.rst: avoid using ReSt :doc:`foo`
markup
docs: PCI: pci.rst: avoid using ReSt :doc:`foo` markup
docs: process: submitting-patches.rst: avoid using ReSt :doc:`foo`
markup
docs: security: landlock.rst: avoid using ReSt :doc:`foo` markup
docs: trace: coresight: coresight.rst: avoid using ReSt :doc:`foo`
markup
docs: trace: ftrace.rst: avoid using ReSt :doc:`foo` markup
docs: userspace-api: landlock.rst: avoid using ReSt :doc:`foo` markup
docs: virt: kvm: s390-pv-boot.rst: avoid using ReSt :doc:`foo` markup
docs: x86: avoid using ReSt :doc:`foo` markup
.../PCI/endpoint/pci-endpoint-cfs.rst | 2 +-
Documentation/PCI/pci.rst | 6 +--
.../special-register-buffer-data-sampling.rst | 3 +-
Documentation/admin-guide/media/bt8xx.rst | 15 ++++----
Documentation/admin-guide/media/bttv.rst | 21 ++++++-----
Documentation/admin-guide/media/index.rst | 12 +++---
Documentation/admin-guide/media/saa7134.rst | 3 +-
Documentation/admin-guide/pm/intel_idle.rst | 16 +++++---
Documentation/admin-guide/pm/intel_pstate.rst | 9 +++--
Documentation/admin-guide/sysctl/abi.rst | 2 +-
Documentation/admin-guide/sysctl/kernel.rst | 37 ++++++++++---------
Documentation/block/biodoc.rst | 2 +-
Documentation/bpf/bpf_lsm.rst | 13 ++++---
.../core-api/bus-virt-phys-mapping.rst | 2 +-
Documentation/core-api/dma-api.rst | 5 ++-
Documentation/core-api/dma-isa-lpc.rst | 2 +-
Documentation/core-api/index.rst | 4 +-
Documentation/dev-tools/kunit/api/index.rst | 8 ++--
Documentation/dev-tools/kunit/faq.rst | 2 +-
Documentation/dev-tools/kunit/index.rst | 14 +++----
Documentation/dev-tools/kunit/start.rst | 6 +--
Documentation/dev-tools/kunit/tips.rst | 5 ++-
Documentation/dev-tools/kunit/usage.rst | 8 ++--
Documentation/dev-tools/testing-overview.rst | 16 ++++----
.../bindings/submitting-patches.rst | 11 +++---
Documentation/doc-guide/contributing.rst | 8 ++--
Documentation/driver-api/gpio/using-gpio.rst | 4 +-
Documentation/driver-api/ioctl.rst | 2 +-
.../driver-api/media/drivers/bttv-devel.rst | 2 +-
Documentation/driver-api/media/index.rst | 10 +++--
Documentation/driver-api/pm/devices.rst | 8 ++--
.../surface_aggregator/clients/index.rst | 3 +-
.../surface_aggregator/internal.rst | 15 ++++----
.../surface_aggregator/overview.rst | 6 ++-
Documentation/driver-api/usb/dma.rst | 6 +--
.../acpi/dsd/data-node-references.rst | 3 +-
.../firmware-guide/acpi/dsd/graph.rst | 2 +-
.../firmware-guide/acpi/enumeration.rst | 7 ++--
Documentation/hwmon/adm1177.rst | 3 +-
Documentation/i2c/instantiating-devices.rst | 2 +-
Documentation/i2c/old-module-parameters.rst | 3 +-
Documentation/i2c/smbus-protocol.rst | 4 +-
Documentation/kernel-hacking/hacking.rst | 4 +-
.../networking/devlink/devlink-region.rst | 2 +-
.../networking/devlink/devlink-trap.rst | 4 +-
Documentation/process/submitting-patches.rst | 32 ++++++++--------
Documentation/security/landlock.rst | 3 +-
Documentation/trace/coresight/coresight.rst | 8 ++--
Documentation/trace/ftrace.rst | 2 +-
Documentation/userspace-api/landlock.rst | 11 +++---
.../userspace-api/media/glossary.rst | 2 +-
Documentation/userspace-api/media/index.rst | 12 +++---
Documentation/virt/kvm/s390-pv-boot.rst | 2 +-
Documentation/x86/boot.rst | 4 +-
Documentation/x86/mtrr.rst | 2 +-
55 files changed, 217 insertions(+), 183 deletions(-)
--
2.31.1