2022-02-11 19:25:34 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
|
|
|
|
|
|
|
|
|
//! The `kernel` crate.
|
|
|
|
|
//!
|
|
|
|
|
//! This crate contains the kernel APIs that have been ported or wrapped for
|
|
|
|
|
//! usage by Rust code in the kernel and is shared by all of them.
|
|
|
|
|
//!
|
|
|
|
|
//! In other words, all the rest of the Rust code in the kernel (e.g. kernel
|
|
|
|
|
//! modules written in Rust) depends on [`core`], [`alloc`] and this crate.
|
|
|
|
|
//!
|
|
|
|
|
//! If you need a kernel C API that is not ported or wrapped yet here, then
|
|
|
|
|
//! do so first instead of bypassing this crate.
|
|
|
|
|
|
|
|
|
|
#![no_std]
|
2024-09-15 13:26:31 +00:00
|
|
|
#![feature(arbitrary_self_types)]
|
2022-12-28 06:03:42 +00:00
|
|
|
#![feature(coerce_unsized)]
|
2022-12-28 06:03:46 +00:00
|
|
|
#![feature(dispatch_from_dyn)]
|
rust: alloc: implement kernel `Vec` type
`Vec` provides a contiguous growable array type with contents allocated
with the kernel's allocators (e.g. `Kmalloc`, `Vmalloc` or `KVmalloc`).
In contrast to Rust's stdlib `Vec` type, the kernel `Vec` type considers
the kernel's GFP flags for all appropriate functions, always reports
allocation failures through `Result<_, AllocError>` and remains
independent from unstable features.
[ This patch starts using a new unstable feature, `inline_const`, but
it was stabilized in Rust 1.79.0, i.e. the next version after the
minimum one, thus it will not be an issue. - Miguel ]
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Danilo Krummrich <dakr@kernel.org>
Link: https://lore.kernel.org/r/20241004154149.93856-17-dakr@kernel.org
[ Cleaned `rustdoc` unescaped backtick warning, added a couple more
backticks elsewhere, fixed typos, sorted `feature`s, rewrapped
documentation lines. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-10-04 15:41:20 +00:00
|
|
|
#![feature(inline_const)]
|
rust: start using the `#[expect(...)]` attribute
In Rust, it is possible to `allow` particular warnings (diagnostics,
lints) locally, making the compiler ignore instances of a given warning
within a given function, module, block, etc.
It is similar to `#pragma GCC diagnostic push` + `ignored` + `pop` in C:
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-function"
static void f(void) {}
#pragma GCC diagnostic pop
But way less verbose:
#[allow(dead_code)]
fn f() {}
By that virtue, it makes it possible to comfortably enable more
diagnostics by default (i.e. outside `W=` levels) that may have some
false positives but that are otherwise quite useful to keep enabled to
catch potential mistakes.
The `#[expect(...)]` attribute [1] takes this further, and makes the
compiler warn if the diagnostic was _not_ produced. For instance, the
following will ensure that, when `f()` is called somewhere, we will have
to remove the attribute:
#[expect(dead_code)]
fn f() {}
If we do not, we get a warning from the compiler:
warning: this lint expectation is unfulfilled
--> x.rs:3:10
|
3 | #[expect(dead_code)]
| ^^^^^^^^^
|
= note: `#[warn(unfulfilled_lint_expectations)]` on by default
This means that `expect`s do not get forgotten when they are not needed.
See the next commit for more details, nuances on its usage and
documentation on the feature.
The attribute requires the `lint_reasons` [2] unstable feature, but it
is becoming stable in 1.81.0 (to be released on 2024-09-05) and it has
already been useful to clean things up in this patch series, finding
cases where the `allow`s should not have been there.
Thus, enable `lint_reasons` and convert some of our `allow`s to `expect`s
where possible.
This feature was also an example of the ongoing collaboration between
Rust and the kernel -- we tested it in the kernel early on and found an
issue that was quickly resolved [3].
Cc: Fridtjof Stoldt <xfrednet@gmail.com>
Cc: Urgau <urgau@numericable.fr>
Link: https://rust-lang.github.io/rfcs/2383-lint-reasons.html#expect-lint-attribute [1]
Link: https://github.com/rust-lang/rust/issues/54503 [2]
Link: https://github.com/rust-lang/rust/issues/114557 [3]
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Trevor Gross <tmgross@umich.edu>
Tested-by: Gary Guo <gary@garyguo.net>
Reviewed-by: Gary Guo <gary@garyguo.net>
Link: https://lore.kernel.org/r/20240904204347.168520-18-ojeda@kernel.org
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-09-04 20:43:45 +00:00
|
|
|
#![feature(lint_reasons)]
|
2022-12-28 06:03:42 +00:00
|
|
|
#![feature(unsize)]
|
2022-02-11 19:25:34 +00:00
|
|
|
|
|
|
|
|
// Ensure conditional compilation based on the kernel configuration works;
|
|
|
|
|
// otherwise we may silently break things like initcall handling.
|
|
|
|
|
#[cfg(not(CONFIG_RUST))]
|
|
|
|
|
compile_error!("Missing kernel configuration for conditional compilation");
|
|
|
|
|
|
rust: add pin-init API core
This API is used to facilitate safe pinned initialization of structs. It
replaces cumbersome `unsafe` manual initialization with elegant safe macro
invocations.
Due to the size of this change it has been split into six commits:
1. This commit introducing the basic public interface: traits and
functions to represent and create initializers.
2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
`try_init!` macros along with their internal types.
3. Adds the `InPlaceInit` trait that allows using an initializer to create
an object inside of a `Box<T>` and other smart pointers.
4. Adds the `PinnedDrop` trait and adds macro support for it in
the `#[pin_data]` macro.
5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
the stack.
6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
types that have `0x00` in all bytes as a valid bit pattern.
--
In this section the problem that the new pin-init API solves is outlined.
This message describes the entirety of the API, not just the parts
introduced in this commit. For a more granular explanation and additional
information on pinning and this issue, view [1].
Pinning is Rust's way of enforcing the address stability of a value. When a
value gets pinned it will be impossible for safe code to move it to another
location. This is done by wrapping pointers to said object with `Pin<P>`.
This wrapper prevents safe code from creating mutable references to the
object, preventing mutable access, which is needed to move the value.
`Pin<P>` provides `unsafe` functions to circumvent this and allow
modifications regardless. It is then the programmer's responsibility to
uphold the pinning guarantee.
Many kernel data structures require a stable address, because there are
foreign pointers to them which would get invalidated by moving the
structure. Since these data structures are usually embedded in structs to
use them, this pinning property propagates to the container struct.
Resulting in most structs in both Rust and C code needing to be pinned.
So if we want to have a `mutex` field in a Rust struct, this struct also
needs to be pinned, because a `mutex` contains a `list_head`. Additionally
initializing a `list_head` requires already having the final memory
location available, because it is initialized by pointing it to itself. But
this presents another challenge in Rust: values have to be initialized at
all times. There is the `MaybeUninit<T>` wrapper type, which allows
handling uninitialized memory, but this requires using the `unsafe` raw
pointers and a casting the type to the initialized variant.
This problem gets exacerbated when considering encapsulation and the normal
safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
not be accessible to normal driver code. After all if anyone can modify
the fields, there is no way to ensure the invariants of the `Mutex<T>` are
upheld. But if the fields are inaccessible, then initialization of a
`Mutex<T>` needs to be somehow achieved via a function or a macro. Because
the `Mutex<T>` must be pinned in memory, the function cannot return it by
value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
that is an unnecessary allocation and indirection which would hurt
performance.
The solution in the rust tree (e.g. this commit: [2]) that is replaced by
this API is to split this function into two parts:
1. A `new` function that returns a partially initialized `Mutex<T>`,
2. An `init` function that requires the `Mutex<T>` to be pinned and that
fully initializes the `Mutex<T>`.
Both of these functions have to be marked `unsafe`, since a call to `new`
needs to be accompanied with a call to `init`, otherwise using the
`Mutex<T>` could result in UB. And because calling `init` twice also is not
safe. While `Mutex<T>` initialization cannot fail, other structs might
also have to allocate memory, which would result in conditional successful
initialization requiring even more manual accommodation work.
Combine this with the problem of pin-projections -- the way of accessing
fields of a pinned struct -- which also have an `unsafe` API, pinned
initialization is riddled with `unsafe` resulting in very poor ergonomics.
Not only that, but also having to call two functions possibly multiple
lines apart makes it very easy to forget it outright or during refactoring.
Here is an example of the current way of initializing a struct with two
synchronization primitives (see [3] for the full example):
struct SharedState {
state_changed: CondVar,
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn try_new() -> Result<Arc<Self>> {
let mut state = Pin::from(UniqueArc::try_new(Self {
// SAFETY: `condvar_init!` is called below.
state_changed: unsafe { CondVar::new() },
// SAFETY: `mutex_init!` is called below.
inner: unsafe {
Mutex::new(SharedStateInner { token_count: 0 })
},
})?);
// SAFETY: `state_changed` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
};
kernel::condvar_init!(pinned, "SharedState::state_changed");
// SAFETY: `inner` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.inner)
};
kernel::mutex_init!(pinned, "SharedState::inner");
Ok(state.into())
}
}
The pin-init API of this patch solves this issue by providing a
comprehensive solution comprised of macros and traits. Here is the example
from above using the pin-init API:
#[pin_data]
struct SharedState {
#[pin]
state_changed: CondVar,
#[pin]
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn new() -> impl PinInit<Self> {
pin_init!(Self {
state_changed <- new_condvar!("SharedState::state_changed"),
inner <- new_mutex!(
SharedStateInner { token_count: 0 },
"SharedState::inner",
),
})
}
}
Notably the way the macro is used here requires no `unsafe` and thus comes
with the usual Rust promise of safe code not introducing any memory
violations. Additionally it is now up to the caller of `new()` to decide
the memory location of the `SharedState`. They can choose at the moment
`Arc<T>`, `Box<T>` or the stack.
--
The API has the following architecture:
1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
closures.
2. Macros to create these initializer traits safely.
3. Functions to allow manually writing initializers.
The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
to uninitialized memory and their job is to fully initialize a `T` at that
location. If initialization fails, they return an error (`E`) by value.
This way of initializing cannot be safely exposed to the user, since it
relies upon these properties outside of the control of the trait:
- the memory location (slot) needs to be valid memory,
- if initialization fails, the slot should not be read from,
- the value in the slot should be pinned, so it cannot move and the memory
cannot be deallocated until the value is dropped.
This is why using an initializer is facilitated by another trait that
ensures these requirements.
These initializers can be created manually by just supplying a closure that
fulfills the same safety requirements as `PinInit<T, E>`. But this is an
`unsafe` operation. To allow safe initializer creation, the `pin_init!` is
provided along with three other variants: `try_pin_init!`, `try_init!` and
`init!`. These take a modified struct initializer as a parameter and
generate a closure that initializes the fields in sequence.
The macros take great care in upholding the safety requirements:
- A shadowed struct type is used as the return type of the closure instead
of `()`. This is to prevent early returns, as these would prevent full
initialization.
- To ensure every field is only initialized once, a normal struct
initializer is placed in unreachable code. The type checker will emit
errors if a field is missing or specified multiple times.
- When initializing a field fails, the whole initializer will fail and
automatically drop fields that have been initialized earlier.
- Only the correct initializer type is allowed for unpinned fields. You
cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
field.
To ensure the last point, an additional macro `#[pin_data]` is needed. This
macro annotates the struct itself and the user specifies structurally
pinned and not pinned fields.
Because dropping a pinned struct is also not allowed to break the pinning
invariants, another macro attribute `#[pinned_drop]` is needed. This
macro is introduced in a following commit.
These two macros also have mechanisms to ensure the overall safety of the
API. Additionally, they utilize a combined proc-macro, declarative macro
design: first a proc-macro enables the outer attribute syntax `#[...]` and
does some important pre-parsing. Notably this prepares the generics such
that the declarative macro can handle them using token trees. Then the
actual parsing of the structure and the emission of code is handled by a
declarative macro.
For pin-projections the crates `pin-project` [4] and `pin-project-lite` [5]
had been considered, but were ultimately rejected:
- `pin-project` depends on `syn` [6] which is a very big dependency, around
50k lines of code.
- `pin-project-lite` is a more reasonable 5k lines of code, but contains a
very complex declarative macro to parse generics. On top of that it
would require modification that would need to be maintained
independently.
Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
Link: https://github.com/Rust-for-Linux/linux/tree/0a04dc4ddd671efb87eef54dde0fb38e9074f4be [2]
Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [3]
Link: https://crates.io/crates/pin-project [4]
Link: https://crates.io/crates/pin-project-lite [5]
Link: https://crates.io/crates/syn [6]
Co-developed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@samsung.com>
Link: https://lore.kernel.org/r/20230408122429.1103522-7-y86-dev@protonmail.com
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2023-04-08 12:25:45 +00:00
|
|
|
// Allow proc-macros to refer to `::kernel` inside the `kernel` crate (this crate).
|
|
|
|
|
extern crate self as kernel;
|
|
|
|
|
|
2024-09-13 21:29:23 +00:00
|
|
|
pub use ffi;
|
|
|
|
|
|
2024-03-28 01:35:54 +00:00
|
|
|
pub mod alloc;
|
2024-06-11 11:45:49 +00:00
|
|
|
#[cfg(CONFIG_BLOCK)]
|
|
|
|
|
pub mod block;
|
2022-11-10 16:41:38 +00:00
|
|
|
mod build_assert;
|
2024-09-15 14:31:30 +00:00
|
|
|
pub mod cred;
|
2024-06-18 15:48:34 +00:00
|
|
|
pub mod device;
|
2022-02-11 19:25:34 +00:00
|
|
|
pub mod error;
|
2024-06-18 15:48:35 +00:00
|
|
|
#[cfg(CONFIG_RUST_FW_LOADER_ABSTRACTIONS)]
|
|
|
|
|
pub mod firmware;
|
rust: file: add Rust abstraction for `struct file`
This abstraction makes it possible to manipulate the open files for a
process. The new `File` struct wraps the C `struct file`. When accessing
it using the smart pointer `ARef<File>`, the pointer will own a
reference count to the file. When accessing it as `&File`, then the
reference does not own a refcount, but the borrow checker will ensure
that the reference count does not hit zero while the `&File` is live.
Since this is intended to manipulate the open files of a process, we
introduce an `fget` constructor that corresponds to the C `fget`
method. In future patches, it will become possible to create a new fd in
a process and bind it to a `File`. Rust Binder will use these to send
fds from one process to another.
We also provide a method for accessing the file's flags. Rust Binder
will use this to access the flags of the Binder fd to check whether the
non-blocking flag is set, which affects what the Binder ioctl does.
This introduces a struct for the EBADF error type, rather than just
using the Error type directly. This has two advantages:
* `File::fget` returns a `Result<ARef<File>, BadFdError>`, which the
compiler will represent as a single pointer, with null being an error.
This is possible because the compiler understands that `BadFdError`
has only one possible value, and it also understands that the
`ARef<File>` smart pointer is guaranteed non-null.
* Additionally, we promise to users of the method that the method can
only fail with EBADF, which means that they can rely on this promise
without having to inspect its implementation.
That said, there are also two disadvantages:
* Defining additional error types involves boilerplate.
* The question mark operator will only utilize the `From` trait once,
which prevents you from using the question mark operator on
`BadFdError` in methods that return some third error type that the
kernel `Error` is convertible into. (However, it works fine in methods
that return `Error`.)
Signed-off-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Co-developed-by: Daniel Xu <dxu@dxuuu.xyz>
Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240915-alice-file-v10-3-88484f7a3dcf@google.com
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Christian Brauner <brauner@kernel.org>
2024-09-15 14:31:29 +00:00
|
|
|
pub mod fs;
|
rust: add pin-init API core
This API is used to facilitate safe pinned initialization of structs. It
replaces cumbersome `unsafe` manual initialization with elegant safe macro
invocations.
Due to the size of this change it has been split into six commits:
1. This commit introducing the basic public interface: traits and
functions to represent and create initializers.
2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
`try_init!` macros along with their internal types.
3. Adds the `InPlaceInit` trait that allows using an initializer to create
an object inside of a `Box<T>` and other smart pointers.
4. Adds the `PinnedDrop` trait and adds macro support for it in
the `#[pin_data]` macro.
5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
the stack.
6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
types that have `0x00` in all bytes as a valid bit pattern.
--
In this section the problem that the new pin-init API solves is outlined.
This message describes the entirety of the API, not just the parts
introduced in this commit. For a more granular explanation and additional
information on pinning and this issue, view [1].
Pinning is Rust's way of enforcing the address stability of a value. When a
value gets pinned it will be impossible for safe code to move it to another
location. This is done by wrapping pointers to said object with `Pin<P>`.
This wrapper prevents safe code from creating mutable references to the
object, preventing mutable access, which is needed to move the value.
`Pin<P>` provides `unsafe` functions to circumvent this and allow
modifications regardless. It is then the programmer's responsibility to
uphold the pinning guarantee.
Many kernel data structures require a stable address, because there are
foreign pointers to them which would get invalidated by moving the
structure. Since these data structures are usually embedded in structs to
use them, this pinning property propagates to the container struct.
Resulting in most structs in both Rust and C code needing to be pinned.
So if we want to have a `mutex` field in a Rust struct, this struct also
needs to be pinned, because a `mutex` contains a `list_head`. Additionally
initializing a `list_head` requires already having the final memory
location available, because it is initialized by pointing it to itself. But
this presents another challenge in Rust: values have to be initialized at
all times. There is the `MaybeUninit<T>` wrapper type, which allows
handling uninitialized memory, but this requires using the `unsafe` raw
pointers and a casting the type to the initialized variant.
This problem gets exacerbated when considering encapsulation and the normal
safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
not be accessible to normal driver code. After all if anyone can modify
the fields, there is no way to ensure the invariants of the `Mutex<T>` are
upheld. But if the fields are inaccessible, then initialization of a
`Mutex<T>` needs to be somehow achieved via a function or a macro. Because
the `Mutex<T>` must be pinned in memory, the function cannot return it by
value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
that is an unnecessary allocation and indirection which would hurt
performance.
The solution in the rust tree (e.g. this commit: [2]) that is replaced by
this API is to split this function into two parts:
1. A `new` function that returns a partially initialized `Mutex<T>`,
2. An `init` function that requires the `Mutex<T>` to be pinned and that
fully initializes the `Mutex<T>`.
Both of these functions have to be marked `unsafe`, since a call to `new`
needs to be accompanied with a call to `init`, otherwise using the
`Mutex<T>` could result in UB. And because calling `init` twice also is not
safe. While `Mutex<T>` initialization cannot fail, other structs might
also have to allocate memory, which would result in conditional successful
initialization requiring even more manual accommodation work.
Combine this with the problem of pin-projections -- the way of accessing
fields of a pinned struct -- which also have an `unsafe` API, pinned
initialization is riddled with `unsafe` resulting in very poor ergonomics.
Not only that, but also having to call two functions possibly multiple
lines apart makes it very easy to forget it outright or during refactoring.
Here is an example of the current way of initializing a struct with two
synchronization primitives (see [3] for the full example):
struct SharedState {
state_changed: CondVar,
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn try_new() -> Result<Arc<Self>> {
let mut state = Pin::from(UniqueArc::try_new(Self {
// SAFETY: `condvar_init!` is called below.
state_changed: unsafe { CondVar::new() },
// SAFETY: `mutex_init!` is called below.
inner: unsafe {
Mutex::new(SharedStateInner { token_count: 0 })
},
})?);
// SAFETY: `state_changed` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
};
kernel::condvar_init!(pinned, "SharedState::state_changed");
// SAFETY: `inner` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.inner)
};
kernel::mutex_init!(pinned, "SharedState::inner");
Ok(state.into())
}
}
The pin-init API of this patch solves this issue by providing a
comprehensive solution comprised of macros and traits. Here is the example
from above using the pin-init API:
#[pin_data]
struct SharedState {
#[pin]
state_changed: CondVar,
#[pin]
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn new() -> impl PinInit<Self> {
pin_init!(Self {
state_changed <- new_condvar!("SharedState::state_changed"),
inner <- new_mutex!(
SharedStateInner { token_count: 0 },
"SharedState::inner",
),
})
}
}
Notably the way the macro is used here requires no `unsafe` and thus comes
with the usual Rust promise of safe code not introducing any memory
violations. Additionally it is now up to the caller of `new()` to decide
the memory location of the `SharedState`. They can choose at the moment
`Arc<T>`, `Box<T>` or the stack.
--
The API has the following architecture:
1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
closures.
2. Macros to create these initializer traits safely.
3. Functions to allow manually writing initializers.
The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
to uninitialized memory and their job is to fully initialize a `T` at that
location. If initialization fails, they return an error (`E`) by value.
This way of initializing cannot be safely exposed to the user, since it
relies upon these properties outside of the control of the trait:
- the memory location (slot) needs to be valid memory,
- if initialization fails, the slot should not be read from,
- the value in the slot should be pinned, so it cannot move and the memory
cannot be deallocated until the value is dropped.
This is why using an initializer is facilitated by another trait that
ensures these requirements.
These initializers can be created manually by just supplying a closure that
fulfills the same safety requirements as `PinInit<T, E>`. But this is an
`unsafe` operation. To allow safe initializer creation, the `pin_init!` is
provided along with three other variants: `try_pin_init!`, `try_init!` and
`init!`. These take a modified struct initializer as a parameter and
generate a closure that initializes the fields in sequence.
The macros take great care in upholding the safety requirements:
- A shadowed struct type is used as the return type of the closure instead
of `()`. This is to prevent early returns, as these would prevent full
initialization.
- To ensure every field is only initialized once, a normal struct
initializer is placed in unreachable code. The type checker will emit
errors if a field is missing or specified multiple times.
- When initializing a field fails, the whole initializer will fail and
automatically drop fields that have been initialized earlier.
- Only the correct initializer type is allowed for unpinned fields. You
cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
field.
To ensure the last point, an additional macro `#[pin_data]` is needed. This
macro annotates the struct itself and the user specifies structurally
pinned and not pinned fields.
Because dropping a pinned struct is also not allowed to break the pinning
invariants, another macro attribute `#[pinned_drop]` is needed. This
macro is introduced in a following commit.
These two macros also have mechanisms to ensure the overall safety of the
API. Additionally, they utilize a combined proc-macro, declarative macro
design: first a proc-macro enables the outer attribute syntax `#[...]` and
does some important pre-parsing. Notably this prepares the generics such
that the declarative macro can handle them using token trees. Then the
actual parsing of the structure and the emission of code is handled by a
declarative macro.
For pin-projections the crates `pin-project` [4] and `pin-project-lite` [5]
had been considered, but were ultimately rejected:
- `pin-project` depends on `syn` [6] which is a very big dependency, around
50k lines of code.
- `pin-project-lite` is a more reasonable 5k lines of code, but contains a
very complex declarative macro to parse generics. On top of that it
would require modification that would need to be maintained
independently.
Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
Link: https://github.com/Rust-for-Linux/linux/tree/0a04dc4ddd671efb87eef54dde0fb38e9074f4be [2]
Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [3]
Link: https://crates.io/crates/pin-project [4]
Link: https://crates.io/crates/pin-project-lite [5]
Link: https://crates.io/crates/syn [6]
Co-developed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@samsung.com>
Link: https://lore.kernel.org/r/20230408122429.1103522-7-y86-dev@protonmail.com
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2023-04-08 12:25:45 +00:00
|
|
|
pub mod init;
|
2023-04-03 09:33:53 +00:00
|
|
|
pub mod ioctl;
|
2024-10-30 16:04:24 +00:00
|
|
|
pub mod jump_label;
|
rust: support running Rust documentation tests as KUnit ones
Rust has documentation tests: these are typically examples of
usage of any item (e.g. function, struct, module...).
They are very convenient because they are just written
alongside the documentation. For instance:
/// Sums two numbers.
///
/// ```
/// assert_eq!(mymod::f(10, 20), 30);
/// ```
pub fn f(a: i32, b: i32) -> i32 {
a + b
}
In userspace, the tests are collected and run via `rustdoc`.
Using the tool as-is would be useful already, since it allows
to compile-test most tests (thus enforcing they are kept
in sync with the code they document) and run those that do not
depend on in-kernel APIs.
However, by transforming the tests into a KUnit test suite,
they can also be run inside the kernel. Moreover, the tests
get to be compiled as other Rust kernel objects instead of
targeting userspace.
On top of that, the integration with KUnit means the Rust
support gets to reuse the existing testing facilities. For
instance, the kernel log would look like:
KTAP version 1
1..1
KTAP version 1
# Subtest: rust_doctests_kernel
1..59
# rust_doctest_kernel_build_assert_rs_0.location: rust/kernel/build_assert.rs:13
ok 1 rust_doctest_kernel_build_assert_rs_0
# rust_doctest_kernel_build_assert_rs_1.location: rust/kernel/build_assert.rs:56
ok 2 rust_doctest_kernel_build_assert_rs_1
# rust_doctest_kernel_init_rs_0.location: rust/kernel/init.rs:122
ok 3 rust_doctest_kernel_init_rs_0
...
# rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
ok 59 rust_doctest_kernel_types_rs_2
# rust_doctests_kernel: pass:59 fail:0 skip:0 total:59
# Totals: pass:59 fail:0 skip:0 total:59
ok 1 rust_doctests_kernel
Therefore, add support for running Rust documentation tests
in KUnit. Some other notes about the current implementation
and support follow.
The transformation is performed by a couple scripts written
as Rust hostprogs.
Tests using the `?` operator are also supported as usual, e.g.:
/// ```
/// # use kernel::{spawn_work_item, workqueue};
/// spawn_work_item!(workqueue::system(), || pr_info!("x"))?;
/// # Ok::<(), Error>(())
/// ```
The tests are also compiled with Clippy under `CLIPPY=1`, just
like normal code, thus also benefitting from extra linting.
The names of the tests are currently automatically generated.
This allows to reduce the burden for documentation writers,
while keeping them fairly stable for bisection. This is an
improvement over the `rustdoc`-generated names, which include
the line number; but ideally we would like to get `rustdoc` to
provide the Rust item path and a number (for multiple examples
in a single documented Rust item).
In order for developers to easily see from which original line
a failed doctests came from, a KTAP diagnostic line is printed
to the log, containing the location (file and line) of the
original test (i.e. instead of the location in the generated
Rust file):
# rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
This line follows the syntax for declaring test metadata in the
proposed KTAP v2 spec [1], which may be used for the proposed
KUnit test attributes API [2]. Thus hopefully this will make
migration easier later on (suggested by David [3]).
The original line in that test attribute is figured out by
providing an anchor (suggested by Boqun [4]). The original file
is found by walking the filesystem, checking directory prefixes
to reduce the amount of combinations to check, and it is only
done once per file. Ambiguities are detected and reported.
A notable difference from KUnit C tests is that the Rust tests
appear to assert using the usual `assert!` and `assert_eq!`
macros from the Rust standard library (`core`). We provide
a custom version that forwards the call to KUnit instead.
Importantly, these macros do not require passing context,
unlike the KUnit C ones (i.e. `struct kunit *`). This makes
them easier to use, and readers of the documentation do not need
to care about which testing framework is used. In addition, it
may allow us to test third-party code more easily in the future.
However, a current limitation is that KUnit does not support
assertions in other tasks. Thus we presently simply print an
error to the kernel log if an assertion actually failed. This
should be revisited to properly fail the test, perhaps saving
the context somewhere else, or letting KUnit handle it.
Link: https://lore.kernel.org/lkml/20230420205734.1288498-1-rmoar@google.com/ [1]
Link: https://lore.kernel.org/linux-kselftest/20230707210947.1208717-1-rmoar@google.com/ [2]
Link: https://lore.kernel.org/rust-for-linux/CABVgOSkOLO-8v6kdAGpmYnZUb+LKOX0CtYCo-Bge7r_2YTuXDQ@mail.gmail.com/ [3]
Link: https://lore.kernel.org/rust-for-linux/ZIps86MbJF%2FiGIzd@boqun-archlinux/ [4]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2023-07-18 05:27:51 +00:00
|
|
|
#[cfg(CONFIG_KUNIT)]
|
|
|
|
|
pub mod kunit;
|
rust: list: add ListArc
The `ListArc` type can be thought of as a special reference to a
refcounted object that owns the permission to manipulate the
`next`/`prev` pointers stored in the refcounted object. By ensuring that
each object has only one `ListArc` reference, the owner of that
reference is assured exclusive access to the `next`/`prev` pointers.
When a `ListArc` is inserted into a `List`, the `List` takes ownership
of the `ListArc` reference.
There are various strategies for ensuring that a value has only one
`ListArc` reference. The simplest is to convert a `UniqueArc` into a
`ListArc`. However, the refcounted object could also keep track of
whether a `ListArc` exists using a boolean, which could allow for the
creation of new `ListArc` references from an `Arc` reference. Whatever
strategy is used, the relevant tracking is referred to as "the tracking
inside `T`", and the `ListArcSafe` trait (and its subtraits) are used to
update the tracking when a `ListArc` is created or destroyed.
Note that we allow the case where the tracking inside `T` thinks that a
`ListArc` exists, but actually, there isn't a `ListArc`. However, we do
not allow the opposite situation where a `ListArc` exists, but the
tracking thinks it doesn't. This is because the former can at most
result in us failing to create a `ListArc` when the operation could
succeed, whereas the latter can result in the creation of two `ListArc`
references. Only the latter situation can lead to memory safety issues.
This patch introduces the `impl_list_arc_safe!` macro that allows you to
implement `ListArcSafe` for types using the strategy where a `ListArc`
can only be created from a `UniqueArc`. Other strategies are introduced
in later patches.
This is part of the linked list that Rust Binder will use for many
different things. The strategy where a `ListArc` can only be created
from a `UniqueArc` is actually sufficient for most of the objects that
Rust Binder needs to insert into linked lists. Usually, these are todo
items that are created and then immediately inserted into a queue.
The const generic ID allows objects to have several prev/next pointer
pairs so that the same object can be inserted into several different
lists. You are able to have several `ListArc` references as long as they
correspond to different pointer pairs. The ID itself is purely a
compile-time concept and will not be present in the final binary. Both
the `List` and the `ListArc` will need to agree on the ID for them to
work together. Rust Binder uses this in a few places (e.g. death
recipients) where the same object can be inserted into both generic todo
lists and some other lists for tracking the status of the object.
The ID is a const generic rather than a type parameter because the
`pair_from_unique` method needs to be able to assert that the two ids
are different. There's no easy way to assert that when using types
instead of integers.
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240814-linked-list-v5-2-f5f5e8075da0@google.com
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-08-14 08:05:21 +00:00
|
|
|
pub mod list;
|
2024-10-01 08:22:22 +00:00
|
|
|
pub mod miscdevice;
|
2023-12-13 00:42:08 +00:00
|
|
|
#[cfg(CONFIG_NET)]
|
|
|
|
|
pub mod net;
|
rust: add abstraction for `struct page`
Adds a new struct called `Page` that wraps a pointer to `struct page`.
This struct is assumed to hold ownership over the page, so that Rust
code can allocate and manage pages directly.
The page type has various methods for reading and writing into the page.
These methods will temporarily map the page to allow the operation. All
of these methods use a helper that takes an offset and length, performs
bounds checks, and returns a pointer to the given offset in the page.
This patch only adds support for pages of order zero, as that is all
Rust Binder needs. However, it is written to make it easy to add support
for higher-order pages in the future. To do that, you would add a const
generic parameter to `Page` that specifies the order. Most of the
methods do not need to be adjusted, as the logic for dealing with
mapping multiple pages at once can be isolated to just the
`with_pointer_into_page` method.
Rust Binder needs to manage pages directly as that is how transactions
are delivered: Each process has an mmap'd region for incoming
transactions. When an incoming transaction arrives, the Binder driver
will choose a region in the mmap, allocate and map the relevant pages
manually, and copy the incoming transaction directly into the page. This
architecture allows the driver to copy transactions directly from the
address space of one process to another, without an intermediate copy
to a kernel buffer.
This code is based on Wedson's page abstractions from the old rust
branch, but it has been modified by Alice by removing the incomplete
support for higher-order pages, by introducing the `with_*` helpers
to consolidate the bounds checking logic into a single place, and
various other changes.
Co-developed-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Signed-off-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@samsung.com>
Reviewed-by: Trevor Gross <tmgross@umich.edu>
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240528-alice-mm-v7-4-78222c31b8f4@google.com
[ Fixed typos and added a few intra-doc links. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-05-28 14:58:05 +00:00
|
|
|
pub mod page;
|
2024-10-02 11:38:10 +00:00
|
|
|
pub mod pid_namespace;
|
2022-02-11 19:25:34 +00:00
|
|
|
pub mod prelude;
|
|
|
|
|
pub mod print;
|
2024-08-22 16:37:53 +00:00
|
|
|
pub mod rbtree;
|
2024-09-15 14:31:31 +00:00
|
|
|
pub mod security;
|
rust: add seqfile abstraction
This adds a simple seq file abstraction that lets you print to a seq
file using ordinary Rust printing syntax.
An example user from Rust Binder:
pub(crate) fn full_debug_print(
&self,
m: &SeqFile,
owner_inner: &mut ProcessInner,
) -> Result<()> {
let prio = self.node_prio();
let inner = self.inner.access_mut(owner_inner);
seq_print!(
m,
" node {}: u{:016x} c{:016x} pri {}:{} hs {} hw {} cs {} cw {}",
self.debug_id,
self.ptr,
self.cookie,
prio.sched_policy,
prio.prio,
inner.strong.has_count,
inner.weak.has_count,
inner.strong.count,
inner.weak.count,
);
if !inner.refs.is_empty() {
seq_print!(m, " proc");
for node_ref in &inner.refs {
seq_print!(m, " {}", node_ref.process.task.pid());
}
}
seq_print!(m, "\n");
for t in &inner.oneway_todo {
t.debug_print_inner(m, " pending async transaction ");
}
Ok(())
}
The `SeqFile` type is marked not thread safe so that `call_printf` can
be a `&self` method. The alternative is to use `self: Pin<&mut Self>`
which is inconvenient, or to have `SeqFile` wrap a pointer instead of
wrapping the C struct directly.
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20241001-seqfile-v1-1-dfcd0fc21e96@google.com
Signed-off-by: Christian Brauner <brauner@kernel.org>
2024-10-01 09:07:02 +00:00
|
|
|
pub mod seq_file;
|
2024-09-26 12:47:51 +00:00
|
|
|
pub mod sizes;
|
2022-11-10 16:41:36 +00:00
|
|
|
mod static_assert;
|
2022-11-10 16:41:35 +00:00
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub mod std_vendor;
|
2022-02-11 19:25:34 +00:00
|
|
|
pub mod str;
|
2022-12-28 06:03:40 +00:00
|
|
|
pub mod sync;
|
2023-04-11 05:45:39 +00:00
|
|
|
pub mod task;
|
2024-01-08 14:49:58 +00:00
|
|
|
pub mod time;
|
2024-10-30 16:04:25 +00:00
|
|
|
pub mod tracepoint;
|
2024-09-18 22:51:14 +00:00
|
|
|
pub mod transmute;
|
2022-11-10 16:41:39 +00:00
|
|
|
pub mod types;
|
rust: uaccess: add userspace pointers
A pointer to an area in userspace memory, which can be either read-only
or read-write.
All methods on this struct are safe: attempting to read or write on bad
addresses (either out of the bound of the slice or unmapped addresses)
will return `EFAULT`. Concurrent access, *including data races to/from
userspace memory*, is permitted, because fundamentally another userspace
thread/process could always be modifying memory at the same time (in the
same way that userspace Rust's `std::io` permits data races with the
contents of files on disk). In the presence of a race, the exact byte
values read/written are unspecified but the operation is well-defined.
Kernelspace code should validate its copy of data after completing a
read, and not expect that multiple reads of the same address will return
the same value.
These APIs are designed to make it difficult to accidentally write
TOCTOU bugs. Every time you read from a memory location, the pointer is
advanced by the length so that you cannot use that reader to read the
same memory location twice. Preventing double-fetches avoids TOCTOU
bugs. This is accomplished by taking `self` by value to prevent
obtaining multiple readers on a given `UserSlice`, and the readers only
permitting forward reads. If double-fetching a memory location is
necessary for some reason, then that is done by creating multiple
readers to the same memory location.
Constructing a `UserSlice` performs no checks on the provided address
and length, it can safely be constructed inside a kernel thread with no
current userspace process. Reads and writes wrap the kernel APIs
`copy_from_user` and `copy_to_user`, which check the memory map of the
current process and enforce that the address range is within the user
range (no additional calls to `access_ok` are needed).
This code is based on something that was originally written by Wedson on
the old rust branch. It was modified by Alice by removing the
`IoBufferReader` and `IoBufferWriter` traits, and various other changes.
Signed-off-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Trevor Gross <tmgross@umich.edu>
Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240528-alice-mm-v7-1-78222c31b8f4@google.com
[ Wrapped docs to 100 and added a few intra-doc links. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-05-28 14:58:02 +00:00
|
|
|
pub mod uaccess;
|
2023-08-28 10:48:02 +00:00
|
|
|
pub mod workqueue;
|
2022-02-11 19:25:34 +00:00
|
|
|
|
|
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub use bindings;
|
|
|
|
|
pub use macros;
|
2023-04-03 09:33:52 +00:00
|
|
|
pub use uapi;
|
2022-02-11 19:25:34 +00:00
|
|
|
|
2022-11-10 16:41:38 +00:00
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub use build_error::build_error;
|
|
|
|
|
|
2022-02-11 19:25:34 +00:00
|
|
|
/// Prefix to appear before log messages printed from within the `kernel` crate.
|
|
|
|
|
const __LOG_PREFIX: &[u8] = b"rust_kernel\0";
|
|
|
|
|
|
|
|
|
|
/// The top level entrypoint to implementing a kernel module.
|
|
|
|
|
///
|
|
|
|
|
/// For any teardown or cleanup operations, your type may implement [`Drop`].
|
2024-03-28 19:54:54 +00:00
|
|
|
pub trait Module: Sized + Sync + Send {
|
2022-02-11 19:25:34 +00:00
|
|
|
/// Called at module initialization time.
|
|
|
|
|
///
|
|
|
|
|
/// Use this method to perform whatever setup or registration your module
|
|
|
|
|
/// should do.
|
|
|
|
|
///
|
|
|
|
|
/// Equivalent to the `module_init` macro in the C API.
|
|
|
|
|
fn init(module: &'static ThisModule) -> error::Result<Self>;
|
|
|
|
|
}
|
|
|
|
|
|
2024-10-22 21:31:39 +00:00
|
|
|
/// A module that is pinned and initialised in-place.
|
|
|
|
|
pub trait InPlaceModule: Sync + Send {
|
|
|
|
|
/// Creates an initialiser for the module.
|
|
|
|
|
///
|
|
|
|
|
/// It is called when the module is loaded.
|
|
|
|
|
fn init(module: &'static ThisModule) -> impl init::PinInit<Self, error::Error>;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T: Module> InPlaceModule for T {
|
|
|
|
|
fn init(module: &'static ThisModule) -> impl init::PinInit<Self, error::Error> {
|
|
|
|
|
let initer = move |slot: *mut Self| {
|
|
|
|
|
let m = <Self as Module>::init(module)?;
|
|
|
|
|
|
|
|
|
|
// SAFETY: `slot` is valid for write per the contract with `pin_init_from_closure`.
|
|
|
|
|
unsafe { slot.write(m) };
|
|
|
|
|
Ok(())
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
// SAFETY: On success, `initer` always fully initialises an instance of `Self`.
|
|
|
|
|
unsafe { init::pin_init_from_closure(initer) }
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2022-02-11 19:25:34 +00:00
|
|
|
/// Equivalent to `THIS_MODULE` in the C API.
|
|
|
|
|
///
|
2024-10-21 02:58:47 +00:00
|
|
|
/// C header: [`include/linux/init.h`](srctree/include/linux/init.h)
|
2022-02-11 19:25:34 +00:00
|
|
|
pub struct ThisModule(*mut bindings::module);
|
|
|
|
|
|
|
|
|
|
// SAFETY: `THIS_MODULE` may be used from all threads within a module.
|
|
|
|
|
unsafe impl Sync for ThisModule {}
|
|
|
|
|
|
|
|
|
|
impl ThisModule {
|
|
|
|
|
/// Creates a [`ThisModule`] given the `THIS_MODULE` pointer.
|
|
|
|
|
///
|
|
|
|
|
/// # Safety
|
|
|
|
|
///
|
|
|
|
|
/// The pointer must be equal to the right `THIS_MODULE`.
|
|
|
|
|
pub const unsafe fn from_ptr(ptr: *mut bindings::module) -> ThisModule {
|
|
|
|
|
ThisModule(ptr)
|
|
|
|
|
}
|
2024-02-26 09:44:02 +00:00
|
|
|
|
|
|
|
|
/// Access the raw pointer for this module.
|
|
|
|
|
///
|
|
|
|
|
/// It is up to the user to use it correctly.
|
|
|
|
|
pub const fn as_ptr(&self) -> *mut bindings::module {
|
|
|
|
|
self.0
|
|
|
|
|
}
|
2022-02-11 19:25:34 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[cfg(not(any(testlib, test)))]
|
|
|
|
|
#[panic_handler]
|
|
|
|
|
fn panic(info: &core::panic::PanicInfo<'_>) -> ! {
|
|
|
|
|
pr_emerg!("{}\n", info);
|
|
|
|
|
// SAFETY: FFI call.
|
|
|
|
|
unsafe { bindings::BUG() };
|
|
|
|
|
}
|
2024-02-19 11:48:08 +00:00
|
|
|
|
|
|
|
|
/// Produces a pointer to an object from a pointer to one of its fields.
|
|
|
|
|
///
|
|
|
|
|
/// # Safety
|
|
|
|
|
///
|
|
|
|
|
/// The pointer passed to this macro, and the pointer returned by this macro, must both be in
|
|
|
|
|
/// bounds of the same allocation.
|
|
|
|
|
///
|
|
|
|
|
/// # Examples
|
|
|
|
|
///
|
|
|
|
|
/// ```
|
|
|
|
|
/// # use kernel::container_of;
|
|
|
|
|
/// struct Test {
|
|
|
|
|
/// a: u64,
|
|
|
|
|
/// b: u32,
|
|
|
|
|
/// }
|
|
|
|
|
///
|
|
|
|
|
/// let test = Test { a: 10, b: 20 };
|
|
|
|
|
/// let b_ptr = &test.b;
|
|
|
|
|
/// // SAFETY: The pointer points at the `b` field of a `Test`, so the resulting pointer will be
|
|
|
|
|
/// // in-bounds of the same allocation as `b_ptr`.
|
|
|
|
|
/// let test_alias = unsafe { container_of!(b_ptr, Test, b) };
|
|
|
|
|
/// assert!(core::ptr::eq(&test, test_alias));
|
|
|
|
|
/// ```
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! container_of {
|
|
|
|
|
($ptr:expr, $type:ty, $($f:tt)*) => {{
|
|
|
|
|
let ptr = $ptr as *const _ as *const u8;
|
|
|
|
|
let offset: usize = ::core::mem::offset_of!($type, $($f)*);
|
|
|
|
|
ptr.sub(offset) as *const $type
|
|
|
|
|
}}
|
|
|
|
|
}
|
2024-10-30 16:04:28 +00:00
|
|
|
|
|
|
|
|
/// Helper for `.rs.S` files.
|
|
|
|
|
#[doc(hidden)]
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! concat_literals {
|
|
|
|
|
($( $asm:literal )* ) => {
|
|
|
|
|
::core::concat!($($asm),*)
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Wrapper around `asm!` configured for use in the kernel.
|
|
|
|
|
///
|
|
|
|
|
/// Uses a semicolon to avoid parsing ambiguities, even though this does not match native `asm!`
|
|
|
|
|
/// syntax.
|
|
|
|
|
// For x86, `asm!` uses intel syntax by default, but we want to use at&t syntax in the kernel.
|
|
|
|
|
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! asm {
|
|
|
|
|
($($asm:expr),* ; $($rest:tt)*) => {
|
|
|
|
|
::core::arch::asm!( $($asm)*, options(att_syntax), $($rest)* )
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Wrapper around `asm!` configured for use in the kernel.
|
|
|
|
|
///
|
|
|
|
|
/// Uses a semicolon to avoid parsing ambiguities, even though this does not match native `asm!`
|
|
|
|
|
/// syntax.
|
|
|
|
|
// For non-x86 arches we just pass through to `asm!`.
|
|
|
|
|
#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! asm {
|
|
|
|
|
($($asm:expr),* ; $($rest:tt)*) => {
|
|
|
|
|
::core::arch::asm!( $($asm)*, $($rest)* )
|
|
|
|
|
};
|
|
|
|
|
}
|