bpf: Introduce bpf_obj_new
Introduce type safe memory allocator bpf_obj_new for BPF programs. The
kernel side kfunc is named bpf_obj_new_impl, as passing hidden arguments
to kfuncs still requires having them in prototype, unlike BPF helpers
which always take 5 arguments and have them checked using bpf_func_proto
in verifier, ignoring unset argument types.
Introduce __ign suffix to ignore a specific kfunc argument during type
checks, then use this to introduce support for passing type metadata to
the bpf_obj_new_impl kfunc.
The user passes BTF ID of the type it wants to allocates in program BTF,
the verifier then rewrites the first argument as the size of this type,
after performing some sanity checks (to ensure it exists and it is a
struct type).
The second argument is also fixed up and passed by the verifier. This is
the btf_struct_meta for the type being allocated. It would be needed
mostly for the offset array which is required for zero initializing
special fields while leaving the rest of storage in unitialized state.
It would also be needed in the next patch to perform proper destruction
of the object's special fields.
Under the hood, bpf_obj_new will call bpf_mem_alloc and bpf_mem_free,
using the any context BPF memory allocator introduced recently. To this
end, a global instance of the BPF memory allocator is initialized on
boot to be used for this purpose. This 'bpf_global_ma' serves all
allocations for bpf_obj_new. In the future, bpf_obj_new variants will
allow specifying a custom allocator.
Note that now that bpf_obj_new can be used to allocate objects that can
be linked to BPF linked list (when future linked list helpers are
available), we need to also free the elements using bpf_mem_free.
However, since the draining of elements is done outside the
bpf_spin_lock, we need to do migrate_disable around the call since
bpf_list_head_free can be called from map free path where migration is
enabled. Otherwise, when called from BPF programs migration is already
disabled.
A convenience macro is included in the bpf_experimental.h header to hide
over the ugly details of the implementation, leading to user code
looking similar to a language level extension which allocates and
constructs fields of a user type.
struct bar {
struct bpf_list_node node;
};
struct foo {
struct bpf_spin_lock lock;
struct bpf_list_head head __contains(bar, node);
};
void prog(void) {
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return;
...
}
A key piece of this story is still missing, i.e. the free function,
which will come in the next patch.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20221118015614.2013203-14-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2022-11-18 07:26:03 +05:30
|
|
|
#ifndef __BPF_EXPERIMENTAL__
|
|
|
|
|
#define __BPF_EXPERIMENTAL__
|
|
|
|
|
|
|
|
|
|
#include <vmlinux.h>
|
|
|
|
|
#include <bpf/bpf_tracing.h>
|
|
|
|
|
#include <bpf/bpf_helpers.h>
|
|
|
|
|
#include <bpf/bpf_core_read.h>
|
|
|
|
|
|
2022-11-18 07:26:09 +05:30
|
|
|
#define __contains(name, node) __attribute__((btf_decl_tag("contains:" #name ":" #node)))
|
|
|
|
|
|
bpf: Introduce bpf_obj_new
Introduce type safe memory allocator bpf_obj_new for BPF programs. The
kernel side kfunc is named bpf_obj_new_impl, as passing hidden arguments
to kfuncs still requires having them in prototype, unlike BPF helpers
which always take 5 arguments and have them checked using bpf_func_proto
in verifier, ignoring unset argument types.
Introduce __ign suffix to ignore a specific kfunc argument during type
checks, then use this to introduce support for passing type metadata to
the bpf_obj_new_impl kfunc.
The user passes BTF ID of the type it wants to allocates in program BTF,
the verifier then rewrites the first argument as the size of this type,
after performing some sanity checks (to ensure it exists and it is a
struct type).
The second argument is also fixed up and passed by the verifier. This is
the btf_struct_meta for the type being allocated. It would be needed
mostly for the offset array which is required for zero initializing
special fields while leaving the rest of storage in unitialized state.
It would also be needed in the next patch to perform proper destruction
of the object's special fields.
Under the hood, bpf_obj_new will call bpf_mem_alloc and bpf_mem_free,
using the any context BPF memory allocator introduced recently. To this
end, a global instance of the BPF memory allocator is initialized on
boot to be used for this purpose. This 'bpf_global_ma' serves all
allocations for bpf_obj_new. In the future, bpf_obj_new variants will
allow specifying a custom allocator.
Note that now that bpf_obj_new can be used to allocate objects that can
be linked to BPF linked list (when future linked list helpers are
available), we need to also free the elements using bpf_mem_free.
However, since the draining of elements is done outside the
bpf_spin_lock, we need to do migrate_disable around the call since
bpf_list_head_free can be called from map free path where migration is
enabled. Otherwise, when called from BPF programs migration is already
disabled.
A convenience macro is included in the bpf_experimental.h header to hide
over the ugly details of the implementation, leading to user code
looking similar to a language level extension which allocates and
constructs fields of a user type.
struct bar {
struct bpf_list_node node;
};
struct foo {
struct bpf_spin_lock lock;
struct bpf_list_head head __contains(bar, node);
};
void prog(void) {
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return;
...
}
A key piece of this story is still missing, i.e. the free function,
which will come in the next patch.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20221118015614.2013203-14-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2022-11-18 07:26:03 +05:30
|
|
|
/* Description
|
|
|
|
|
* Allocates an object of the type represented by 'local_type_id' in
|
|
|
|
|
* program BTF. User may use the bpf_core_type_id_local macro to pass the
|
|
|
|
|
* type ID of a struct in program BTF.
|
|
|
|
|
*
|
|
|
|
|
* The 'local_type_id' parameter must be a known constant.
|
|
|
|
|
* The 'meta' parameter is a hidden argument that is ignored.
|
|
|
|
|
* Returns
|
|
|
|
|
* A pointer to an object of the type corresponding to the passed in
|
|
|
|
|
* 'local_type_id', or NULL on failure.
|
|
|
|
|
*/
|
|
|
|
|
extern void *bpf_obj_new_impl(__u64 local_type_id, void *meta) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Convenience macro to wrap over bpf_obj_new_impl */
|
|
|
|
|
#define bpf_obj_new(type) ((type *)bpf_obj_new_impl(bpf_core_type_id_local(type), NULL))
|
|
|
|
|
|
2022-11-18 07:26:04 +05:30
|
|
|
/* Description
|
|
|
|
|
* Free an allocated object. All fields of the object that require
|
|
|
|
|
* destruction will be destructed before the storage is freed.
|
|
|
|
|
*
|
|
|
|
|
* The 'meta' parameter is a hidden argument that is ignored.
|
|
|
|
|
* Returns
|
|
|
|
|
* Void.
|
|
|
|
|
*/
|
|
|
|
|
extern void bpf_obj_drop_impl(void *kptr, void *meta) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Convenience macro to wrap over bpf_obj_drop_impl */
|
|
|
|
|
#define bpf_obj_drop(kptr) bpf_obj_drop_impl(kptr, NULL)
|
|
|
|
|
|
bpf: Introduce single ownership BPF linked list API
Add a linked list API for use in BPF programs, where it expects
protection from the bpf_spin_lock in the same allocation as the
bpf_list_head. For now, only one bpf_spin_lock can be present hence that
is assumed to be the one protecting the bpf_list_head.
The following functions are added to kick things off:
// Add node to beginning of list
void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node);
// Add node to end of list
void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node);
// Remove node at beginning of list and return it
struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head);
// Remove node at end of list and return it
struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head);
The lock protecting the bpf_list_head needs to be taken for all
operations. The verifier ensures that the lock that needs to be taken is
always held, and only the correct lock is taken for these operations.
These checks are made statically by relying on the reg->id preserved for
registers pointing into regions having both bpf_spin_lock and the
objects protected by it. The comment over check_reg_allocation_locked in
this change describes the logic in detail.
Note that bpf_list_push_front and bpf_list_push_back are meant to
consume the object containing the node in the 1st argument, however that
specific mechanism is intended to not release the ref_obj_id directly
until the bpf_spin_unlock is called. In this commit, nothing is done,
but the next commit will be introducing logic to handle this case, so it
has been left as is for now.
bpf_list_pop_front and bpf_list_pop_back delete the first or last item
of the list respectively, and return pointer to the element at the
list_node offset. The user can then use container_of style macro to get
the actual entry type. The verifier however statically knows the actual
type, so the safety properties are still preserved.
With these additions, programs can now manage their own linked lists and
store their objects in them.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20221118015614.2013203-17-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2022-11-18 07:26:06 +05:30
|
|
|
/* Description
|
|
|
|
|
* Add a new entry to the beginning of the BPF linked list.
|
|
|
|
|
* Returns
|
|
|
|
|
* Void.
|
|
|
|
|
*/
|
|
|
|
|
extern void bpf_list_push_front(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Description
|
|
|
|
|
* Add a new entry to the end of the BPF linked list.
|
|
|
|
|
* Returns
|
|
|
|
|
* Void.
|
|
|
|
|
*/
|
|
|
|
|
extern void bpf_list_push_back(struct bpf_list_head *head, struct bpf_list_node *node) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Description
|
|
|
|
|
* Remove the entry at the beginning of the BPF linked list.
|
|
|
|
|
* Returns
|
|
|
|
|
* Pointer to bpf_list_node of deleted entry, or NULL if list is empty.
|
|
|
|
|
*/
|
|
|
|
|
extern struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Description
|
|
|
|
|
* Remove the entry at the end of the BPF linked list.
|
|
|
|
|
* Returns
|
|
|
|
|
* Pointer to bpf_list_node of deleted entry, or NULL if list is empty.
|
|
|
|
|
*/
|
|
|
|
|
extern struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) __ksym;
|
|
|
|
|
|
2023-02-13 16:40:15 -08:00
|
|
|
/* Description
|
|
|
|
|
* Remove 'node' from rbtree with root 'root'
|
|
|
|
|
* Returns
|
|
|
|
|
* Pointer to the removed node, or NULL if 'root' didn't contain 'node'
|
|
|
|
|
*/
|
|
|
|
|
extern struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root,
|
|
|
|
|
struct bpf_rb_node *node) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Description
|
|
|
|
|
* Add 'node' to rbtree with root 'root' using comparator 'less'
|
|
|
|
|
* Returns
|
|
|
|
|
* Nothing
|
|
|
|
|
*/
|
|
|
|
|
extern void bpf_rbtree_add(struct bpf_rb_root *root, struct bpf_rb_node *node,
|
|
|
|
|
bool (less)(struct bpf_rb_node *a, const struct bpf_rb_node *b)) __ksym;
|
|
|
|
|
|
|
|
|
|
/* Description
|
|
|
|
|
* Return the first (leftmost) node in input tree
|
|
|
|
|
* Returns
|
|
|
|
|
* Pointer to the node, which is _not_ removed from the tree. If the tree
|
|
|
|
|
* contains no nodes, returns NULL.
|
|
|
|
|
*/
|
|
|
|
|
extern struct bpf_rb_node *bpf_rbtree_first(struct bpf_rb_root *root) __ksym;
|
|
|
|
|
|
bpf: Introduce bpf_obj_new
Introduce type safe memory allocator bpf_obj_new for BPF programs. The
kernel side kfunc is named bpf_obj_new_impl, as passing hidden arguments
to kfuncs still requires having them in prototype, unlike BPF helpers
which always take 5 arguments and have them checked using bpf_func_proto
in verifier, ignoring unset argument types.
Introduce __ign suffix to ignore a specific kfunc argument during type
checks, then use this to introduce support for passing type metadata to
the bpf_obj_new_impl kfunc.
The user passes BTF ID of the type it wants to allocates in program BTF,
the verifier then rewrites the first argument as the size of this type,
after performing some sanity checks (to ensure it exists and it is a
struct type).
The second argument is also fixed up and passed by the verifier. This is
the btf_struct_meta for the type being allocated. It would be needed
mostly for the offset array which is required for zero initializing
special fields while leaving the rest of storage in unitialized state.
It would also be needed in the next patch to perform proper destruction
of the object's special fields.
Under the hood, bpf_obj_new will call bpf_mem_alloc and bpf_mem_free,
using the any context BPF memory allocator introduced recently. To this
end, a global instance of the BPF memory allocator is initialized on
boot to be used for this purpose. This 'bpf_global_ma' serves all
allocations for bpf_obj_new. In the future, bpf_obj_new variants will
allow specifying a custom allocator.
Note that now that bpf_obj_new can be used to allocate objects that can
be linked to BPF linked list (when future linked list helpers are
available), we need to also free the elements using bpf_mem_free.
However, since the draining of elements is done outside the
bpf_spin_lock, we need to do migrate_disable around the call since
bpf_list_head_free can be called from map free path where migration is
enabled. Otherwise, when called from BPF programs migration is already
disabled.
A convenience macro is included in the bpf_experimental.h header to hide
over the ugly details of the implementation, leading to user code
looking similar to a language level extension which allocates and
constructs fields of a user type.
struct bar {
struct bpf_list_node node;
};
struct foo {
struct bpf_spin_lock lock;
struct bpf_list_head head __contains(bar, node);
};
void prog(void) {
struct foo *f;
f = bpf_obj_new(typeof(*f));
if (!f)
return;
...
}
A key piece of this story is still missing, i.e. the free function,
which will come in the next patch.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://lore.kernel.org/r/20221118015614.2013203-14-memxor@gmail.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2022-11-18 07:26:03 +05:30
|
|
|
#endif
|