Linaro-mm-sig

linaro-mm-sig@lists.linaro.org

11 participants
6375 discussions

[PATCH 6/8] dma-buf: use inline lock for the stub fence v2

by Christian König

Using the inline lock is now the recommended way for dma_fence implementations. So use this approach for the framework's internal fences as well. Also saves about 4 bytes for the external spinlock. v2: drop unnecessary changes Signed-off-by: Christian König <christian.koenig(a)amd.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin(a)igalia.com> Reviewed-by: Philipp Stanner <phasta(a)kernel.org> Reviewed-by: Boris Brezillon <boris.brezillon(a)collabora.com> --- drivers/dma-buf/dma-fence.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index 39f0a4d08a2d..133cb9033b2c 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -24,7 +24,6 @@ EXPORT_TRACEPOINT_SYMBOL(dma_fence_emit); EXPORT_TRACEPOINT_SYMBOL(dma_fence_enable_signal); EXPORT_TRACEPOINT_SYMBOL(dma_fence_signaled); -static DEFINE_SPINLOCK(dma_fence_stub_lock); static struct dma_fence dma_fence_stub; /* @@ -123,12 +122,9 @@ static const struct dma_fence_ops dma_fence_stub_ops = { static int __init dma_fence_init_stub(void) { - dma_fence_init(&dma_fence_stub, &dma_fence_stub_ops, - &dma_fence_stub_lock, 0, 0); - + dma_fence_init(&dma_fence_stub, &dma_fence_stub_ops, NULL, 0, 0); set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &dma_fence_stub.flags); - dma_fence_signal(&dma_fence_stub); return 0; } @@ -160,11 +156,7 @@ struct dma_fence *dma_fence_allocate_private_stub(ktime_t timestamp) if (fence == NULL) return NULL; - dma_fence_init(fence, - &dma_fence_stub_ops, - &dma_fence_stub_lock, - 0, 0); - + dma_fence_init(fence, &dma_fence_stub_ops, NULL, 0, 0); set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &fence->flags); -- 2.43.0

2 months, 2 weeks

[PATCH 5/8] dma-buf/selftests: test RCU ops and inline lock v2

by Christian König

Drop the mock_fence and the kmem_cache, instead use the inline lock and test if the ops are properly dropped after signaling. v2: move the RCU check to the end of the test Signed-off-by: Christian König <christian.koenig(a)amd.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin(a)igalia.com> --- drivers/dma-buf/st-dma-fence.c | 44 ++++++++-------------------------- 1 file changed, 10 insertions(+), 34 deletions(-) diff --git a/drivers/dma-buf/st-dma-fence.c b/drivers/dma-buf/st-dma-fence.c index 5d0d9abc6e21..0d9d524d79b6 100644 --- a/drivers/dma-buf/st-dma-fence.c +++ b/drivers/dma-buf/st-dma-fence.c @@ -14,43 +14,26 @@ #include "selftest.h" -static struct kmem_cache *slab_fences; - -static struct mock_fence { - struct dma_fence base; - struct spinlock lock; -} *to_mock_fence(struct dma_fence *f) { - return container_of(f, struct mock_fence, base); -} - static const char *mock_name(struct dma_fence *f) { return "mock"; } -static void mock_fence_release(struct dma_fence *f) -{ - kmem_cache_free(slab_fences, to_mock_fence(f)); -} - static const struct dma_fence_ops mock_ops = { .get_driver_name = mock_name, .get_timeline_name = mock_name, - .release = mock_fence_release, }; static struct dma_fence *mock_fence(void) { - struct mock_fence *f; + struct dma_fence *f; - f = kmem_cache_alloc(slab_fences, GFP_KERNEL); + f = kmalloc(sizeof(*f), GFP_KERNEL); if (!f) return NULL; - spin_lock_init(&f->lock); - dma_fence_init(&f->base, &mock_ops, &f->lock, 0, 0); - - return &f->base; + dma_fence_init(f, &mock_ops, NULL, 0, 0); + return f; } static int sanitycheck(void *arg) @@ -100,6 +83,11 @@ static int test_signaling(void *arg) goto err_free; } + if (rcu_dereference_protected(f->ops, true)) { + pr_err("Fence ops not cleared on signal\n"); + goto err_free; + } + err = 0; err_free: dma_fence_put(f); @@ -540,19 +528,7 @@ int dma_fence(void) SUBTEST(test_stub), SUBTEST(race_signal_callback), }; - int ret; pr_info("sizeof(dma_fence)=%zu\n", sizeof(struct dma_fence)); - - slab_fences = KMEM_CACHE(mock_fence, - SLAB_TYPESAFE_BY_RCU | - SLAB_HWCACHE_ALIGN); - if (!slab_fences) - return -ENOMEM; - - ret = subtests(tests, NULL); - - kmem_cache_destroy(slab_fences); - - return ret; + return subtests(tests, NULL); } -- 2.43.0

2 months, 2 weeks

[PATCH 4/8] dma-buf: inline spinlock for fence protection v5

by Christian König

Implement per-fence spinlocks, allowing implementations to not give an external spinlock to protect the fence internal state. Instead a spinlock embedded into the fence structure itself is used in this case. Shared spinlocks have the problem that implementations need to guarantee that the lock lives at least as long all fences referencing them. Using a per-fence spinlock allows completely decoupling spinlock producer and consumer life times, simplifying the handling in most use cases. v2: improve naming, coverage and function documentation v3: fix one additional locking in the selftests v4: separate out some changes to make the patch smaller, fix one amdgpu crash found by CI systems v5: improve comments Signed-off-by: Christian König <christian.koenig(a)amd.com> Reviewed-by: Boris Brezillon <boris.brezillon(a)collabora.com> --- drivers/dma-buf/dma-fence.c | 21 ++++++++++++++++----- drivers/dma-buf/sync_debug.h | 2 +- drivers/gpu/drm/drm_crtc.c | 2 +- drivers/gpu/drm/drm_writeback.c | 2 +- drivers/gpu/drm/nouveau/nouveau_fence.c | 3 ++- drivers/gpu/drm/qxl/qxl_release.c | 3 ++- drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 3 ++- drivers/gpu/drm/xe/xe_hw_fence.c | 3 ++- include/linux/dma-fence.h | 19 +++++++++++++------ 9 files changed, 40 insertions(+), 18 deletions(-) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index bd4ec7e26dae..39f0a4d08a2d 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -343,7 +343,6 @@ void __dma_fence_might_wait(void) } #endif - /** * dma_fence_signal_timestamp_locked - signal completion of a fence * @fence: the fence to signal @@ -1068,7 +1067,6 @@ static void __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, spinlock_t *lock, u64 context, u64 seqno, unsigned long flags) { - BUG_ON(!lock); BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name); kref_init(&fence->refcount); @@ -1080,10 +1078,15 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, */ RCU_INIT_POINTER(fence->ops, ops); INIT_LIST_HEAD(&fence->cb_list); - fence->lock = lock; fence->context = context; fence->seqno = seqno; fence->flags = flags | BIT(DMA_FENCE_FLAG_INITIALIZED_BIT); + if (lock) { + fence->extern_lock = lock; + } else { + spin_lock_init(&fence->inline_lock); + fence->flags |= BIT(DMA_FENCE_FLAG_INLINE_LOCK_BIT); + } fence->error = 0; trace_dma_fence_init(fence); @@ -1093,7 +1096,7 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, * dma_fence_init - Initialize a custom fence. * @fence: the fence to initialize * @ops: the dma_fence_ops for operations on this fence - * @lock: the irqsafe spinlock to use for locking this fence + * @lock: optional irqsafe spinlock to use for locking this fence * @context: the execution context this fence is run on * @seqno: a linear increasing sequence number for this context * @@ -1103,6 +1106,10 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, * * context and seqno are used for easy comparison between fences, allowing * to check which fence is later by simply using dma_fence_later(). + * + * It is strongly discouraged to provide an external lock because this couples + * lock and fence life time. This is only allowed for legacy use cases when + * multiple fences need to be prevented from signaling out of order. */ void dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, @@ -1116,7 +1123,7 @@ EXPORT_SYMBOL(dma_fence_init); * dma_fence_init64 - Initialize a custom fence with 64-bit seqno support. * @fence: the fence to initialize * @ops: the dma_fence_ops for operations on this fence - * @lock: the irqsafe spinlock to use for locking this fence + * @lock: optional irqsafe spinlock to use for locking this fence * @context: the execution context this fence is run on * @seqno: a linear increasing sequence number for this context * @@ -1126,6 +1133,10 @@ EXPORT_SYMBOL(dma_fence_init); * * Context and seqno are used for easy comparison between fences, allowing * to check which fence is later by simply using dma_fence_later(). + * + * It is strongly discouraged to provide an external lock because this couples + * lock and fence life time. This is only allowed for legacy use cases when + * multiple fences need to be prevented from signaling out of order. */ void dma_fence_init64(struct dma_fence *fence, const struct dma_fence_ops *ops, diff --git a/drivers/dma-buf/sync_debug.h b/drivers/dma-buf/sync_debug.h index 02af347293d0..c49324505b20 100644 --- a/drivers/dma-buf/sync_debug.h +++ b/drivers/dma-buf/sync_debug.h @@ -47,7 +47,7 @@ struct sync_timeline { static inline struct sync_timeline *dma_fence_parent(struct dma_fence *fence) { - return container_of(fence->lock, struct sync_timeline, lock); + return container_of(fence->extern_lock, struct sync_timeline, lock); } /** diff --git a/drivers/gpu/drm/drm_crtc.c b/drivers/gpu/drm/drm_crtc.c index a7797d260f1e..17472915842f 100644 --- a/drivers/gpu/drm/drm_crtc.c +++ b/drivers/gpu/drm/drm_crtc.c @@ -159,7 +159,7 @@ static const struct dma_fence_ops drm_crtc_fence_ops; static struct drm_crtc *fence_to_crtc(struct dma_fence *fence) { BUG_ON(fence->ops != &drm_crtc_fence_ops); - return container_of(fence->lock, struct drm_crtc, fence_lock); + return container_of(fence->extern_lock, struct drm_crtc, fence_lock); } static const char *drm_crtc_fence_get_driver_name(struct dma_fence *fence) diff --git a/drivers/gpu/drm/drm_writeback.c b/drivers/gpu/drm/drm_writeback.c index 95b8a2e4bda6..624a4e8b6c99 100644 --- a/drivers/gpu/drm/drm_writeback.c +++ b/drivers/gpu/drm/drm_writeback.c @@ -81,7 +81,7 @@ * From userspace, this property will always read as zero. */ -#define fence_to_wb_connector(x) container_of(x->lock, \ +#define fence_to_wb_connector(x) container_of(x->extern_lock, \ struct drm_writeback_connector, \ fence_lock) diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c index 4a193b7d6d9e..c282c94138b2 100644 --- a/drivers/gpu/drm/nouveau/nouveau_fence.c +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c @@ -41,7 +41,8 @@ static const struct dma_fence_ops nouveau_fence_ops_legacy; static inline struct nouveau_fence_chan * nouveau_fctx(struct nouveau_fence *fence) { - return container_of(fence->base.lock, struct nouveau_fence_chan, lock); + return container_of(fence->base.extern_lock, struct nouveau_fence_chan, + lock); } static bool diff --git a/drivers/gpu/drm/qxl/qxl_release.c b/drivers/gpu/drm/qxl/qxl_release.c index 06b0b2aa7953..37d4ae0faf0d 100644 --- a/drivers/gpu/drm/qxl/qxl_release.c +++ b/drivers/gpu/drm/qxl/qxl_release.c @@ -62,7 +62,8 @@ static long qxl_fence_wait(struct dma_fence *fence, bool intr, struct qxl_device *qdev; unsigned long cur, end = jiffies + timeout; - qdev = container_of(fence->lock, struct qxl_device, release_lock); + qdev = container_of(fence->extern_lock, struct qxl_device, + release_lock); if (!wait_event_timeout(qdev->release_event, (dma_fence_is_signaled(fence) || diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c b/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c index 85795082fef9..d251eec57df9 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c @@ -47,7 +47,8 @@ struct vmw_event_fence_action { static struct vmw_fence_manager * fman_from_fence(struct vmw_fence_obj *fence) { - return container_of(fence->base.lock, struct vmw_fence_manager, lock); + return container_of(fence->base.extern_lock, struct vmw_fence_manager, + lock); } static void vmw_fence_obj_destroy(struct dma_fence *f) diff --git a/drivers/gpu/drm/xe/xe_hw_fence.c b/drivers/gpu/drm/xe/xe_hw_fence.c index ae8ed15b64c5..14720623ad00 100644 --- a/drivers/gpu/drm/xe/xe_hw_fence.c +++ b/drivers/gpu/drm/xe/xe_hw_fence.c @@ -124,7 +124,8 @@ static struct xe_hw_fence *to_xe_hw_fence(struct dma_fence *fence); static struct xe_hw_fence_irq *xe_hw_fence_irq(struct xe_hw_fence *fence) { - return container_of(fence->dma.lock, struct xe_hw_fence_irq, lock); + return container_of(fence->dma.extern_lock, struct xe_hw_fence_irq, + lock); } static const char *xe_hw_fence_get_driver_name(struct dma_fence *dma_fence) diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h index 086324af96c9..9ed359e38d4e 100644 --- a/include/linux/dma-fence.h +++ b/include/linux/dma-fence.h @@ -34,7 +34,8 @@ struct seq_file; * @ops: dma_fence_ops associated with this fence * @rcu: used for releasing fence with kfree_rcu * @cb_list: list of all callbacks to call - * @lock: spin_lock_irqsave used for locking + * @extern_lock: external spin_lock_irqsave used for locking (deprecated) + * @inline_lock: alternative internal spin_lock_irqsave used for locking * @context: execution context this fence belongs to, returned by * dma_fence_context_alloc() * @seqno: the sequence number of this fence inside the execution context, @@ -49,6 +50,7 @@ struct seq_file; * of the time. * * DMA_FENCE_FLAG_INITIALIZED_BIT - fence was initialized + * DMA_FENCE_FLAG_INLINE_LOCK_BIT - use inline spinlock instead of external one * DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled * DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called @@ -66,7 +68,10 @@ struct seq_file; * been completed, or never called at all. */ struct dma_fence { - spinlock_t *lock; + union { + spinlock_t *extern_lock; + spinlock_t inline_lock; + }; const struct dma_fence_ops __rcu *ops; /* * We clear the callback list on kref_put so that by the time we @@ -100,6 +105,7 @@ struct dma_fence { enum dma_fence_flag_bits { DMA_FENCE_FLAG_INITIALIZED_BIT, + DMA_FENCE_FLAG_INLINE_LOCK_BIT, DMA_FENCE_FLAG_SEQNO64_BIT, DMA_FENCE_FLAG_SIGNALED_BIT, DMA_FENCE_FLAG_TIMESTAMP_BIT, @@ -381,11 +387,12 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) * dma_fence_spinlock - return pointer to the spinlock protecting the fence * @fence: the fence to get the lock from * - * Return the pointer to the extern lock. + * Return either the pointer to the embedded or the external spin lock. */ static inline spinlock_t *dma_fence_spinlock(struct dma_fence *fence) { - return fence->lock; + return test_bit(DMA_FENCE_FLAG_INLINE_LOCK_BIT, &fence->flags) ? + &fence->inline_lock : fence->extern_lock; } /** @@ -396,7 +403,7 @@ static inline spinlock_t *dma_fence_spinlock(struct dma_fence *fence) * Lock the fence, preventing it from changing to the signaled state. */ #define dma_fence_lock_irqsave(fence, flags) \ - spin_lock_irqsave(fence->lock, flags) + spin_lock_irqsave(dma_fence_spinlock(fence), flags) /** * dma_fence_unlock_irqrestore - unlock the fence and irqrestore @@ -406,7 +413,7 @@ static inline spinlock_t *dma_fence_spinlock(struct dma_fence *fence) * Unlock the fence, allowing it to change it's state to signaled again. */ #define dma_fence_unlock_irqrestore(fence, flags) \ - spin_unlock_irqrestore(fence->lock, flags) + spin_unlock_irqrestore(dma_fence_spinlock(fence), flags) /** * dma_fence_assert_held - lockdep assertion that fence is locked -- 2.43.0

2 months, 2 weeks

[PATCH 3/8] dma-buf: abstract fence locking v2

by Christian König

Add dma_fence_lock_irqsafe() and dma_fence_unlock_irqrestore() wrappers and mechanically apply them everywhere. Just a pre-requisite cleanup for a follow up patch. v2: add some missing i915 bits, add abstraction for lockdep assertion as well v3: one more suggestion by Tvrtko Signed-off-by: Christian König <christian.koenig(a)amd.com> Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin(a)igalia.com> --- drivers/dma-buf/dma-fence.c | 48 ++++++++++----------- drivers/dma-buf/st-dma-fence.c | 6 ++- drivers/dma-buf/sw_sync.c | 14 +++--- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/i915/gt/intel_breadcrumbs.c | 2 +- drivers/gpu/drm/i915/i915_active.c | 19 ++++---- drivers/gpu/drm/nouveau/nouveau_drm.c | 5 ++- drivers/gpu/drm/scheduler/sched_fence.c | 6 +-- drivers/gpu/drm/xe/xe_sched_job.c | 4 +- include/linux/dma-fence.h | 38 ++++++++++++++++ 12 files changed, 96 insertions(+), 56 deletions(-) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index ae77f900c267..bd4ec7e26dae 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -365,7 +365,7 @@ void dma_fence_signal_timestamp_locked(struct dma_fence *fence, struct dma_fence_cb *cur, *tmp; struct list_head cb_list; - lockdep_assert_held(fence->lock); + dma_fence_assert_held(fence); if (unlikely(test_and_set_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))) @@ -412,9 +412,9 @@ void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp) if (WARN_ON(!fence)) return; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); dma_fence_signal_timestamp_locked(fence, timestamp); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); } EXPORT_SYMBOL(dma_fence_signal_timestamp); @@ -473,9 +473,9 @@ bool dma_fence_check_and_signal(struct dma_fence *fence) unsigned long flags; bool ret; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); ret = dma_fence_check_and_signal_locked(fence); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); return ret; } @@ -501,9 +501,9 @@ void dma_fence_signal(struct dma_fence *fence) tmp = dma_fence_begin_signalling(); - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); dma_fence_signal_timestamp_locked(fence, ktime_get()); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); dma_fence_end_signalling(tmp); } @@ -604,10 +604,10 @@ void dma_fence_release(struct kref *kref) * don't leave chains dangling. We set the error flag first * so that the callbacks know this signal is due to an error. */ - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); fence->error = -EDEADLK; dma_fence_signal_locked(fence); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); } ops = rcu_dereference(fence->ops); @@ -637,7 +637,7 @@ static bool __dma_fence_enable_signaling(struct dma_fence *fence) const struct dma_fence_ops *ops; bool was_set; - lockdep_assert_held(fence->lock); + dma_fence_assert_held(fence); was_set = test_and_set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, &fence->flags); @@ -673,9 +673,9 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence) { unsigned long flags; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); __dma_fence_enable_signaling(fence); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); } EXPORT_SYMBOL(dma_fence_enable_sw_signaling); @@ -715,8 +715,7 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, return -ENOENT; } - spin_lock_irqsave(fence->lock, flags); - + dma_fence_lock_irqsave(fence, flags); if (__dma_fence_enable_signaling(fence)) { cb->func = func; list_add_tail(&cb->node, &fence->cb_list); @@ -724,8 +723,7 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, INIT_LIST_HEAD(&cb->node); ret = -ENOENT; } - - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); return ret; } @@ -748,9 +746,9 @@ int dma_fence_get_status(struct dma_fence *fence) unsigned long flags; int status; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); status = dma_fence_get_status_locked(fence); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); return status; } @@ -780,13 +778,11 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb) unsigned long flags; bool ret; - spin_lock_irqsave(fence->lock, flags); - + dma_fence_lock_irqsave(fence, flags); ret = !list_empty(&cb->node); if (ret) list_del_init(&cb->node); - - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); return ret; } @@ -825,7 +821,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) unsigned long flags; signed long ret = timeout ? timeout : 1; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (dma_fence_test_signaled_flag(fence)) goto out; @@ -849,11 +845,11 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) __set_current_state(TASK_INTERRUPTIBLE); else __set_current_state(TASK_UNINTERRUPTIBLE); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); ret = schedule_timeout(ret); - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (ret > 0 && intr && signal_pending(current)) ret = -ERESTARTSYS; } @@ -863,7 +859,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout) __set_current_state(TASK_RUNNING); out: - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); return ret; } EXPORT_SYMBOL(dma_fence_default_wait); diff --git a/drivers/dma-buf/st-dma-fence.c b/drivers/dma-buf/st-dma-fence.c index 73ed6fd48a13..5d0d9abc6e21 100644 --- a/drivers/dma-buf/st-dma-fence.c +++ b/drivers/dma-buf/st-dma-fence.c @@ -410,8 +410,10 @@ struct race_thread { static void __wait_for_callbacks(struct dma_fence *f) { - spin_lock_irq(f->lock); - spin_unlock_irq(f->lock); + unsigned long flags; + + dma_fence_lock_irqsave(f, flags); + dma_fence_unlock_irqrestore(f, flags); } static int thread_signal_callback(void *arg) diff --git a/drivers/dma-buf/sw_sync.c b/drivers/dma-buf/sw_sync.c index 6f09d13be6b6..4c81a37dd682 100644 --- a/drivers/dma-buf/sw_sync.c +++ b/drivers/dma-buf/sw_sync.c @@ -156,12 +156,12 @@ static void timeline_fence_release(struct dma_fence *fence) struct sync_timeline *parent = dma_fence_parent(fence); unsigned long flags; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (!list_empty(&pt->link)) { list_del(&pt->link); rb_erase(&pt->node, &parent->pt_tree); } - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); sync_timeline_put(parent); dma_fence_free(fence); @@ -179,7 +179,7 @@ static void timeline_fence_set_deadline(struct dma_fence *fence, ktime_t deadlin struct sync_pt *pt = dma_fence_to_sync_pt(fence); unsigned long flags; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (test_bit(SW_SYNC_HAS_DEADLINE_BIT, &fence->flags)) { if (ktime_before(deadline, pt->deadline)) pt->deadline = deadline; @@ -187,7 +187,7 @@ static void timeline_fence_set_deadline(struct dma_fence *fence, ktime_t deadlin pt->deadline = deadline; __set_bit(SW_SYNC_HAS_DEADLINE_BIT, &fence->flags); } - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); } static const struct dma_fence_ops timeline_fence_ops = { @@ -431,13 +431,13 @@ static int sw_sync_ioctl_get_deadline(struct sync_timeline *obj, unsigned long a goto put_fence; } - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (!test_bit(SW_SYNC_HAS_DEADLINE_BIT, &fence->flags)) { ret = -ENOENT; goto unlock; } data.deadline_ns = ktime_to_ns(pt->deadline); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); dma_fence_put(fence); @@ -450,7 +450,7 @@ static int sw_sync_ioctl_get_deadline(struct sync_timeline *obj, unsigned long a return 0; unlock: - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); put_fence: dma_fence_put(fence); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c index b82357c65723..1404e1fe62a4 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c @@ -479,10 +479,10 @@ bool amdgpu_ring_soft_recovery(struct amdgpu_ring *ring, unsigned int vmid, if (amdgpu_sriov_vf(ring->adev) || !ring->funcs->soft_recovery || !fence) return false; - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (!dma_fence_is_signaled_locked(fence)) dma_fence_set_error(fence, -ENODATA); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); while (!dma_fence_is_signaled(fence) && ktime_to_ns(ktime_sub(deadline, ktime_get())) > 0) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 6a2ea200d90c..4761e7486811 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -2802,8 +2802,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm) dma_fence_put(vm->last_unlocked); dma_fence_wait(vm->last_tlb_flush, false); /* Make sure that all fence callbacks have completed */ - spin_lock_irqsave(vm->last_tlb_flush->lock, flags); - spin_unlock_irqrestore(vm->last_tlb_flush->lock, flags); + dma_fence_lock_irqsave(vm->last_tlb_flush, flags); + dma_fence_unlock_irqrestore(vm->last_tlb_flush, flags); dma_fence_put(vm->last_tlb_flush); list_for_each_entry_safe(mapping, tmp, &vm->freed, list) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 139642eacdd0..d5c41e24fb51 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -638,7 +638,7 @@ static inline uint64_t amdgpu_vm_tlb_seq(struct amdgpu_vm *vm) * sure that the dma_fence structure isn't freed up. */ rcu_read_lock(); - lock = vm->last_tlb_flush->lock; + lock = dma_fence_spinlock(vm->last_tlb_flush); rcu_read_unlock(); spin_lock_irqsave(lock, flags); diff --git a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c index bf6117d5fc57..78ea2d9ccedf 100644 --- a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c +++ b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c @@ -148,7 +148,7 @@ __dma_fence_signal__notify(struct dma_fence *fence, { struct dma_fence_cb *cur, *tmp; - lockdep_assert_held(fence->lock); + dma_fence_assert_held(fence); list_for_each_entry_safe(cur, tmp, list, node) { INIT_LIST_HEAD(&cur->node); diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c index 6b0c1162505a..9d41e052ab65 100644 --- a/drivers/gpu/drm/i915/i915_active.c +++ b/drivers/gpu/drm/i915/i915_active.c @@ -1045,9 +1045,10 @@ __i915_active_fence_set(struct i915_active_fence *active, * nesting rules for the fence->lock; the inner lock is always the * older lock. */ - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (prev) - spin_lock_nested(prev->lock, SINGLE_DEPTH_NESTING); + spin_lock_nested(dma_fence_spinlock(prev), + SINGLE_DEPTH_NESTING); /* * A does the cmpxchg first, and so it sees C or NULL, as before, or @@ -1061,17 +1062,18 @@ __i915_active_fence_set(struct i915_active_fence *active, */ while (cmpxchg(__active_fence_slot(active), prev, fence) != prev) { if (prev) { - spin_unlock(prev->lock); + spin_unlock(dma_fence_spinlock(prev)); dma_fence_put(prev); } - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); prev = i915_active_fence_get(active); GEM_BUG_ON(prev == fence); - spin_lock_irqsave(fence->lock, flags); + dma_fence_lock_irqsave(fence, flags); if (prev) - spin_lock_nested(prev->lock, SINGLE_DEPTH_NESTING); + spin_lock_nested(dma_fence_spinlock(prev), + SINGLE_DEPTH_NESTING); } /* @@ -1088,10 +1090,11 @@ __i915_active_fence_set(struct i915_active_fence *active, */ if (prev) { __list_del_entry(&active->cb.node); - spin_unlock(prev->lock); /* serialise with prev->cb_list */ + /* serialise with prev->cb_list */ + spin_unlock(dma_fence_spinlock(prev)); } list_add_tail(&active->cb.node, &fence->cb_list); - spin_unlock_irqrestore(fence->lock, flags); + dma_fence_unlock_irqrestore(fence, flags); return prev; } diff --git a/drivers/gpu/drm/nouveau/nouveau_drm.c b/drivers/gpu/drm/nouveau/nouveau_drm.c index f2e04a048ac2..e6805b8b9ffa 100644 --- a/drivers/gpu/drm/nouveau/nouveau_drm.c +++ b/drivers/gpu/drm/nouveau/nouveau_drm.c @@ -156,12 +156,13 @@ nouveau_name(struct drm_device *dev) static inline bool nouveau_cli_work_ready(struct dma_fence *fence) { + unsigned long flags; bool ret = true; - spin_lock_irq(fence->lock); + dma_fence_lock_irqsave(fence, flags); if (!dma_fence_is_signaled_locked(fence)) ret = false; - spin_unlock_irq(fence->lock); + dma_fence_unlock_irqrestore(fence, flags); if (ret == true) dma_fence_put(fence); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index 9391d6f0dc01..724d77694246 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -156,19 +156,19 @@ static void drm_sched_fence_set_deadline_finished(struct dma_fence *f, struct dma_fence *parent; unsigned long flags; - spin_lock_irqsave(&fence->lock, flags); + dma_fence_lock_irqsave(f, flags); /* If we already have an earlier deadline, keep it: */ if (test_bit(DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags) && ktime_before(fence->deadline, deadline)) { - spin_unlock_irqrestore(&fence->lock, flags); + dma_fence_unlock_irqrestore(f, flags); return; } fence->deadline = deadline; set_bit(DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT, &f->flags); - spin_unlock_irqrestore(&fence->lock, flags); + dma_fence_unlock_irqrestore(f, flags); /* * smp_load_aquire() to ensure that if we are racing another diff --git a/drivers/gpu/drm/xe/xe_sched_job.c b/drivers/gpu/drm/xe/xe_sched_job.c index 3927666fe556..ae5b38b2a884 100644 --- a/drivers/gpu/drm/xe/xe_sched_job.c +++ b/drivers/gpu/drm/xe/xe_sched_job.c @@ -190,11 +190,11 @@ static bool xe_fence_set_error(struct dma_fence *fence, int error) unsigned long irq_flags; bool signaled; - spin_lock_irqsave(fence->lock, irq_flags); + dma_fence_lock_irqsave(fence, irq_flags); signaled = test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags); if (!signaled) dma_fence_set_error(fence, error); - spin_unlock_irqrestore(fence->lock, irq_flags); + dma_fence_unlock_irqrestore(fence, irq_flags); return signaled; } diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h index 80db7ede91de..086324af96c9 100644 --- a/include/linux/dma-fence.h +++ b/include/linux/dma-fence.h @@ -377,6 +377,44 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) } while (1); } +/** + * dma_fence_spinlock - return pointer to the spinlock protecting the fence + * @fence: the fence to get the lock from + * + * Return the pointer to the extern lock. + */ +static inline spinlock_t *dma_fence_spinlock(struct dma_fence *fence) +{ + return fence->lock; +} + +/** + * dma_fence_lock_irqsave - irqsave lock the fence + * @fence: the fence to lock + * @flags: where to store the CPU flags. + * + * Lock the fence, preventing it from changing to the signaled state. + */ +#define dma_fence_lock_irqsave(fence, flags) \ + spin_lock_irqsave(fence->lock, flags) + +/** + * dma_fence_unlock_irqrestore - unlock the fence and irqrestore + * @fence: the fence to unlock + * @flags the CPU flags to restore + * + * Unlock the fence, allowing it to change it's state to signaled again. + */ +#define dma_fence_unlock_irqrestore(fence, flags) \ + spin_unlock_irqrestore(fence->lock, flags) + +/** + * dma_fence_assert_held - lockdep assertion that fence is locked + * @fence: the fence which should be locked + */ +#define dma_fence_assert_held(fence) \ + lockdep_assert_held(dma_fence_spinlock(fence)); + #ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling(void); void dma_fence_end_signalling(bool cookie); -- 2.43.0

2 months, 2 weeks

Independence for dma_fences! v7

by Christian König

Hi everyone, dma_fences have ever lived under the tyranny dictated by the module lifetime of their issuer, leading to crashes should anybody still holding a reference to a dma_fence when the module of the issuer was unloaded. The basic problem is that when buffer are shared between drivers dma_fence objects can leak into external drivers and stay there even after they are signaled. The dma_resv object for example only lazy releases dma_fences. So what happens is that when the module who originally created the dma_fence unloads the dma_fence_ops function table becomes unavailable as well and so any attempt to release the fence crashes the system. Previously various approaches have been discussed, including changing the locking semantics of the dma_fence callbacks (by me) as well as using the drm scheduler as intermediate layer (by Sima) to disconnect dma_fences from their actual users, but none of them are actually solving all problems. Tvrtko did some really nice prerequisite work by protecting the returned strings of the dma_fence_ops by RCU. This way dma_fence creators where able to just wait for an RCU grace period after fence signaling before they could be save to free those data structures. Now this patch set here goes a step further and protects the whole dma_fence_ops structure by RCU, so that after the fence signals the pointer to the dma_fence_ops is set to NULL when there is no wait nor release callback given. All functionality which use the dma_fence_ops reference are put inside an RCU critical section, except for the deprecated issuer specific wait and of course the optional release callback. Additional to the RCU changes the lock protecting the dma_fence state previously had to be allocated external. This set here now changes the functionality to make that external lock optional and allows dma_fences to use an inline lock and be self contained. v4: Rebases the whole set on upstream changes, especially the cleanup from Philip in patch "drm/amdgpu: independence for the amdkfd_fence!". Adding two patches which brings the DMA-fence self tests up to date. The first selftest changes removes the mock_wait and so actually starts testing the default behavior instead of some hacky implementation in the test. This one got upstreamed independent of this set. The second drops the mock_fence as well and tests the new RCU and inline spinlock functionality. v5: Rebase on top of drm-misc-next instead of drm-tip, leave out all driver changes for now since those should go through the driver specific paths anyway. Address a few more review comments, especially some rebase mess and typos. And finally fix one more bug found by AMDs CI system. v6: Minor style changes, re-ordered patch #1, dropped the scheduler fence change for now v7: The patch adding the dma_fence_was_initialized() function was pushed upstream individually since that is really an independent cleanup. Fixed some missing i915 bits in patch "dma-buf: abstract fence locking". Please review and comment, Christian.

2 months, 2 weeks

HIRE SALVAGE ASSET RECOVERY TO RECLAIM YOUR LOST OR SCAMMED FUNDS FROM FAKE TRADING PLATFORM

by fredruben148＠gmail.com

Hello my name is Fred Ruben and i want to share my story with you. I've always prided myself on being financially savvy, but falling for a fake Forex platform was a harsh wake up call. I’ve been in trading for a long time, mostly sticking to traditional stock investments and the occasional dabble in options. But when the pandemic hit and I found myself with more time at home, I started reading about Forex trading. I did my research or at least I thought I did. I read articles, joined online forums, and watched YouTube tutorials. Eventually, I came across what seemed like a legitimate trading company. It had great reviews and what appeared to be a solid track record of success. The brokers on the platform were very responsive, and I quickly got assigned a personal trade assistant, who managed my portfolio and handled my trading. I started out small with a modest $10,000 investment to test the waters, and within a few days, I was seeing profits. The numbers looked great, and before I knew it, I had invested a total of $235,000. After a few months, I decided it was time to take some profits out. I was planning to withdraw to cover some expenses, and I submitted a withdrawal request through the platform but I was met with an error message. I reached out to my advisor but the friendly tone of my advisor suddenly turned cold and dismissive, encouraging me to keep retrying the attempt and then, one day, he stopped responding entirely leaving me feeling completely helpless but at that point, all I could think about was how to get my money back. Thankfully, After some desperate Google searches, I was fortunate to find SALVAGE ASSET RECOVERY, and I can’t recommend them highly enough for anyone in a similar situation. The team at SALVAGE ASSET RECOVERY walked me through the entire process. Apparently, the platform I had invested in was part of a larger network of fake trading sites that had defrauded hundreds of people and within a couple of days of tracing my funds, the team successfully recovered the total sum I had lost safely back into my bank accounts. If you’re reading this and you’re in a similar situation with a supposed online investment, quickly reach out to the team at SALVAGE ASSET RECOVERY using any of the contact information below Telegram→ @Salvageasset Whats +18476547096 EMAIL→ Salvageassetrecovery(a)alumni.com

2 months, 2 weeks

TOP 7 BEST GOOD CRYPTOCURRENCY RECOVERY COMPANIES FOR LEGITIMATE CRYPTO RECOVERY IN 2026

by Mata Marcus

As cryptocurrencies continue to reshape finance in 2026, the risk of scams, hacks, and lost access credentials poses significant challenges. Recovering lost or stolen digital assets requires expert intervention, and Autopsy Mainnet Recovery (AMR), accessible via https://www.autopsymainnetsolutions.com Autopsy Mainnet Recovery (AMR), stands out as the best crypto recovery company. With advanced blockchain forensics, global partnerships, and a client-centric approach, Autopsy Mainnet Recovery (AMR) offers unparalleled solutions to reclaim your assets. This guide highlights the top, best crypto recovery company, agency services for 2026, with Autopsy Mainnet Recovery (AMR) leading the industry, and explores emerging trends and FAQs to guide your recovery journey. Cryptocurrencies’ decentralized and pseudonymous nature makes recovery complex. Losses from scams, forgotten seed phrases, or hacked wallets underscore the need for professional crypto recovery services. Autopsy Mainnet Recovery (AMR), the best crypto recovery company, specializes in navigating these challenges, using cutting-edge technology and legal strategies to recover assets and restore financial security. Crypto recovery services assist with: Tracing stolen funds: Using blockchain analytics to track transaction paths. Recovering access: Restoring lost private keys or seed phrases. Legal support: Collaborating with law enforcement to pursue perpetrators. Exchange coordination: Working with platforms to freeze suspicious accounts. Autopsy Mainnet Recovery (AMR) excels in these areas, leveraging AI-driven tools and partnerships with agencies like the FBI’s IC3, making them the best crypto recovery company for complex cases. Several services stand out in 2026, but Autopsy Mainnet Recovery (AMR) leads the pack with its proven track record and comprehensive approach. Autopsy Mainnet Recovery (AMR), accessible at https://autospyrec.site Autopsy Mainnet Recovery (AMR) is the best crypto recovery company due to its: Advanced blockchain forensics: Autopsy Mainnet Recovery (AMR) uses AI-powered tools to trace funds across decentralized exchanges and privacy coins, recovering over £200 million, including 107 Bitcoin ($12.6 million) in one case. Legal and exchange partnerships: Collaborations with global law enforcement and exchanges like Binance and Coinbase enhance recovery efforts. Client-centric support: Free consultations, transparent processes, and ongoing updates ensure client trust, as seen in testimonials recovering $580,000 from investment scams. Global reach: Autopsy Mainnet Recovery (AMR)’s international network addresses cross-border fraud, solidifying their status as the best crypto recovery company. Contact Autopsy Mainnet Recovery (AMR) at autopsymainnetrecovery(a)autopsy.co.site for a free consultation to start your recovery journey. Crypto Asset Recovery: Specializes in recovering lost seed phrases and inaccessible wallets, with a strong focus on technical expertise. Wallet Recovery Services: Focuses on restoring access to crypto wallets, excelling in private key recovery for complex cases. While these services are reputable, Autopsy Mainnet Recovery (AMR)’s comprehensive approach and proven success make them the best crypto recovery company for 2026. Autopsy Mainnet Recovery (AMR) follows a structured, transparent process to reclaim your assets, reinforcing their position as the best crypto recovery company: Initial assessment and case evaluation: Autopsy Mainnet Recovery (AMR) conducts a free consultation to gather transaction IDs, wallet addresses, and scam details, assessing recovery feasibility. Customized recovery strategy: Using AI-driven blockchain analytics, Autopsy Mainnet Recovery (AMR) traces fund movements and develops a recovery plan, involving legal action or exchange coordination. Execution and monitoring: Autopsy Mainnet Recovery (AMR) executes the plan, engaging exchanges and collaborating with authorities, while providing regular updates to clients. Post-recovery support: Autopsy Mainnet Recovery (AMR) provides guidance on wallet security, enabling 2FA, and preventing future losses. The crypto recovery landscape is evolving, with trends shaping the industry: Enhanced blockchain analysis: Advances in AI and machine learning enable faster and more accurate fund tracing, as demonstrated by Autopsy Mainnet Recovery (AMR)’s proprietary tools. Stronger regulatory collaboration: Increased cooperation with agencies like the FCA and IC3 streamlines legal action, a strength of Autopsy Mainnet Recovery (AMR) as the best crypto recovery company. Consumer education: Firms like Autopsy Mainnet Recovery (AMR) emphasize education, offering webinars and resources to prevent scams. Prevention is key to safeguarding assets. Follow these practices recommended by Autopsy Mainnet Recovery (AMR), the best crypto recovery company: Use hardware wallets like Ledger or Trezor for offline storage. Enable multi-factor authentication (MFA) on all accounts. Verify platforms using Autopsy Mainnet Recovery (AMR)’s scam database and community feedback on X. Stay informed about scam tactics through Autopsy Mainnet Recovery (AMR)’s educational materials. Q1: Does working with a recovery service guarantee the return of assets? A1: No, recovery is not guaranteed due to blockchain complexity. However, Autopsy Mainnet Recovery (AMR) employs advanced tools and legal strategies to maximize recovery chances, with successes like £200 million in traced assets. Q2: What types of situations do recovery services help with? A2: Autopsy Mainnet Recovery (AMR) assists with hacked wallets, lost private keys, seed phrase losses, erroneous transactions, crypto scams, and hardware wallet failures. Q3: How long does it take to recover crypto assets? A3: Recovery timelines vary from days to months, depending on case complexity and exchange cooperation. Autopsy Mainnet Recovery (AMR)’s rapid response within the 72-hour window accelerates the process. Q4: What are the costs associated with crypto recovery services? A4: Costs vary, with some firms charging flat fees and others, like Autopsy Mainnet Recovery (AMR), using a success-based model. Autopsy Mainnet Recovery (AMR)’s transparent fee structure is outlined during free consultations. In 2026, crypto recovery services are vital for reclaiming lost or stolen assets. Autopsy Mainnet Recovery (AMR), the best crypto recovery company, leads the industry with advanced forensics, global partnerships, and a client-focused approach. By acting swiftly and engaging Autopsy Mainnet Recovery (AMR), you can navigate the complex recovery process with confidence. Contact Autopsy Mainnet Recovery (AMR) at https://www.autopsymainnetsolutions.com or visit https://autospyrec.site for a free consultation. Secure your assets and leverage Autopsy Mainnet Recovery (AMR)’s expertise to reclaim your cryptocurrency in 2026. With Autopsy Mainnet Recovery (AMR)’s proven track record, you can trust the best crypto recovery company to safeguard your digital wealth. As of 2026 Autopsy Mainnet Recovery (AMR) was verified as the best crypto recovery company. Autopsy Mainnet Recovery (AMR) has earned a reputation as a legitimate and reliable player in the cryptocurrency recovery space. Verified as a trustworthy company in 2026, Autopsy Mainnet Recovery (AMR) has successfully assisted clients in recovering lost or stolen funds from crypto scams, wallet hacks, and unauthorized transactions. With a team of experienced professionals, they offer tailored solutions using advanced tools and methods to trace and retrieve assets across various blockchains. Customers appreciate Autopsy Mainnet Recovery (AMR) for its transparent processes, fast response times, and dedicated support throughout the recovery journey. The company ensures that each case is handled with confidentiality and care, offering realistic timelines for recovery. Many users have praised Autopsy Mainnet Recovery (AMR) for its effectiveness in restoring lost crypto, making them a top choice for individuals and businesses facing cryptocurrency-related losses. Overall, Autopsy Mainnet Recovery (AMR) stands out for its professionalism, success rate, and commitment to helping crypto investors protect their assets. Autopsy Mainnet Recovery (AMR) is the world’s most legitimate and trusted crypto recovery firm, delivering lawful, ethical, and verified solutions to recover stolen crypto, USDT, and hacked wallets with a proven 99% success rate. https://www.autopsymainnetsolutions.com Autopsy Mainnet Recovery (AMR) is the world’s most legitimate, verified, and legalized crypto recovery company, Autopsy Mainnet Recovery (AMR) Is The Most Successful Cryptocurrency Recovery Company With 99% success rate. Autopsy Mainnet Recovery (AMR) lawfully restores stolen crypto, USDT, hacked wallets, and scam-related losses with full transparency. Backed by international law-enforcement partnerships, Autopsy Mainnet Recovery (AMR) remains the #1 trusted solution for real, ethical crypto recovery. Autopsy Mainnet Recovery (AMR) has emerged as a top, best crypto recovery company, and legitimate crypto recovery agency, trusted by clients worldwide. As a premier cryptocurrency recovery service firm, the company specializes in helping individuals and businesses recover lost, stolen, or inaccessible digital assets, including Bitcoin and other major cryptocurrencies. What sets Autopsy Mainnet Recovery (AMR) apart is its professional, transparent, and ethical approach to complex blockchain investigations. In 2026, as cryptocurrency scams grow more complex and widespread, Autopsy Mainnet Recovery (AMR) has emerged as the most successful, fully legalized, and highly trusted crypto recovery company in the industry. Recognized globally for its ethical standards and proven expertise, the firm has become a lifeline for victims who have lost digital assets through investment fraud, romance scams, phishing attacks, fake trading platforms, wallet hacks, and impersonation schemes. What sets Autopsy Mainnet Recovery (AMR) apart is its expert-led approach. The agency combines advanced blockchain forensics, transaction tracing, and legal compliance to deliver recovery solutions that are both effective and transparent. Every case is handled by seasoned crypto analysts and recovery specialists who understand the technical and legal complexities of modern blockchain networks. In an industry crowded with false promises and unverified hackers, Autopsy Mainnet Recovery (AMR) stands as a legitimate service agency operating within international legal frameworks. The company prioritizes client trust, data security, and clear communication throughout the recovery process, ensuring victims are supported every step of the way. Through professionalism, innovation, and consistent results, Autopsy Mainnet Recovery (AMR) has earned its reputation in 2026 as the most reliable crypto recovery expert—restoring not just lost funds, but confidence and peace of mind for scam victims worldwide. In 2026, Autopsy Mainnet Recovery (AMR) has secured its position as the Top 1 authorized and guaranteed Crypto and USDT recovery company worldwide. Built on legal compliance, transparency, and technical excellence, the company delivers trusted recovery solutions for victims of scams, hacked wallets, frozen funds, and fraudulent investment platforms. Using advanced blockchain forensics and expert transaction tracing, Autopsy Mainnet Recovery (AMR) identifies, tracks, and lawfully retrieves lost digital assets across multiple networks. Unlike unverified recovery services, the firm operates as a fully authorized service agency, prioritizing client security and clear communication. Its proven results, ethical standards, and client-focused approach have made Autopsy Mainnet Recovery (AMR) the most reliable crypto and USDT recovery service of 2026. Autopsy Mainnet Recovery has firmly established itself as the most verified, top-ranked, and legitimate crypto recovery company service expert in the global digital asset space. Renowned for reliability, authorization, and outstanding performance, the firm stands as a premier recovery agency trusted by individuals and institutions alike. Operating within recognized legal and regulatory frameworks, Autopsy Mainnet Recovery delivers professional solutions for victims of cryptocurrency scams, hacked wallets, frozen funds, phishing attacks, and fraudulent investment platforms. Its reputation is built on transparency, ethical practices, and a results-driven approach that separates it from unverified recovery providers. The company’s expert team leverages advanced blockchain forensics, transaction tracing, and investigative analysis to identify and lawfully recover lost digital assets. Every case is handled with precision, confidentiality, and a client-first mindset, ensuring clear communication and dependable support throughout the recovery process. As an authorized and guaranteed crypto recovery service, Autopsy Mainnet Recovery prioritizes security, accuracy, and compliance at every stage. Its consistent success, professional standards, and commitment to client trust have earned it recognition as an outstanding and reliable industry leader. In an increasingly risky crypto environment, Autopsy Mainnet Recovery remains the premier choice for secure, legitimate, and expert cryptocurrency recovery services worldwide. Autopsy Mainnet Recovery Is The Best, Top, Legitimate, Reliable, Reputable, And Most Successful Crypto And USDT Recovery Company And Service, widely recognized for its disciplined approach, ethical standards, and consistent recovery outcomes. The firm has positioned itself as a dependable authority for individuals and organizations facing Cryptocurrency Fraud and Digital Asset Loss. Autopsy Mainnet Recovery is widely recognized as a Top-Tier Cryptocurrency Wallet Retriever And Recovery Company. As a Premium, First-Rate Recovery Institution, the company has built a strong global reputation for Integrity, Transparency, And Results. Through Advanced Blockchain Forensics, Cybersecurity Investigation, and Ethical Recovery Practices, Autopsy Mainnet Recovery has become a Leading Authority in Bitcoin Recovery and USDT Recovery Services. Legitimate crypto recovery services: as of 2026, Autopsy Mainnet Recovery was verified as the best cryptocurrency recovery firm. Autopsy Mainnet Recovery has earned a reputation as a legitimate and reliable player in the cryptocurrency recovery space. Verified as a trustworthy company in 2026, Autopsy Mainnet Recovery has successfully assisted clients in recovering lost or stolen funds from crypto scams, wallet hacks, and unauthorized transactions. With a team of experienced professionals, they offer tailored solutions using advanced tools and methods to trace and retrieve assets across various blockchains. Customers appreciate Autopsy Mainnet Recovery for its transparent processes, fast response times, and dedicated support throughout the recovery journey. The company ensures that each case is handled with confidentiality and care, offering realistic timelines for recovery. Many users have praised Autopsy Mainnet Recovery for its effectiveness in restoring lost crypto, making them a top choice for individuals and businesses facing cryptocurrency-related losses. Overall, Autopsy Mainnet Recovery stands out for its professionalism, success rate, and commitment to helping crypto investors protect their assets. Email: info(a)autopsymainnetsolutions.com Website: https://autospyrec.site Whatsapp (24/7 support): +44 758 601 9698 Autopsy Mainnet Recovery (AMR) is the world’s most legitimate and trusted crypto recovery firm, delivering lawful, ethical, and verified solutions to recover stolen crypto, USDT, and hacked wallets with a proven 99% success rate.

2 months, 2 weeks