Linaro-mm-sig

linaro-mm-sig@lists.linaro.org

93 participants
6571 discussions

[PATCH] drm/amdgpu: fix recursive ww_mutex acquire in amdgpu_devcoredump_format

by Mikhail Gavrilov

When dumping IB contents from a hung job, amdgpu_devcoredump_format() acquires the VM root PD's reservation lock via amdgpu_vm_lock_by_pasid() and then, for each IB referenced by the job, calls amdgpu_bo_reserve() on the BO that backs the IB. Both reservations are taken on reservation_ww_class_mutex objects but neither uses a ww_acquire_ctx, which trips lockdep: WARNING: possible recursive locking detected -------------------------------------------- kworker/u128:0 is trying to acquire lock: ffff88838b16e1f0 (reservation_ww_class_mutex){+.+.}-{4:4}, at: amdgpu_devcoredump_format+0x1594/0x23f0 [amdgpu] but task is already holding lock: ffff8882f82681f0 (reservation_ww_class_mutex){+.+.}-{4:4}, at: amdgpu_devcoredump_format+0x1594/0x23f0 [amdgpu] Possible unsafe locking scenario: CPU0 ---- lock(reservation_ww_class_mutex); lock(reservation_ww_class_mutex); *** DEADLOCK *** May be due to missing lock nesting notation Workqueue: events_unbound amdgpu_devcoredump_deferred_work [amdgpu] Call Trace: __ww_mutex_lock.constprop.0 ww_mutex_lock amdgpu_bo_reserve amdgpu_devcoredump_format+0x1594 [amdgpu] amdgpu_devcoredump_deferred_work+0xea [amdgpu] process_one_work worker_thread kthread The two reservations are on different BOs in the captured trace, so the splat is a lockdep-correctness warning, not an observed deadlock. It becomes a real self-deadlock whenever the IB BO shares its dma_resv with the root PD (the always-valid case, see amdgpu_vm_is_bo_always_valid()): amdgpu_bo_reserve(abo) re-acquires the same ww_mutex without a ticket and blocks forever. With amdgpu.gpu_recovery=0 the timeout handler refires every ~2 s and each invocation produces this splat, drowning the kernel ring buffer. Fix it by collecting the per-IB BO references under the root PD's reservation, then releasing the root before reserving each IB BO individually. The walk over the VM mapping tree must remain under the root lock (mappings can be torn down without it), but the actual content copies do not need to nest inside it. Each per-IB reservation is now an independent top-level acquire, eliminating the nested ww_mutex. The collect/release logic is factored out into two small helpers (amdgpu_devcoredump_collect_ib_refs / amdgpu_devcoredump_release_ib_refs) to keep the main function's indentation reasonable. This also fixes a BO refcount leak in the original code: when amdgpu_bo_reserve() failed, control jumped to free_ib_content without running amdgpu_bo_unref(). In the new structure the per-IB BO refs are released unconditionally in the cleanup helper. Reproducer (~150 LoC libdrm_amdgpu): submit a single GFX IB containing PACKET3_INDIRECT_BUFFER chained at GPU VA 0 and wait for the fence. The TDR fires within ~10 s and the deferred coredump worker produces the splat above on every invocation. Fixes: 7b15fc2d1f1a ("drm/amdgpu: dump job ibs in the devcoredump") Cc: stable(a)vger.kernel.org # 7.1 Signed-off-by: Mikhail Gavrilov <mikhail.v.gavrilov(a)gmail.com> --- .../gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c | 147 +++++++++++++----- 1 file changed, 110 insertions(+), 37 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c index d386bc775d03..f6bb968de756 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dev_coredump.c @@ -207,6 +207,72 @@ static void amdgpu_devcoredump_fw_info(struct amdgpu_device *adev, } } +struct amdgpu_devcoredump_ib_ref { + struct amdgpu_bo *bo; + u64 offset; +}; + +/* + * Walk the VM's mapping tree under the root PD's reservation to obtain the BO + * that backs each IB and pin it with a refcount. The root PD reservation is + * dropped before this function returns; the caller can then reserve each IB + * BO individually without nesting ww_mutex acquires on + * reservation_ww_class_mutex. + * + * Returns an array of num_ibs entries (each ib_refs[i].bo may be NULL if its + * mapping was not found), or NULL on allocation failure / VM lookup failure. + * The caller must release the BO refs and free the array. + */ +static struct amdgpu_devcoredump_ib_ref * +amdgpu_devcoredump_collect_ib_refs(struct amdgpu_device *adev, + struct amdgpu_coredump_info *coredump) +{ + struct amdgpu_devcoredump_ib_ref *ib_refs; + struct amdgpu_bo_va_mapping *mapping; + struct amdgpu_bo *root; + struct amdgpu_vm *vm; + u64 va_start; + + ib_refs = kcalloc(coredump->num_ibs, sizeof(*ib_refs), GFP_KERNEL); + if (!ib_refs) + return NULL; + + vm = amdgpu_vm_lock_by_pasid(adev, &root, coredump->pasid); + if (!vm) { + kfree(ib_refs); + return NULL; + } + + for (int i = 0; i < coredump->num_ibs; i++) { + va_start = coredump->ibs[i].gpu_addr & AMDGPU_GMC_HOLE_MASK; + mapping = amdgpu_vm_bo_lookup_mapping(vm, va_start / AMDGPU_GPU_PAGE_SIZE); + if (!mapping) + continue; + + ib_refs[i].bo = amdgpu_bo_ref(mapping->bo_va->base.bo); + ib_refs[i].offset = va_start - + mapping->start * AMDGPU_GPU_PAGE_SIZE; + } + + amdgpu_bo_unreserve(root); + amdgpu_bo_unref(&root); + + return ib_refs; +} + +static void +amdgpu_devcoredump_release_ib_refs(struct amdgpu_devcoredump_ib_ref *ib_refs, + int num_ibs) +{ + if (!ib_refs) + return; + + for (int i = 0; i < num_ibs; i++) + if (ib_refs[i].bo) + amdgpu_bo_unref(&ib_refs[i].bo); + kfree(ib_refs); +} + static ssize_t amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_info *coredump) { @@ -214,13 +280,11 @@ amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_inf struct drm_printer p; struct drm_print_iterator iter; struct amdgpu_vm_fault_info *fault_info; - struct amdgpu_bo_va_mapping *mapping; struct amdgpu_ip_block *ip_block; struct amdgpu_res_cursor cursor; - struct amdgpu_bo *abo, *root; - uint64_t va_start, offset; + struct amdgpu_bo *abo; + uint64_t offset; struct amdgpu_ring *ring; - struct amdgpu_vm *vm; u32 *ib_content; uint8_t *kptr; int ver, i, j, r; @@ -343,43 +407,52 @@ amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_inf drm_printf(&p, "VRAM is lost due to GPU reset!\n"); if (coredump->num_ibs) { - /* Don't try to lookup the VM or map the BOs when calculating the - * size required to store the devcoredump. + struct amdgpu_devcoredump_ib_ref *ib_refs = NULL; + + /* + * Snapshot per-IB BO references under the root PD's reservation, + * then release the root before reserving each IB BO individually + * to copy its contents. + * + * Reserving an IB BO while the root PD is still reserved would + * be a nested ww_mutex acquire on reservation_ww_class_mutex + * without a ww_acquire_ctx, which trips lockdep's recursive- + * locking check and self-deadlocks for IB BOs that share their + * dma_resv with the root PD (always-valid BOs). + * + * Skip lookup/reservation entirely on the sizing pass: it does + * not write IB content, and the size estimate doesn't depend on + * whether the BOs are reachable. */ - if (sizing_pass) - vm = NULL; - else - vm = amdgpu_vm_lock_by_pasid(adev, &root, coredump->pasid); + if (!sizing_pass) + ib_refs = amdgpu_devcoredump_collect_ib_refs(adev, coredump); - for (int i = 0; i < coredump->num_ibs && (sizing_pass || vm); i++) { + for (int i = 0; i < coredump->num_ibs; i++) { ib_content = kvmalloc_array(coredump->ibs[i].ib_size_dw, 4, GFP_KERNEL); if (!ib_content) continue; - /* vm=NULL can only happen when 'sizing_pass' is true. Skip to the - * drm_printf() calls (ib_content doesn't need to be initialized - * as its content won't be written anywhere). - */ - if (!vm) + if (sizing_pass) goto output_ib_content; - va_start = coredump->ibs[i].gpu_addr & AMDGPU_GMC_HOLE_MASK; - mapping = amdgpu_vm_bo_lookup_mapping(vm, va_start / AMDGPU_GPU_PAGE_SIZE); - if (!mapping) - goto free_ib_content; + if (!ib_refs || !ib_refs[i].bo) + goto output_ib_content; + + abo = ib_refs[i].bo; + offset = ib_refs[i].offset; - offset = va_start - (mapping->start * AMDGPU_GPU_PAGE_SIZE); - abo = amdgpu_bo_ref(mapping->bo_va->base.bo); r = amdgpu_bo_reserve(abo, false); if (r) - goto free_ib_content; + goto output_ib_content; if (abo->flags & AMDGPU_GEM_CREATE_NO_CPU_ACCESS) { off = 0; - if (abo->tbo.resource->mem_type != TTM_PL_VRAM) - goto unreserve_abo; + if (abo->tbo.resource->mem_type != TTM_PL_VRAM) { + amdgpu_bo_unreserve(abo); + goto output_ib_content; + } amdgpu_res_first(abo->tbo.resource, offset, coredump->ibs[i].ib_size_dw * 4, @@ -395,8 +468,10 @@ amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_inf r = ttm_bo_kmap(&abo->tbo, 0, PFN_UP(abo->tbo.base.size), &abo->kmap); - if (r) - goto unreserve_abo; + if (r) { + amdgpu_bo_unreserve(abo); + goto output_ib_content; + } kptr = amdgpu_bo_kptr(abo); kptr += offset; @@ -406,21 +481,19 @@ amdgpu_devcoredump_format(char *buffer, size_t count, struct amdgpu_coredump_inf amdgpu_bo_kunmap(abo); } + amdgpu_bo_unreserve(abo); + output_ib_content: drm_printf(&p, "\nIB #%d 0x%llx %d dw\n", i, coredump->ibs[i].gpu_addr, coredump->ibs[i].ib_size_dw); - for (int j = 0; j < coredump->ibs[i].ib_size_dw; j++) - drm_printf(&p, "0x%08x\n", ib_content[j]); -unreserve_abo: - if (vm) - amdgpu_bo_unreserve(abo); -free_ib_content: + if (!sizing_pass && ib_refs && ib_refs[i].bo) { + for (int j = 0; j < coredump->ibs[i].ib_size_dw; j++) + drm_printf(&p, "0x%08x\n", ib_content[j]); + } kvfree(ib_content); } - if (vm) { - amdgpu_bo_unreserve(root); - amdgpu_bo_unref(&root); - } + + amdgpu_devcoredump_release_ib_refs(ib_refs, coredump->num_ibs); } return count - iter.remain; -- 2.54.0

9 hours, 34 minutes

Re: [PATCH v2] drm/prime: fix dangling dmabuf entries after handle release

by Christian König

Hi Mingyu, On 5/28/26 15:49, w15303746062 wrote: > Hi Christian, > > Thank you for insisting on this. I've now gone through all callers > of drm_prime_add_buf_handle() in drm_prime.c. > > You are absolutely right: both drm_gem_prime_fd_to_handle() and > drm_gem_prime_handle_to_dmabuf() perform the lookup under > prime_fpriv->lock before adding, so a duplicate handle should indeed > never be inserted through those paths. > > That said, the syzkaller report clearly shows that the dmabufs tree > is not empty when drm_prime_destroy_file_private() runs, which means > some entry wasn't removed. Given that the normal add/remove paths > appear correct, the trigger might be something more subtle — perhaps > a driver-specific callback that bypasses the generic helpers, or an > error path that leaves an orphan in the dmabufs tree. I haven't been > able to identify the exact trigger yet. > > The proposed change to drm_prime_remove_buf_handle() (restart search > instead of break) is intended as a small robustness improvement, not > a fix for a confirmed race. In the normal case it will still execute > only once, but if the trees ever become inconsistent for any reason, > it will clean up all entries for the given handle and prevent the > WARNING. > > Would you be okay with such a defensive approach, or would you prefer > that we first track down the precise trigger (e.g. with additional > WARNs or tracing)? I don't think so. As far as I can see this is not a robustness improvement but just papering over an issue. Leaking memory is usually only a very minor problem, things like use after free or random memory corruption is much more worse. And such things is exactly what starts to happens when you start papering over issues. So I would say find the root cause of what is going on here, you have certainly stumbled over something, and then we can look into how to fix that. But just sending out random patches where a bit of simple code reading can prove them incorrect is not really helpful. Regards, Christian. > > Thanks, > Mingyu

9 hours, 36 minutes

A Guide to Slither io and the Joy of Simple Gaming

by kaywilson＠wshu.net

In a world filled with hyper-realistic graphics and complex narratives, sometimes the most captivating experiences are the simplest. Enter Slither io, a game that takes the classic "snake" concept and injects it with a healthy dose of online multiplayer mayhem. If you've ever found yourself drawn to the addictive nature of growing a digital creature, or the thrill of outsmarting your opponents in a low-stakes, high-fun environment, then read on. This article will guide you through the basics of Slither io and offer some insights into how to master its charmingly straightforward gameplay. What is Slither io? A Journey into the World of Glowing Serpents Slither io is an online multiplayer arcade game where you control a vibrant, glowing snake. Your objective is deceptively simple: grow your snake by consuming glowing pellets scattered across the map, while avoiding collisions with other players. The twist? Unlike traditional snake games, touching another snake's body (not just its head) means instant demise. This core mechanic creates a dynamic and surprisingly strategic environment where even the smallest snake can take down the largest, given the right timing and tactical positioning. The game's accessibility, with its browser-based play and intuitive controls, makes it a perfect pick-up-and-play experience for anyone looking to unwind and have a little competitive fun. The Dance of the Serpent: Understanding Slither io Gameplay Getting started in Slither io is remarkably easy. Upon loading the game, you'll be prompted to enter a nickname. Once you're in, you'll find yourself as a small, brightly colored snake on a vast, dark arena filled with glowing food. https://slitherio.onl Movement: Your snake follows your mouse cursor. Simply move your mouse to guide your snake in the desired direction. There are no "up, down, left, right" keys – it's all about smooth, continuous movement. Boosting: This is where the strategic element truly shines. By holding down the left mouse button (or spacebar), your snake will accelerate rapidly. While boosting allows you to quickly grab food or outmaneuver opponents, it comes at a cost: your snake will shrink slightly as it consumes a portion of its own mass to fuel the boost. Mastering when and where to boost is crucial for survival and growth. Eating: The primary way to grow your snake is by consuming the colorful pellets scattered across the map. These pellets appear naturally and also drop from deceased snakes. Larger snakes leave behind more food, creating opportunities for smaller snakes to quickly bulk up. Eliminating Opponents: The most satisfying (and often frustrating) part of Slither io is taking down other players. To do this, you need to trick another snake into colliding with your body. There are various tactics, such as circling an opponent, cutting them off, or using a well-timed boost to get in front of their head. When a snake dies, it explodes into a burst of highly nutritious pellets, ready for consumption. Tips for Serpent Supremacy: Mastering the Arena While the gameplay is simple, becoming a truly massive snake requires a blend of patience, observation, and clever tactics. Patience is a Virtue: Don't rush into every engagement. As a small snake, your primary goal is to grow. Focus on consuming loose pellets and avoiding larger players. The Art of the Boost: Use your boost wisely. It's excellent for escaping dangerous situations or quickly grabbing a cluster of food, but over-boosting will shrink you unnecessarily. Learn to anticipate where food will be and use short bursts to get there efficiently. Circle of Life (and Death): Once you've grown to a respectable size, you can start to "circle" smaller snakes. By gradually enclosing them, you force them into a smaller and smaller space until they inevitably collide with your body. Embrace the Edge: The edges of the map can be surprisingly safe, as fewer players tend to congregate there. It’s a good place to grow in relative peace during the early stages of your game. Capitalize on Chaos: When a large snake dies, it creates a feeding frenzy. While it's tempting to dive in immediately, observe the situation. Often, other players will collide in their eagerness, leaving even more food for a patient observer. Don't Be Afraid to Die: Seriously! Each death is a learning opportunity. Pay attention to how you were eliminated and try to avoid making the same mistake twice. The beauty of Slither io is that you can restart instantly and jump back into the action. The Endless Dance: Why Slither io Remains Relevant Slither io, available to play at Slither io, is more than just a casual game; it's a testament to the power of simple yet engaging mechanics. Its low barrier to entry, combined with surprisingly deep strategic possibilities, makes it a timeless classic. Whether you're looking for a quick break, a competitive outlet, or just a chance to unwind, embracing your inner serpent in the vibrant world of Slither io is an experience well worth trying. Go forth, consume, and dominate the leaderboard!

9 hours, 40 minutes

Terrifying Escape Challenges Inside Granny Horror Survival Adventure

by reasonable.rabbit.epoq＠hidingmail.net

Introduction to Granny Horror Gameplay Granny is one of the most popular horror games in the mobile gaming world. The game creates a tense atmosphere filled with darkness, fear, and unpredictable danger. Players wake up inside an old house and must find a way to escape before time runs out. Every sound can attract danger, and every room may hide useful tools or deadly traps. https://grannyfree.io/ The simple gameplay mechanics make Granny easy to understand, but the increasing tension keeps players engaged for hours. Horror fans enjoy the thrilling experience because every match feels different. Random item locations and unpredictable enemy movement create excitement in every playthrough. The game combines stealth mechanics, puzzle-solving, and survival elements into a complete horror adventure. Players must stay quiet, move carefully, and think strategically to survive inside the terrifying house. Dark Atmosphere and Immersive Horror Experience Creepy Environment and Sound Design One of the strongest features of Granny is the frightening atmosphere. The old house contains dark hallways, broken furniture, hidden basements, and mysterious locked doors. Every area creates tension and uncertainty. The sound effects increase the horror experience dramatically. Footsteps, creaking floors, and sudden noises make every movement feel dangerous. Silence becomes just as terrifying as loud sounds because players never know where danger may appear next. The visual design also supports the horror theme. Dim lighting and shadow-filled rooms create suspense throughout the game. Even simple exploration becomes stressful because danger may appear at any moment. Fear Created Through Constant Pressure Granny creates fear through pressure instead of excessive action. Players cannot fight freely or move carelessly. One wrong decision may lead to immediate failure. This constant danger keeps players focused and emotionally involved. The limited time system adds more intensity. Players usually have only a few days to escape the house. Each failed attempt increases tension and forces players to learn from mistakes. The combination of stealth and survival mechanics creates memorable horror moments that many players continue discussing online. Survival Mechanics That Keep Players Engaged Stealth Gameplay and Smart Movement Stealth is the core mechanic of Granny. Running loudly or dropping objects may attract danger instantly. Players must crouch, hide under beds, and move slowly to avoid detection. Different areas of the house require careful observation. Some floors create noise, while certain doors open slowly and loudly. Understanding the environment becomes essential for survival. The game rewards patience and strategic thinking. Quick reactions help during emergencies, but long-term success depends on planning and observation. Puzzle Solving and Exploration Granny includes many puzzles that require exploration and logical thinking. Players must search for keys, tools, batteries, and hidden objects throughout the house. Many items have multiple purposes. A hammer may remove wooden barriers, while a key may unlock important escape routes. Players need to remember item locations and manage resources efficiently. The puzzles remain simple enough for beginners but challenging enough to maintain excitement. Exploration becomes rewarding because every discovered item may help progress toward freedom. Different Escape Routes and Replay Value Multiple Ways to Escape One reason behind the popularity of Granny is the variety of escape methods. Players can unlock doors, repair vehicles, or discover secret routes hidden inside the house. Different strategies create different experiences. Some players focus on quick escapes, while others explore every room carefully. This flexibility keeps the gameplay fresh over multiple sessions. The hidden secrets encourage replayability. Many players return to discover alternative endings and hidden features that they missed during earlier attempts. Difficulty Levels for All Players Granny includes several difficulty settings that make the game accessible for different skill levels. Beginners can choose easier modes with fewer dangers, while experienced players can select harder difficulties for more intense gameplay. Harder modes increase enemy speed, reduce available resources, and create additional challenges. These options allow players to customize the horror experience according to personal preference. This adjustable difficulty system helps Granny appeal to both casual gamers and hardcore horror fans. Why Granny Became Popular Worldwide Simple Controls and Accessible Gameplay Granny became successful partly because of its simple controls. Mobile players can easily learn movement, interaction, and hiding mechanics without complicated tutorials. The accessibility allows players from many age groups to enjoy the game. Quick learning combined with intense horror creates an experience that remains entertaining from the first session. The lightweight design also helps the game run smoothly on many devices, increasing global popularity. Viral Content and Streaming Success Many content creators helped Granny gain worldwide attention. Horror reactions, escape challenges, and gameplay videos became extremely popular on streaming platforms and social media. The unpredictable gameplay creates entertaining reactions that viewers enjoy watching. Every scream, failed escape, or close encounter becomes exciting content for audiences. Online communities continue sharing strategies, secrets, and challenge ideas related to Granny. This active fanbase helps maintain interest in the game years after release. Advanced Tips for Better Survival Learn Sound Management Techniques Sound management is extremely important in Granny. Players should avoid throwing objects unnecessarily and close doors carefully whenever possible. Listening carefully also provides valuable information. Enemy movement sounds may reveal safe paths or dangerous locations nearby. Headphones improve the overall experience because directional sound helps players understand threats more accurately. Memorize Important Locations Successful players often memorize room layouts and item spawn locations. Fast navigation becomes critical during dangerous situations. Important escape tools usually appear in random positions, but understanding the general map structure helps players search more efficiently. Practice improves confidence and reduces panic during stressful moments. Horror Elements That Make Granny Unique Psychological Fear Instead of Constant Action Unlike many action-heavy horror games, Granny focuses on psychological tension. Silence, uncertainty, and limited visibility create fear naturally. Players spend much of the game anticipating danger instead of constantly fighting enemies. This slower pacing creates stronger emotional impact. The feeling of helplessness increases immersion and makes every successful escape feel rewarding. Minimalist Design With Strong Impact Granny proves that simple design can still create powerful horror experiences. The game does not rely on advanced graphics or complex mechanics. Instead, strong atmosphere, effective sound design, and intelligent pacing create memorable gameplay. This minimalist approach allows the horror elements to remain the main focus. Many modern indie horror games use similar techniques inspired by the success of Granny. Conclusion Granny continues to attract horror fans because of intense survival gameplay, frightening atmosphere, and engaging escape mechanics. The combination of stealth, puzzles, and psychological horror creates a unique experience that remains entertaining even after multiple playthroughs. Simple controls, multiple difficulty settings, and different escape routes make the game accessible for many players around the world. Whether players enjoy exploration, suspense, or strategic survival, Granny delivers a thrilling horror adventure filled with tension and excitement. For anyone searching for a memorable horror game with challenging survival mechanics, Granny remains an excellent choice in the horror gaming genre.

9 hours, 52 minutes

[PATCH v2 0/9] vfio/pci: Add mmap() for DMABUFs

by Matt Evans

Hi all, This series is based on previous RFCs/discussions: Tech topic: https://lore.kernel.org/linux-iommu/20250918214425.2677057-1-amastro@fb.com/ RFCv1: https://lore.kernel.org/all/20260226202211.929005-1-mattev@meta.com/ RFCv2: https://lore.kernel.org/kvm/20260312184613.3710705-1-mattev@meta.com/ The background/rationale is covered in more detail in the RFC cover letters. The TL;DR is: The goal is to enable userspace driver designs that use VFIO to export DMABUFs representing subsets of PCI device BARs, and "vend" those buffers from a primary process to other subordinate processes by fd. These processes then mmap() the buffers and their access to the device is isolated to the exported ranges. This is an improvement on sharing the VFIO device fd to subordinate processes, which would allow unfettered access. This is achieved by enabling mmap() of vfio-pci DMABUFs, passed by fd to subordinate processes. Second, a new ioctl()-based revocation mechanism is added to allow the primary process to forcibly revoke access to previously-shared BAR spans, even if the subordinate processes haven't cleanly exited. (The related topic of safe delegation of iommufd control to the subordinate processes is not addressed here, and is follow-up work.) As well as isolation and revocation, another advantage to accessing a BAR through a VMA backed by a DMABUF is that it's straightforward to mmap() the buffer with access attributes, such as write-combining. Feedback from the RFCs requested that, instead of creating DMABUF-specific vm_ops and .fault paths, to go the whole way and migrate the existing VFIO PCI BAR mmap() to be backed by a DMABUF too, resulting in a common vm_ops and fault handler for mmap()s of both the VFIO device and explicitly-exported DMABUFs. This will help future iommufd emulation of VFIO Type1 peer-to-peer, making it easier to get a DMABUF for a VFIO BAR as a DMA target. mmap() conversion to use DMABUF underneath has been done for vfio-pci, but not sub-drivers: nvgrace-gpu's mmap() override path is unchanged; I kept this out of scope for now not least because I don't have a thorough test setup for this system. I would prefer to help the nvgrace-gpu maintainers enable BAR mmap() DMABUFs themselves. Notes on patches ================ PCI/P2PDMA: Add CONFIG_PCI_P2PDMA_CORE Later in the series, vfio-pci's mmap() is going to depend on pcim_p2pdma_provider() which depended on CONFIG_PCI_P2PDMA, which in turn depended on ZONE_DEVICE (which isn't available on 32-bit and some archs, because they lack MEMORY_HOTPLUG and friends). VFIO does _not_ require actual P2P to be present for basic mmap() functionality, only for the optional CONFIG_DMA_SHARED_BUFFER feature. This splits P2PDMA into a CONFIG_PCI_P2PDMA_CORE (which currently contains pcim_p2pdma_provider()) and an optional CONFIG_PCI_P2PDMA (which depends on ZONE_DEVICE etc., and provides P2P functionality). vfio/pci: Add a helper to look up PFNs for DMABUFs vfio/pci: Add a helper to create a DMABUF for a BAR-map VMA The first is for a DMABUF VMA fault handler to determine arbitrary-sized PFNs from ranges in DMABUF. Secondly, refactor DMABUF export for use by the existing export feature and add a new helper that creates a DMABUF corresponding to a VFIO BAR mmap() request. vfio/pci: Convert BAR mmap() to use a DMABUF The vfio-pci core mmap() creates a DMABUF with the helper, and the vm_ops fault handler uses the other helper to resolve the fault. Because this depends on DMABUF structs/code, CONFIG_VFIO_PCI_CORE needs to depend on CONFIG_DMA_SHARED_BUFFER. The CONFIG_VFIO_PCI_DMABUF still conditionally enables the export support code. NOTE: The user mmap()s a device fd, but the resulting VMA's vm_file becomes that of the DMABUF which takes ownership of the device and puts it on release. This maintains the existing behaviour of a VMA keeping the VFIO device open. BAR zapping then happens via the existing vfio_pci_dma_buf_move() path, which now needs to unmap PTEs in the DMABUF's address_space. vfio/pci: Provide a user-facing name for BAR mappings There was a request for decent debug naming in /proc/<pid>/maps etc. comparable to the existing VFIO names: since the VMAs are DMABUFs, they have a "dmabuf:" prefix and can't be 100% identical to before. This is a user-visible change, but this patch at least now gives us extra info on the BDF & BAR being mapped. vfio/pci: Clean up BAR zap and revocation In general (see NOTE!) the vfio_pci_zap_bars() is now obsolete, since it unmaps PTEs in the VFIO device address_space which is now unused. This consolidates all calls (e.g. around reset) with the neighbouring vfio_pci_dma_buf_move()s into new functions, to revoke-zap/unrevoke. !!! NOTE: the nvgrace-gpu driver continues to use its own private vm_ops, fault handler, etc. for its special memregions, and these DO still add PTEs to the VFIO device address_space. So, a temporary flag, vdev->bar_needs_zap, maintains the old behaviour for this use. At least this patch's consolidation makes it easy to remove the remaining zap when this need goes away; a FIXME reminds that this can be removed when nvgrace-gpu is converted. vfio/pci: Support mmap() of a VFIO DMABUF Adds mmap() for a DMABUF fd exported from vfio-pci. It was a goal to keep the VFIO device fd lifetime behaviour unchanged with respect to the DMABUFs. An application can close all device fds, and this will revoke/clean up all DMABUFs; no mappings or other access can be performed now. When enabling mmap() of the DMABUFs, this means access through the VMA is also revoked. This complicates the fault handler because whilst the DMABUF exists, it has no guarantee that the corresponding VFIO device is still alive. Adds synchronisation ensuring the vdev is available before vdev->memory_lock is touched; this holds the device registration so that even if the buffer has been cleaned up, vdev hasn't been freed and so the lock can be safely taken. (I decided against the alternative of preventing cleanup by holding the VFIO device open if any DMABUFs exist, because it's both a change of behaviour and less clean overall.) I've added a chonky comment in place, happy to clarify more if you have ideas. This commit makes VFIO_PCI_CORE depend on PCI_P2PDMA_CORE (commit 1) to bring in (only) the P2PDMA provider code. vfio/pci: Permanently revoke a DMABUF on request By weight, this is mostly a rename of revoked to an enum, status. There are now 3 states for a buffer, usable and revoked temporary/permanent. A new VFIO device ioctl is added, VFIO_DEVICE_PCI_DMABUF_REVOKE, which passes a DMABUF (exported from that device) and permanently revokes it. Thus a userspace driver can guarantee any downstream consumers of a shared fd are prevented from accessing a BAR range, and that range can be reused. The code doing revocation in vfio_pci_dma_buf_move() is moved, unchanged, to a common function for use by _move() and the new ioctl path. Q: I can't think of a good reason to temporarily revoke/unrevoke buffers from userspace, so didn't add a 'flags' field to the ioctl struct. Easy to add if people think it's worthwhile for future use. vfio/pci: Add mmap() attributes to DMABUF feature Adds a new VFIO feature, VFIO_DEVICE_FEATURE_DMA_BUF_MEMATTR. After a DMABUF is exported, this feature ioctl() isused to set a memory attribute that will be used by future mmap()s of the DMABUF fd (i.e. it does nothing for any existing maps). The default is UC, and via the feature one can specify CPU access as WC. The attribute is an enum/scalar rather than bitmap/cumulative. The attributes follow a "try-fail" model where a client can request an attribute and either succeed or fail with ENOTSUPP if it's unknown; if future attributes are platform-specific then their support can be probed. (Since it's just UC/WC for now, there is no reservation or numeric structure to the namespace yet, but we could support system/arch-specific values in future by carving out base + arch-specific + IMPDEF ranges.) Testing ======= (The [RFC ONLY] userspace test program, for QEMU edu-plus, has been dropped from the series, but can be found in the GitHub branch below. It at least illustrates the export, map, revoke, attribute, and close semantics interoperate.) This code has been tested in mapping DMABUFs of single/multiple ranges, aliasing mmap()s, aliasing ranges across DMABUFs, vm_pgoff > 0, revocation, shutdown/cleanup scenarios, and hugepage mappings seem to work correctly. I've lightly tested WC mappings also (by observing resulting PTEs as having the correct attributes...). No regressions observed on the VFIO selftests, or on our internal vfio-pci applications. End === This is based on VFIO next (e.g. at b9285405c5f6). These commits are on GitHub for easier browsing, along with "[RFC ONLY] selftests: vfio: Add standalone vfio_dmabuf_mmap_test": https://github.com/metamev/linux/compare/b9285405c5f6...metamev:linux:dev/m… Thanks for reading, Matt ================================================================================ Change log: v2: - Rebase on VFIO next, picking up Alex's vfio_pci_dma_buf_move()/vfio_pci_dma_buf_cleanup() fixes, and dropping "vfio/pci: Fix vfio_pci_dma_buf_cleanup() double-put" - Added "PCI/P2PDMA: Add CONFIG_PCI_P2PDMA_CORE" so that the newly-added vfio-pci hard dependency on the P2PDMA provider instead pulls in the _CORE variant and not the full-fat CONFIG_PCI_P2PDMA. This means that the core of vfio-pci does not need ZONE_DEVICE, but if it's available then enabling P2PDMA in turn enables DMABUF export. Fixes basic VFIO operation on 32b or other platforms without ZONE_DEVICE. - Fixed comment inaccuracy in vfio_pci_dma_buf_revoke() and cleaned up vdev validity test. - vfio_pci_dma_buf_find_pfn(): use PAGE_ALIGN(), better span variable naming, OVF check - Made vm_pgoffs use consistent (keeping the resource index at the top and masking where offset is used). For BAR mmap, use new vma_pgoff_adjust to create the DMABUF with the exact mmap()ed span instead of from the start of the BAR with an invisible portion before the mapping. - Added VFIO_DEVICE_FEATURE_DMA_BUF_MEMATTR to set memory attributes, instead of using the export `flags` field. - vfio_pci_ioctl_reset: Moved vfio_pci_zap_revoke_bars() (effectively, vfio_pci_dma_buf_move()) back after D0 transition. Note, if a BAR zap is needed, it's done in this function so now happens after this D0 transition with the _move; it was done before it at the time of the memory_lock taking. - Minimised vfio_pci_dma_buf_mmap() (removed redundant span check), added READ_ONCE for memattr - Misc fixes: comment in DMABUF name generation, removed superfluous READ_ONCE from faulthandler v1: https://lore.kernel.org/kvm/20260416131815.2729131-1-mattev@meta.com/ - Cleanup of the common DMABUF-aware VMA vm_ops fault handler and export code. - Fixed a lot of races, particularly faults racing with DMABUF cleanup (if the VFIO device fds close, for example). - Added nicer human-readable names for VFIO mmap() VMAs RFCv2: Respin based on the feedback/suggestions: https://lore.kernel.org/kvm/20260312184613.3710705-1-mattev@meta.com/ - Transform the existing VFIO BAR mmap path to also use DMABUFs behind the scenes, and then simply share that code for explicitly-mapped DMABUFs. Jason wanted to go that direction to enable iommufd VFIO type 1 emulation to pick up a DMABUF for an IO mapping. - Revoke buffers using a VFIO device fd ioctl RFCv1: https://lore.kernel.org/all/20260226202211.929005-1-mattev@meta.com/ Matt Evans (9): PCI/P2PDMA: Add CONFIG_PCI_P2PDMA_CORE vfio/pci: Add a helper to look up PFNs for DMABUFs vfio/pci: Add a helper to create a DMABUF for a BAR-map VMA vfio/pci: Convert BAR mmap() to use a DMABUF vfio/pci: Provide a user-facing name for BAR mappings vfio/pci: Clean up BAR zap and revocation vfio/pci: Support mmap() of a VFIO DMABUF vfio/pci: Permanently revoke a DMABUF on request vfio/pci: Add mmap() attributes to DMABUF feature drivers/pci/Kconfig | 10 +- drivers/pci/Makefile | 2 +- drivers/pci/p2pdma.c | 16 + drivers/vfio/pci/Kconfig | 4 +- drivers/vfio/pci/Makefile | 3 +- drivers/vfio/pci/nvgrace-gpu/main.c | 5 + drivers/vfio/pci/vfio_pci_config.c | 30 +- drivers/vfio/pci/vfio_pci_core.c | 225 +++++++++--- drivers/vfio/pci/vfio_pci_dmabuf.c | 548 ++++++++++++++++++++++++---- drivers/vfio/pci/vfio_pci_priv.h | 57 ++- include/linux/pci-p2pdma.h | 24 +- include/linux/pci.h | 2 +- include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 57 +++ 14 files changed, 815 insertions(+), 169 deletions(-) -- 2.47.3

17 hours, 7 minutes

[PATCH v4] dma-buf: Fix silent overflow for phys vec to sgt

by David Hu

In case MMIO size is bigger than 4G and peer2peer DMA goes through host bridge, we trigger a code path that assigns the total linked IOVA (which is greater than 4G) to mapped_len. Previously, `mapped_len` was declared as 32-bit `unsigned int`. When accumulating `size_t` lengths, this leads to a silent wrap-around. This truncation causes truncated lengths to be passed to functions like `fill_sg_entry()`. Fix this by changing `mapped_len` to `size_t` (64-bit). While at it, fix similar potential overflow issues in `calc_sg_nents` by using `size_t` for `nents` and checking against `UINT_MAX` and using `unsigned int` for the loop iterator in `fill_sg_entry` to match. Fixes: 3aa31a8bb11e ("dma-buf: provide phys_vec to scatter-gather mapping routine") Cc: stable(a)vger.kernel.org Cc: iommu(a)lists.linux.dev Reviewed-by: Pranjal Shrivastava <praan(a)google.com> Signed-off-by: David Hu <xuehaohu(a)google.com> --- Changes in v4: - Added WARN_ON_ONCE() to the nents overflow check to prevent silent failures (Claude Bot). Changes in v3: - Removed leftover sentence fragment from the commit message. - Kept `nents = 0` initialization (previously stated as removed in the v2 changelog) as it is strictly required for the `+=` accumulation loop in `calc_sg_nents()`. Changes in v2: - Fixed 'IVOA' -> 'IOVA' typo and expanded commit message (Claude Bot). - Added Reverse Xmas tree formatting (Pranjal). - Folded in extra bounds checking for calc_sg_nents() (Pranjal). - Folded in type consistency fix for fill_sg_entry() (Pranjal). drivers/dma-buf/dma-buf-mapping.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/drivers/dma-buf/dma-buf-mapping.c b/drivers/dma-buf/dma-buf-mapping.c index 794acff2546a..1aabc0ee70bb 100644 --- a/drivers/dma-buf/dma-buf-mapping.c +++ b/drivers/dma-buf/dma-buf-mapping.c @@ -10,7 +10,7 @@ static struct scatterlist *fill_sg_entry(struct scatterlist *sgl, size_t length, dma_addr_t addr) { unsigned int len, nents; - int i; + unsigned int i; nents = DIV_ROUND_UP(length, UINT_MAX); for (i = 0; i < nents; i++) { @@ -36,7 +36,7 @@ static unsigned int calc_sg_nents(struct dma_iova_state *state, struct phys_vec *phys_vec, size_t nr_ranges, size_t size) { - unsigned int nents = 0; + size_t nents = 0; size_t i; if (!state || !dma_use_iova(state)) { @@ -51,6 +51,9 @@ static unsigned int calc_sg_nents(struct dma_iova_state *state, nents = DIV_ROUND_UP(size, UINT_MAX); } + if (WARN_ON_ONCE(nents > UINT_MAX)) + return 0; + return nents; } @@ -95,9 +98,10 @@ struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach, size_t nr_ranges, size_t size, enum dma_data_direction dir) { - unsigned int nents, mapped_len = 0; struct dma_buf_dma *dma; struct scatterlist *sgl; + size_t mapped_len = 0; + unsigned int nents; dma_addr_t addr; size_t i; int ret; -- 2.54.0.929.g9b7fa37559-goog

21 hours, 4 minutes

[PATCH v3] dma-buf: Fix silent overflow for phys vec to sgt

by David Hu

In case MMIO size is bigger than 4G and peer2peer DMA goes through host bridge, we trigger a code path that assigns the total linked IOVA (which is greater than 4G) to mapped_len. Previously, `mapped_len` was declared as 32-bit `unsigned int`. When accumulating `size_t` lengths, this leads to a silent wrap-around. This truncation causes truncated lengths to be passed to functions like `fill_sg_entry()`. Fix this by changing `mapped_len` to `size_t` (64-bit). While at it, fix similar potential overflow issues in `calc_sg_nents` by using `size_t` for `nents` and checking against `UINT_MAX` and using `unsigned int` for the loop iterator in `fill_sg_entry` to match. Fixes: 3aa31a8bb11e ("dma-buf: provide phys_vec to scatter-gather mapping routine") Cc: stable(a)vger.kernel.org Cc: iommu(a)lists.linux.dev Reviewed-by: Pranjal Shrivastava <praan(a)google.com> Signed-off-by: David Hu <xuehaohu(a)google.com> --- Changes in v3: - Removed leftover sentence fragment from the commit message. - Kept `nents = 0` initialization (previously stated as removed in the v2 changelog) as it is strictly required for the `+=` accumulation loop in `calc_sg_nents()`. Changes in v2: - Fixed 'IVOA' -> 'IOVA' typo and expanded commit message (Claude Bot). - Added Reverse Xmas tree formatting (Pranjal). - Folded in extra bounds checking for calc_sg_nents() (Pranjal). - Folded in type consistency fix for fill_sg_entry() (Pranjal). drivers/dma-buf/dma-buf-mapping.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/drivers/dma-buf/dma-buf-mapping.c b/drivers/dma-buf/dma-buf-mapping.c index 794acff2546a..5bc769fc42ea 100644 --- a/drivers/dma-buf/dma-buf-mapping.c +++ b/drivers/dma-buf/dma-buf-mapping.c @@ -10,7 +10,7 @@ static struct scatterlist *fill_sg_entry(struct scatterlist *sgl, size_t length, dma_addr_t addr) { unsigned int len, nents; - int i; + unsigned int i; nents = DIV_ROUND_UP(length, UINT_MAX); for (i = 0; i < nents; i++) { @@ -36,7 +36,7 @@ static unsigned int calc_sg_nents(struct dma_iova_state *state, struct phys_vec *phys_vec, size_t nr_ranges, size_t size) { - unsigned int nents = 0; + size_t nents = 0; size_t i; if (!state || !dma_use_iova(state)) { @@ -51,6 +51,9 @@ static unsigned int calc_sg_nents(struct dma_iova_state *state, nents = DIV_ROUND_UP(size, UINT_MAX); } + if (nents > UINT_MAX) + return 0; + return nents; } @@ -95,9 +98,10 @@ struct sg_table *dma_buf_phys_vec_to_sgt(struct dma_buf_attachment *attach, size_t nr_ranges, size_t size, enum dma_data_direction dir) { - unsigned int nents, mapped_len = 0; struct dma_buf_dma *dma; struct scatterlist *sgl; + size_t mapped_len = 0; + unsigned int nents; dma_addr_t addr; size_t i; int ret; -- 2.54.0.794.g4f17f83d09-goog

1 day

[PATCH] dma-buf: fix UAF in dma_buf_fd() tracepoint

by David Carlier

Once FD_ADD() returns, the fd is live in the file descriptor table and a thread sharing that table can close() it before DMA_BUF_TRACE() runs. The close drops the last reference, __fput() frees the dma_buf, and the tracepoint then dereferences dmabuf to take dmabuf->name_lock -- slab-use-after-free. Split FD_ADD() back into get_unused_fd_flags() + fd_install() and emit the tracepoint between them. While the fdtable slot is reserved with a NULL file pointer, a racing close() returns -EBADF without entering __fput(), so the dma_buf stays alive across the trace. Same approach as commit 2d76319c4cbb ("dma-buf: fix UAF in dma_buf_put() tracepoint"). This undoes the FD_ADD() conversion done in commit 34dfce523c90 ("dma: convert dma_buf_fd() to FD_ADD()"); FD_ADD() has no place to hook the tracepoint safely. Reported-by: syzbot+7f4987d0afb97dd090cb(a)syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=7f4987d0afb97dd090cb Fixes: 281a22631423 ("dma-buf: add some tracepoints to debug.") Cc: stable(a)vger.kernel.org # 7.0.x Signed-off-by: David Carlier <devnexen(a)gmail.com> --- drivers/dma-buf/dma-buf.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c index 71f37544a5c6..d504c636dc29 100644 --- a/drivers/dma-buf/dma-buf.c +++ b/drivers/dma-buf/dma-buf.c @@ -792,9 +792,13 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags) if (!dmabuf || !dmabuf->file) return -EINVAL; - fd = FD_ADD(flags, dmabuf->file); + fd = get_unused_fd_flags(flags); + if (fd < 0) + return fd; + DMA_BUF_TRACE(trace_dma_buf_fd, dmabuf, fd); + fd_install(fd, dmabuf->file); return fd; } EXPORT_SYMBOL_NS_GPL(dma_buf_fd, "DMA_BUF"); -- 2.53.0

1 day, 1 hour

Re: [PATCH v2] drm/prime: fix dangling dmabuf entries after handle release

by Christian König

On 5/28/26 15:29, w15303746062(a)163.com wrote: > From: Mingyu Wang <25181214217(a)stu.xidian.edu.cn> > > When a GEM handle already exists in the drm_prime_file_private, repeated > calls to DRM_IOCTL_PRIME_HANDLE_TO_FD can cause drm_prime_add_buf_handle() > to insert multiple entries with the same handle into the handles rb_tree. > Because the insertion walk moves left on equality, duplicate keys are > structurally accepted by the tree. That should never happen and would be a major bug. All callers should check if a handler exists before calling drm_prime_add_buf_handle(). How do you see that a handle is added twice? Regards, Christian. > > Later, when the handle is released via drm_gem_release() -> > drm_gem_object_release_handle() -> drm_prime_remove_buf_handle(), the > latter iterates the handles tree, removes the first matching node, and > breaks out of the loop. Any remaining duplicate nodes that share the > same handle are left orphaned in the dmabufs tree - they are no longer > reachable through the handles tree and are never freed. > > When the drm file is finally closed, drm_prime_destroy_file_private() > triggers: > > WARN_ON(!RB_EMPTY_ROOT(&prime_fpriv->dmabufs)); > > because the dmabufs tree is still non-empty. With CONFIG_PANIC_ON_WARN > this becomes a kernel panic: > > ------------[ cut here ]------------ > WARNING: CPU: 0 PID: 19739 at drivers/gpu/drm/drm_prime.c:223 drm_prime_destroy_file_private+0x43/0x60 > ... > Kernel panic - not syncing: kernel: panic_on_warn set ... > > Fix this by restarting the lookup from the root of the handles tree > after each successful removal, so that all duplicate nodes for the given > handle are erased. The caller (drm_gem_object_release_handle) already > holds prime_fpriv->lock, so this does not change the locking strategy. > > Signed-off-by: Mingyu Wang <25181214217(a)stu.xidian.edu.cn> > --- > Changes in v2: > - Drop the unnecessary mutex_lock addition, as the caller (drm_gem_object_release_handle) already holds the lock. > - Rewrite the commit message to accurately reflect the root cause (duplicate handle insertions) rather than an assumed lack of synchronization. > - Restart the rb_tree lookup from the root instead of breaking the loop to ensure all orphaned duplicate nodes are thoroughly removed. > > drivers/gpu/drm/drm_prime.c | 5 ++++- > 1 file changed, 4 insertions(+), 1 deletion(-) > > diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c > index 9b44c78cd77f..dc28df1c6698 100644 > --- a/drivers/gpu/drm/drm_prime.c > +++ b/drivers/gpu/drm/drm_prime.c > @@ -202,7 +202,10 @@ void drm_prime_remove_buf_handle(struct drm_prime_file_private *prime_fpriv, > > dma_buf_put(member->dma_buf); > kfree(member); > - break; > + /* Duplicate handles may exist; restart search from root > + * to guarantee removal of all matching entries. > + */ > + rb = prime_fpriv->handles.rb_node; > } else if (member->handle < handle) { > rb = rb->rb_right; > } else { > -- > 2.34.1 >

1 day, 2 hours

Whatsapp: +33 754.090.961, How get location in Abou Dhabi, UAE read full story .

by brekkea78＠gmail.com

Whatsapp: +33 754.090.961, How i buy Dispensary in Dubai, read full story . Order the free ebook on Whatsapp: +33 754.090.961 of how i buy 70mg lyrica in #jeddah. For those wondering where to buy #THC vapes in UAE, it’s important to find a provider that values purity, potency, and customer confidentiality go on: https://uaetherapist.com/ the online drop. I like spent my holiday In United Arab Emirates , the streets are amazing most in the night one day inside a uber the driver who take me from #Al_wahda ask me if he can play music i told him ‘’ YOU ARE WELCOME BRO ’’ then i heard 50cent in window shopper i get a small smile in the corner of my face. At that moment i don’t know why, suddenly i started to think about Coffee Sh0p how i can roll a split , but DAMMN MAN YOU ARE CLEAN OVER 3 YEARS!! You are in Dubai, is like Bahrain, as Saudi , like Oman it’s impossible to get a coffee shop here anyway you risk to go through huge problem with arabic country law !! I was saw my pregnant girlfiend who are waiting me at Charjah, my new work who permit me to travel as i want with my family all the life i have for , no man stay focus on your job in MANAMA ... FOLLOW HAPPENED STORY ON Whatsapp: +33 754.090.961 and you will know where i bought THC gummies in Dubai. Contacts: Telegram: Go0dTherapist Gmail: uaetherapist(a)gmail.com Signal: addsilkroad.47 Whatsapp: https://whatsapp.com/channel/0029VbCcd6PLCoX7L05OeX3a Tik Tok: https://vm.tiktok.com/ZS9R3Cn5sdvLu-YwLbv/ https://uaetherapist.com/how-to-order/ https://uaetherapist.com/shop/ https://uaetherapist.com/faqs/ https://uaetherapist.com/policies/ https://uaetherapist.com/contact/ #uaetherapist.com #uaetherapistdotcom #dubaidispensary #uaemedicated

1 day, 6 hours

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Linaro-mm-sig