The Graphics Execution Manager Part of the Direct Rendering Manager ============================== Keith Packard Eric Anholt 2008-5-9 Contents: 1. GEM Overview 2. API overview and conventions 3. Object Creation/Destruction 4. Reading/writing contents 5. Mapping objects to userspace 6. Memory Domains 7. Execution (Intel specific) 8. Other misc Intel-specific functions 1. Graphics Execution Manager Overview Gem is designed to manage graphics memory, control access to the graphics device execution context and handle the essentially NUMA environment unique to modern graphics hardware. Gem allows multiple applications to share graphics device resources without the need to constantly reload the entire graphics card. Data may be shared between multiple applications with gem ensuring that the correct memory synchronization occurs. Graphics data can consume arbitrary amounts of memory, with 3D applications constructing ever larger sets of textures and vertices. With graphics cards memory space growing larger every year, and graphics APIs growing more complex, we can no longer insist that each application save a complete copy of their graphics state so that the card can be re-initialized from user space at each context switch. Ensuring that graphics data remains persistent across context switches allows applications significant new functionality while also improving performance for existing APIs. Modern linux desktops include significant 3D rendering as a fundemental component of the desktop image construction process. 2D and 3D applications paint their content to offscreen storage and the central 'compositing manager' constructs the final screen image from those window contents. This means that pixel image data from these applications must move within reach of the compositing manager and used as source operands for screen image rendering operations. Gem provides simple mechanisms to manage graphics data and control execution flow within the linux operating system. Using many existing kernel subsystems, it does this with a modest amount of code. 2. API Overview and Conventions All APIs here are defined in terms of ioctls appplied to the DRM file descriptor. To create and manipulate objects, an application must be 'authorized' using the DRI or DRI2 protocols with the X server. To relax that, we will need to implement some better access control mechanisms within the hardware portion of the driver to prevent inappropriate cross-application data access. Any DRM driver which does not support GEM will return -ENODEV for all of these ioctls. Invalid object handles return -EINVAL. Invalid object names return -ENOENT. Other errors are as documented in the specific API below. To avoid the need to translate ioctl contents on mixed-size systems (with 32-bit user space running on a 64-bit kernel), the ioctl data structures contain explicitly sized objects, using 64-bits for all size and pointer data and 32-bits for identifiers. In addition, the 64-bit objects are all carefully aligned on 64-bit boundaries. Because of this, all pointers in the ioctl data structures are passed as uint64_t values. Suitable casts will be necessary. One significant operation which is explicitly left out of this API is object locking. Applications are expected to perform locking of shared objects outside of the GEM api. This kind of locking is not necessary to safely manipulate the graphics engine, and with multiple objects interacting in unknown ways, per-object locking would likely introduce all kinds of lock-order issues. Punting this to the application seems like the only sensible plan. Given that DRM already offers a global lock on the hardware, this doesn't change the current situation. 3. Object Creation and Destruction Gem provides explicit memory management primitives. System pages are allocated when the object is created, either as the fundemental storage for hardware where system memory is used by the graphics processor directly, or as backing store for graphics-processor resident memory. Objects are referenced from user space using handles. These are, for all intents and purposes, equivalent to file descriptors. We could simply use file descriptors were it not for the small limit (1024) of file descriptors available to applications, and for the fact that the X server (a rather significant user of this API) uses 'select' and has a limited maximum file descriptor for that operation. Given the ability to allocate more file descriptors, and given the ability to place these 'higher' in the file descriptor space, we'd love to simply use file descriptors. Objects may be published with a name so that other applications can access them. The name remains valid as long as the object exists. Right now, our DRI APIs use 32-bit integer names, so that's what we expose here A. Creation struct drm_gem_create { /** * Requested size for the object. * * The (page-aligned) allocated size for the object * will be returned. */ uint64_t size; /** * Returned handle for the object. * * Object handles are nonzero. */ uint32_t handle; uint32_t pad; }; /* usage */ create.size = 16384; ret = ioctl (fd, DRM_IOCTL_GEM_CREATE, &create); if (ret == 0) return create.handle; Note that the size is rounded up to a page boundary, and that the rounded-up size is returned in 'size'. No name is assigned to this object, making it local to this process. If insufficient memory is availabe, -ENOMEM will be returned. B. Closing struct drm_gem_close { /** Handle of the object to be closed. */ uint32_t handle; uint32_t pad; }; /* usage */ close.handle = ; ret = ioctl (fd, DRM_IOCTL_GEM_CLOSE, &close); This call makes the specified handle invalid, and if no other applications are using the object, any necessary graphics hardware synchronization is performed and the resources used by the object released. C. Naming struct drm_gem_flink { /** Handle for the object being named */ uint32_t handle; /** Returned global name */ uint32_t name; }; /* usage */ flink.handle = ; ret = ioctl (fd, DRM_IOCTL_GEM_FLINK, &flink); if (ret == 0) return flink.name; Flink creates a name for the object and returns it to the application. This name can be used by other applications to gain access to the same object. D. Opening by name struct drm_gem_open { /** Name of object being opened */ uint32_t name; /** Returned handle for the object */ uint32_t handle; /** Returned size of the object */ uint64_t size; }; /* usage */ open.name = ; ret = ioctl (fd, DRM_IOCTL_GEM_OPEN, &open); if (ret == 0) { *sizep = open.size; return open.handle; } Open accesses an existing object and returns a handle for it. If the object doesn't exist, -ENOENT is returned. The size of the object is also returned. This handle has all the same capabilities as the handle used to create the object. In particular, the object is not destroyed until all handles are closed. 4. Basic read/write operations By default, gem objects are not mapped to the applications address space, getting data in and out of them is done with I/O operations instead. This allows the data to reside in otherwise unmapped pages, including pages in video memory on an attached discrete graphics card. In addition, using explicit I/O operations allows better control over cache contents, as graphics devices are generally not cache coherent with the CPU, mapping pages used for graphics into an application address space requires the use of expensive cache flushing operations. Providing direct control over graphics data access ensures that data are handled in the most efficient possible fashion. A. Reading struct drm_gem_pread { /** Handle for the object being read. */ uint32_t handle; uint32_t pad; /** Offset into the object to read from */ uint64_t offset; /** Length of data to read */ uint64_t size; /** Pointer to write the data into. */ uint64_t data_ptr; /* void * */ }; This copies data into the specified object at the specified position. Any necessary graphics device synchronization and flushing will be done automatically. struct drm_gem_pwrite { /** Handle for the object being written to. */ uint32_t handle; uint32_t pad; /** Offset into the object to write to */ uint64_t offset; /** Length of data to write */ uint64_t size; /** Pointer to read the data from. */ uint64_t data_ptr; /* void * */ }; This copies data out of the specified object into the waiting user memory. Again, device synchronization will be handled by the kernel to ensure user space sees a consistent view of the graphics device. 5. Mapping objects to user space For most objects, reading/writing is the preferred interaction mode. However, when the CPU is involved in rendering to cover deficiencies in hardware support for particular operations, the CPU will want to directly access the relevant objects. Because mmap is fairly heavyweight, we allow applications to retain maps to objects persistently and then update how they're using the memory through a separate interface. Applications which fail to use this separate interface may exhibit unpredictable behaviour as memory consistency will not be preserved. A. Mapping struct drm_gem_mmap { /** Handle for the object being mapped. */ uint32_t handle; uint32_t pad; /** Offset in the object to map. */ uint64_t offset; /** * Length of data to map. * * The value will be page-aligned. */ uint64_t size; /** Returned pointer the data was mapped at */ uint64_t addr_ptr; /* void * */ }; /* usage */ mmap.handle = ; mmap.offset = ; mmap.size = ; ret = ioctl (fd, DRM_IOCTL_GEM_MMAP, &mmap); if (ret == 0) return (void *) (uintptr_t) mmap.addr_ptr; B. Unmapping munmap (addr, length); Nothing strange here, just use the normal munmap syscall. 6. Memory Domains Graphics devices remain a strong bastion of non cache-coherent memory. As a result, accessing data through one functional unit will end up loading that cache with data which then needs to be manually synchronized when that data is used with another functional unit. Tracking where data are resident is done by identifying how functional units deal with caches. Each cache is labeled as a separate memory domain. Then, each sequence of operations is expected to load data into various read domains and leave data in at most one write domain. Gem tracks the read and write memory domains of each object and performs the necessary synchronization operations when objects move from one domain set to another. For example, if operation 'A' constructs an image that is immediately used by operation 'B', then when the read domain for 'B' is not the same as the write domain for 'A', then the write domain must be flushed, and the read domain invalidated. If these two operations are both executed in the same command queue, then the flush operation can go inbetween them in the same queue, avoiding any kind of CPU-based synchronization and leaving the GPU to do the work itself. 6.1 Memory Domains (GPU-independent) * DRM_GEM_DOMAIN_CPU. Objects in this domain are using caches which are connected to the CPU. Moving objects from non-CPU domains into the CPU domain can involve waiting for the GPU to finish with operations using this object. Moving objects from this domain to a GPU domain can involve flushing CPU caches and chipset buffers. 6.1 GPU-independent memory domain ioctl This ioctl is independent of the GPU in use. So far, no use other than synchronizing objects to the CPU domain have been found; if that turns out to be generally true, this ioctl may be simplified further. A. Explicit domain control struct drm_gem_set_domain { /** Handle for the object */ uint32_t handle; /** New read domains */ uint32_t read_domains; /** New write domain */ uint32_t write_domain; }; /* usage */ set_domain.handle = ; set_domain.read_domains = ; set_domain.write_domain = ; ret = ioctl (fd, DRM_IOCTL_GEM_SET_DOMAIN, &set_domain); When the application wants to explicitly manage memory domains for an object, it can use this function. Usually, this is only used when the application wants to synchronize object contents between the GPU and CPU-based application rendering. In that case, the would be set to DRM_GEM_DOMAIN_CPU, and if the application were going to write to the object, the would also be set to DRM_GEM_DOMAIN_CPU. After the call, gem guarantees that all previous rendering operations involving this object are complete. The application is then free to access the object through the address returned by the mmap call. Afterwards, when the application again uses the object through the GPU, any necessary CPU flushing will occur and the object will be correctly synchronized with the GPU. Note that this synchronization is not required for any accesses going through the driver itself. The pread, pwrite and execbuffer ioctls all perform the necessary domain management internally. Explicit synchronization is only necessary when accessing the object through the mmap'd address. 7. Execution (Intel specific) Managing the command buffers is inherently chip-specific, so the core of gem doesn't have any intrinsic functions. Rather, execution is left to the device-specific portions of the driver. The Intel DRM_I915_GEM_EXECBUFFER ioctl takes a list of gem objects, all of which are mapped to the graphics device. The last object in the list is the command buffer. 7.1. Relocations Command buffers often refer to other objects, and to allow the kernel driver to move objects around, a sequence of relocations is associated with each object. Device-specific relocation operations are used to place the target-object relative value into the object. The Intel driver has a single relocation type: struct drm_i915_gem_relocation_entry { /** * Handle of the buffer being pointed to by this * relocation entry. * * It's appealing to make this be an index into the * mm_validate_entry list to refer to the buffer, * but this allows the driver to create a relocation * list for state buffers and not re-write it per * exec using the buffer. */ uint32_t target_handle; /** * Value to be added to the offset of the target * buffer to make up the relocation entry. */ uint32_t delta; /** * Offset in the buffer the relocation entry will be * written into */ uint64_t offset; /** * Offset value of the target buffer that the * relocation entry was last written as. * * If the buffer has the same offset as last time, we * can skip syncing and writing the relocation. This * value is written back out by the execbuffer ioctl * when the relocation is written. */ uint64_t presumed_offset; /** * Target memory domains read by this operation. */ uint32_t read_domains; /* * Target memory domains written by this operation. * * Note that only one domain may be written by the * whole execbuffer operation, so that where there are * conflicts, the application will get -EINVAL back. */ uint32_t write_domain; }; 'target_handle', the handle to the target object. This object must be one of the objects listed in the execbuffer request or bad things will happen. The kernel doesn't check for this. 'offset' is where, in the source object, the relocation data are written. Each relocation value is a 32-bit value consisting of the location of the target object in the GPU memory space plus the 'delta' value included in the relocation. 'presumed_offset' is where user-space believes the target object lies in GPU memory space. If this value matches where the object actually is, then no relocation data are written, the kernel assumes that user space has set up data in the source object using this presumption. This offers a fairly important optimization as writing relocation data requires mapping of the source object into the kernel memory space. 'read_domains' and 'write_domains' list the usage by the source object of the target object. The kernel unions all of the domain information from all relocations in the execbuffer request. No more than one write_domain is allowed, otherwise an EINVAL error is returned. read_domains must contain write_domain. This domain information is used to synchronize buffer contents as described above in the section on domains. 7.1.1 Memory Domains (Intel specific) The Intel GPU has several internal caches which are not coherent and hence require explicit synchronization. Memory domains provide the necessary data to synchronize what is needed while leaving other cache contents intact. * DRM_GEM_DOMAIN_I915_RENDER. The GPU 3D and 2D rendering operations use a unified rendering cache, so operations doing 3D painting and 2D blts will use this domain * DRM_GEM_DOMAIN_I915_SAMPLER Textures are loaded by the sampler through a separate cache, so any texture reading will use this domain. Note that the sampler and renderer use different caches, so moving an object from render target to texture source will require a domain transfer. * DRM_GEM_DOMAIN_I915_COMMAND The command buffer doesn't have an explicit cache (although it does read ahead quite a bit), so this domain just indicates that the object needs to be flushed to the GPU. * DRM_GEM_DOMAIN_I915_INSTRUCTION All of the programs on Gen4 and later chips use an instruction cache to speed program execution. It must be explicitly flushed when new programs are written to memory by the CPU. * DRM_GEM_DOMAIN_I915_VERTEX Vertex data uses two different vertex caches, but they're both flushed with the same instruction. 7.2 Execution object list (Intel specific) struct drm_i915_gem_exec_object { /** * User's handle for a buffer to be bound into the GTT * for this operation. */ uint32_t handle; /** * List of relocations to be performed on this buffer */ uint32_t relocation_count; /* struct drm_i915_gem_relocation_entry *relocs */ uint64_t relocs_ptr; /** * Required alignment in graphics aperture */ uint64_t alignment; /** * Returned value of the updated offset of the object, * for future presumed_offset writes. */ uint64_t offset; }; Each object involved in a particular execution operation must be listed using one of these structures. 'handle' references the object. 'relocs_ptr' is a user-mode pointer to a array of 'relocation_count' drm_i915_gem_relocation_entry structs (see above) that define the relocations necessary in this buffer. Note that all relocations must reference other exec_object structures in the same execbuffer ioctl and that those other buffers must come earlier in the exec_object array. In other words, the dependencies mapped by the exec_object relocations must form a directed acyclic graph. 'alignment' is the byte alignment necessary for this buffer. Each object has specific alignment requirements, as the kernel doesn't know what each object is being used for, those requirements must be provided by user mode. If an object is used in two different ways, it's quite possible that the alignment requirements will differ. 'offset' is a return value, receiving the location of the object during this execbuffer operation. The application should use this as the presumed offset in future operations; if the object does not move, then kernel need not write relocation data. 7.3 Execbuffer ioctl (Intel specific) struct drm_i915_gem_execbuffer { /** * List of buffers to be validated with their * relocations to be performend on them. * * These buffers must be listed in an order such that * all relocations a buffer is performing refer to * buffers that have already appeared in the validate * list. */ /* struct drm_i915_gem_validate_entry *buffers */ uint64_t buffers_ptr; uint32_t buffer_count; /** * Offset in the batchbuffer to start execution from. */ uint32_t batch_start_offset; /** * Bytes used in batchbuffer from batch_start_offset */ uint32_t batch_len; uint32_t DR1; uint32_t DR4; uint32_t num_cliprects; uint64_t cliprects_ptr; /* struct drm_clip_rect *cliprects */ }; 'buffers_ptr' is a user-mode pointer to an array of 'buffer_count' drm_i915_gem_exec_object structures which contains the complete set of objects required for this execbuffer operation. The last entry in this array, the 'batch buffer', is the buffer of commands which will be linked to the ring and executed. 'batch_start_offset' is the byte offset within the batch buffer which contains the first command to execute. So far, we haven't found a reason to use anything other than '0' here, but the thought was that some space might be allocated for additional initialization which could be skipped in some cases. This must be a multiple of 4. 'batch_len' is the length, in bytes, of the data to be executed (i.e., the amount of data after batch_start_offset). This must be a multiple of 4. 'num_cliprects' and 'cliprects_ptr' reference an array of drm_clip_rect structures that is num_cliprects long. The entire batch buffer will be executed multiple times, once for each rectangle in this list. If num_cliprects is 0, then no clipping rectangle will be set. 'DR1' and 'DR4' are portions of the 3DSTATE_DRAWING_RECTANGLE command which will be queued when this operation is clipped (num_cliprects != 0). DR1 bit definition 31 Fast Scissor Clip Disable (debug only). Disables a hardware optimization that improves performance. This should have no visible effect, other than reducing performance 30 Depth Buffer Coordinate Offset Disable. This disables the addition of the depth buffer offset bits which are used to change the location of the depth buffer relative to the front buffer. 27:26 X Dither Offset. Specifies the X pixel offset to use when accessing the dither table 25:24 Y Dither Offset. Specifies the Y pixel offset to use when accessing the dither table. DR4 bit definition 31:16 Drawing Rectangle Origin Y. Specifies the Y origin of coordinates relative to the draw buffer. 15:0 Drawing Rectangle Origin X. Specifies the X origin of coordinates relative to the draw buffer. As you can see, these two fields are necessary for correctly offsetting drawing within a buffer which contains multiple surfaces. Note that DR1 is only used on Gen3 and earlier hardware and that newer hardware sticks the dither offset elsewhere. 7.3.1 Detailed Execution Description Execution of a single batch buffer requires several preparatory steps to make the objects visible to the graphics engine and resolve relocations to account for their current addresses. A. Mapping and Relocation Each exec_object structure in the array is examined in turn. If the object is not already bound to the GTT, it is assigned a location in the graphics address space. If no space is available in the GTT, some other object will be evicted. This may require waiting for previous execbuffer requests to complete before that object can be unmapped. With the location assigned, the pages for the object are pinned in memory using find_or_create_page and the GTT entries updated to point at the relevant pages using drm_agp_bind_pages. Then the array of relocations is traversed. Each relocation record looks up the target object and, if the presumed offset does not match the current offset (remember that this buffer has already been assigned an address as it must have been mapped earlier), the relocation value is computed using the current offset. If the object is currently in use by the graphics engine, writing the data out must be preceeded by a delay while the object is still busy. Once it is idle, then the page containing the relocation is mapped by the CPU and the updated relocation data written out. The read_domains and write_domain entries in each relocation are used to compute the new read_domains and write_domain values for the target buffers. The actual execution of the domain changes must wait until all of the exec_object entries have been evaluated as the complete set of domain information will not be available until then. B. Memory Domain Resolution After all of the new memory domain data has been pulled out of the relocations and computed for each object, the list of objects is again traversed and the new memory domains compared against the current memory domains. There are two basic operations involved here: * Flushing the current write domain. If the new read domains are not equal to the current write domain, then the current write domain must be flushed. Otherwise, reads will not see data present in the write domain cache. In addition, any new read domains other than the current write domain must be invalidated to ensure that the flushed data are re-read into their caches. * Invaliding new read domains. Any domains which were not currently used for this object must be invalidated as old objects which were mapped at the same location may have stale data in the new domain caches. If the CPU cache is being invalidated and some GPU cache is being flushed, then we'll have to wait for rendering to complete so that any pending GPU writes will be complete before we flush the GPU cache. If the CPU cache is being flushed, then we use 'clflush' to get data written from the CPU. Because the GPU caches cannot be partially flushed or invalidated, we don't actually flush them during this traversal stage. Rather, we gather the invalidate and flush bits up in the device structure. Once all of the object domain changes have been evaluated, then the gathered invalidate and flush bits are examined. For any GPU flush operations, we emit a single MI_FLUSH command that performs all of the necessary flushes. We then look to see if the CPU cache was flushed. If so, we use the chipset flush magic (writing to a special page) to get the data out of the chipset and into memory. C. Queuing Batch Buffer to the Ring With all of the objects resident in graphics memory space, and all of the caches prepared with appropriate data, the batch buffer object can be queued to the ring. If there are clip rectangles, then the buffer is queued once per rectangle, with suitable clipping inserted into the ring just before the batch buffer. D. Creating an IRQ Cookie Right after the batch buffer is placed in the ring, a request to generate an IRQ is added to the ring along with a command to write a marker into memory. When the IRQ fires, the driver can look at the memory location to see where in the ring the GPU has passed. This magic cookie value is stored in each object used in this execbuffer command; it is used whereever you saw 'wait for rendering' above in this document. E. Writing back the new object offsets So that the application has a better idea what to use for 'presumed_offset' values later, the current object offsets are written back to the exec_object structures. 8. Other misc Intel-specific functions. To complete the driver, a few other functions were necessary. 8.1 Initialization from the X server As the X server is currently responsible for apportioning memory between 2D and 3D, it must tell the kernel which region of the GTT aperture is available for 3D objects to be mapped into. struct drm_i915_gem_init { /** * Beginning offset in the GTT to be managed by the * DRM memory manager. */ uint64_t gtt_start; /** * Ending offset in the GTT to be managed by the DRM * memory manager. */ uint64_t gtt_end; }; /* usage */ init.gtt_start = ; init.gtt_end = ; ret = ioctl (fd, DRM_IOCTL_I915_GEM_INIT, &init); The GTT aperture between gtt_start and gtt_end will be used to map objects. This also tells the kernel that the ring can be used, pulling the ring addresses from the device registers. 8.2 Pinning objects in the GTT For scan-out buffers and the current shared depth and back buffers, we need to have them always available in the GTT, at least for now. Pinning means to lock their pages in memory along with keeping them at a fixed offset in the graphics aperture. These operations are available only to root. struct drm_i915_gem_pin { /** Handle of the buffer to be pinned. */ uint32_t handle; uint32_t pad; /** alignment required within the aperture */ uint64_t alignment; /** Returned GTT offset of the buffer. */ uint64_t offset; }; /* usage */ pin.handle = ; pin.alignment = ; ret = ioctl (fd, DRM_IOCTL_I915_GEM_PIN, &pin); if (ret == 0) return pin.offset; Pinning an object ensures that it will not be evicted from the GTT or moved. It will stay resident until destroyed or unpinned. struct drm_i915_gem_unpin { /** Handle of the buffer to be unpinned. */ uint32_t handle; uint32_t pad; }; /* usage */ unpin.handle = ; ret = ioctl (fd, DRM_IOCTL_I915_GEM_UNPIN, &unpin); Unpinning an object makes it possible to evict this object from the GTT. It doesn't ensure that it will be evicted, just that it may.