[SOLVED] Linux Kernel : AXI interconnect, where and how is it handled?

Issue

I'm writing a Linux driver for my company in order to port our hardware to GNU/Linux desktops. I'm not a hardware guy at all and I'm struggling to understand how communication between the kernel and the hardware is made.

We basically have an AXI interconnect on which some IPs are wired (we are using Xilinx boards running PetaLinux).

I've already been able to send requests to the hardware, it works well but I feel like I'm missing something. In the kernel I'm mapping physical addresses to virtual ones thanks to ioremap() and I made my own implementation of read/write like so :

static void __iomem *mailbox;

static ssize_t mailbox_read(struct file *filp, char __user *buf, size_t count, loff_t *f_pos)
{
    ssize_t retval = 0;
    uint32_t mlb_data = 0; 

    if(down_interruptible(&sem))
        return -ERESTARTSYS;

    // *f_pos must be a multiple of 4
    // *f_pos must be in bounds
    // count must be 4 (mailbox supports only reading 4 bytes at a time)
    if(*f_pos % 4 || *f_pos >= MAILBOX_SIZE || count != sizeof(mlb_data)) {
        retval = -EINVAL;
        goto out;
    }

    mlb_data = readl(mailbox + *f_pos);

    if(copy_to_user(buf, &mlb_data, count)) {
        retval = -EFAULT;
        goto out;
    }

    *f_pos += count;
    retval = count;

out:
    up(&sem);
    return retval;
}

int mailbox_init(dev_t device)
{
    mailbox = ioremap_nocache(MLB_BASE_ADDR, MAILBOX_SIZE);

    if(!mailbox) {
        printk(KERN_ERR DRIVER_NAME ": cannot map mailbox, ioremap failed.\n");
        return err;
    }

    return 0;
}

On the user side, I try to read/write like so :

int dev_fd = open("/dev/" DRIVER_NAME, O_RDWR);
if(dev_fd < 0) return whatever;

int data;
pread(dev_fd, &data, sizeof data, 0);

close(dev_fd);

It works great but I do not understand how can it be that simple, where is all the AXI stuff handled ? I thought that was something I'd have to do, but I was surprised to see everything works fine already.

I find it really nice that everything is transparent and all but the thing is I would like to implement some error handling and I have no clue on how to do it.

For example if I try to use devmem (which uses /dev/mem internally) to read from an unsupported address, I get a bus error :

root@petalinux:~# devmem 0xCAFEBABE
Bus error

But if I try to do the same through my own character device, it just hangs. No SIGBUS seems to be received.

I'm not sure I've been very clear so to recap here are my 2 questions :

• Where is handled the AXI stuff in the Linux Kernel ? Could it be a Xilinx driver "overriding" mine ?
• How to handle hardware failure like /dev/mem does by sending a SIGBUS to the userland app ?

Solution

1. Looking at the Xilinx AXI Ethernet Driver (drivers/net/ethernet/xilinx/xilinx_axienet_main.c), setting up the iomem cookie (mailbox) causes readl(), writel(), memcpy_fromio() et cetera to handle the access properly.

   The details of how this is done at the hardware level, depends on the hardware architecture. For example, mach-ipx4xx uses __is_io_address() macro to determine whether ipx4xx_pci_read() (via inl()) or __raw_readl()/__indirect_readl() should be used.

2. loff_t is signed, and it probably makes more sense to return -EFAULT rather that -EINVAL when the offset is invalid:

   // *f_pos must be within bounds
   if (*f_pos < 0 || *f_pos >= MAILBOX_SIZE) {
       retval = -EFAULT;
       goto out;
   }
   
   // *f_pos must be a multiple of 4, and
   // count must be 4 (mailbox supports only reading 4 bytes at a time)
   if ((*f_pos & 3) || count != sizeof mlb_data) {
       retval = -EINVAL;
       goto out;
   }
   

3. Instead of readl(), consider using the newer ioread32() (or ioread32be(), if the device always uses big-endian byte ordering). See include/asm-generic/iomap.h for details.

   On many architectures, ioread32() does just call readl(), so this is not a functional change; this is more about being easier to maintain in the long term, by keeping up with the currently recommended internal kernel interfaces.

   (The comment that "[certain] architectures cannot use this generic interface" is directed to architecture maintainers, that they cannot just rely on the generic interface, but need to implement the macros for their hardware architecture in arch-specific files. Driver developers can rely on these macros being available and working correctly.)

4. Raising SIGBUS here does not really make sense, since the userspace is not trying to access memory directly, but use a syscall; returning -EFAULT is definitely more appropriate here. However, to raise SIGBUS, you do

   struct kernel_siginfo info;
   
   clear_siginfo(&info);      // Important! So you don't leak to userspace.
   
   info.si_signo = SIGBUS;
   info.si_code = BUS_ADRALN; // Reason, see include/uapi/asm-generic/siginfo.h
   info.si_errno = 0;
   info.si_addr = addr;       // Address of the error
   info.si_addr_lsb = lsb;    // Least significant bit of the address
   
   force_siginfo(&info);
   

I hope this helps, and that you'll push your GPL driver upstream, even if there are only a few users. Greg KH in particular has helped with such work in the past.

Answered By - Glärbo