How to configure the device in a easy way?

[wjhtinger]

Hi, I want to run Linux+RTOS on bao。The Linux will use all the hardware devices, and the RTOS don't use hardware device, only do some numerical calculations on CPU。
Now I have to add the device configuration into config file one by one, which is really troublesome and easy to go wrong. So if there some easy way that can configure the whole hardware devices to Linux? Can you give a example config file?
Very thanks.

[josecm]

You don't need to do it one by one, you can actually group them by large MMIO regions. Only the interrupts you will need to add one by one. You are not the first to want to do this. We've discussed changing the interrupt configuration to be a struct representing a range instead of a simple array of ids, but never got to it.

[wjhtinger]

You don't need to do it one by one, you can actually group them by large MMIO regions. Only the interrupts you will need to add one by one. You are not the first to want to do this. We've discussed changing the interrupt configuration to be a struct representing a range instead of a simple array of ids, but never got to it.

Just considering the scenario in bao useage, mostly there will be two types of OS: big system(Linux or Android etc.) and RTOS. And the big system would use most hardware devices and the RTOS use a little(even not use any).
So, I think the config file based on "exclusive method" is better and easyer to use. Just like, there is a configuration in config file indicate that Linux occupy the total hardware devices（include MMIO, Interrupt etc.）defaulty, and with some "exclusive" configuration for some devices indicate these devices are removed from Linux for RTOS use.
In this way, we dont need add device configuration one bye one for the VMs. If you implement it , I can do the test.

[josecm]

That is an interesting idea that seems indeed very useful for many use cases. The only obvious issue I see here is that Bao is blind regarding the MMIO regions and associated interrupts existent in the platform, so we would have to add this information to the platform descriptions. I will add this to my TODO list. However, I probably won't have the time to work on it in the near future. You would be most welcome to submit a PR even if with only a draft implementation. We could collaborate on it from your start point.

[wjhtinger]

That is an interesting idea that seems indeed very useful for many use cases. The only obvious issue I see here is that Bao is blind regarding the MMIO regions and associated interrupts existent in the platform, so we would have to add this information to the platform descriptions. I will add this to my TODO list. However, I probably won't have the time to work on it in the near future. You would be most welcome to submit a PR even if with only a draft implementation. We could collaborate on it from your start point.

Your concern is right, the Bao hard code should not know the IO config dietails which is according to different hardware.

At first, there should be global discriptor to cover the whole IO region, just like:
{
/* global IO region*/
.pa = 0x00000000,
.va = 0x00000000,
.size = 0x20000000,
.interrupts_list = {32,256} /* the interrupts list, from 32 to 256 */
},
So, based on current code, only need to implement a interrupts list mechanism to include interrupts in easy way.

Second, there should be some excluded discriptors to indicate that the devices are removed for other OS use, just like:
{
/* gpio for RTOS use*/
.pa = 0x2200000,
.va = 0x2200000,
.size = 0x20000,
.interrupts = {79,82,85,88,91}, / *interrupts /
.interrupts_list = {210,212}, / interrupts list */
.excluded = 1 /excluded field/
},
At current discriptor, only add a .excluded field to indicate this device IO should removed from this OS, just like digging some holes in the whole device IO region. Or, there can be a excluded sub field, in which all the device discriptors are excluded.

At last, just the config has "include" and "exclude", which makes the config layout more complex. So the hypervisor need to do some conflicting checking for IO mapping in different OS.

[josecm]

Your concern is right, the Bao hard code should not know the IO config dietails which is according to different hardware.

I'm not saying it should not, just that it currently does not have that information and adding it wouldn't be trivial.

based on current code, only need to implement a interrupts list mechanism to include interrupts in easy way.

As you point out, defining interrupts as ranges can indeed be easily implemented. However, I'd suggest to implement the interrupts field as list of "interrupt range" structures, not as single range.

Regarding the rest of your proposal: you are suggesting that each IO resource description in a given VM's devs list might be either an assignment or revocation according to the excluded field. Is this right? If not, please clarify. I'd suggest appending a full configuration. I can't discern if the two excerpts are part of the same vm_config_t or not.

Or, there can be a excluded sub field, in which all the device discriptors are excluded.

I don't understand what is the alternative you are suggesting here.

At last, just the config has "include" and "exclude", which makes the config layout more complex. So the hypervisor need to do some conflicting checking for IO mapping in different OS.

Also, I'm not clear on what you mean here. Nevertheless, excluding interrupts is also simple because this is implemented just by flipping bits in a bitmap part the vm management structures. As for the "holes" in the IO regions, it might get tricky. I'd suggest having the configuration in such a format that wouldn't force the hypervisor to look at other VM configurations while setting up a given VM. This would help minimize the overall logic added to the hypervisor. Right now it is not obvious to me how we'd implement this in a straightforward way.

[wjhtinger]

Regarding the rest of your proposal: you are suggesting that each IO resource description in a given VM's devs list might be either an assignment or revocation according to the excluded field. Is this right?

Yes, right. But if there has a global discriptor which already cover the whole IO region, you dont need
include discriptor any more except some exclude discriptors for revocation.

Or, there can be a excluded sub field, in which all the device discriptors are excluded.
I don't understand what is the alternative you are suggesting here.

I mean there could be a sub field, in which all the discriptors are excluded, without adding .excluded flag in every discriptor. Anyway, this suggest is only a style issue, not important.

Whatever, I dont know the code in details, so the suggestions is only based on useage view, only for your reference, and the implementaion would think more. But anyway, if there is a way to configure the whole IO in a easy way, which would help a big.

[DavidMCerdeira]

The easiest way to do this right now is something like this (excerpt):

        .dev_num = 4,
        .devs =  (struct dev_region[]) {
            {
                .pa = 0x00000000,
                .va = 0x00000000,
                .size = 0x38800000,
	    },
	    /* hole for gic */
	    {
                .pa = 0x38900000,
                .va = 0x38900000,
                .size = 0x40000000 - 0x38900000,
	    },
	    {
		.interrupt_num = 1,
		.interrupts = (uint64_t[]) { 27 }
	    },
	    {
		.interrupt_num = 128,
		.interrupts = (uint64_t[]) {
		    32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,
		    52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,
		    72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,
		    92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,
		    109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,
		    124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,
		    139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,
		    154,155,156,157,158,159,
		},
	    }
        },

        .arch = {
	    .gic = {
		.gicd_addr = 0x38800000,
		.gicr_addr = 0x38880000,
	    },
        }

This creates a hole in the MMIO range to allow bao to emulate the interrupt controller, but otherwise gives full control of other hardware devices to the guest.
A solution to this could to have a MMIO region which is "main", and have an additional attribute called "blacklisted".
This could result in a configuration file that looks like this

        .dev_num = 4,
        .devs =  (struct dev_region[]) {
            {
                .pa = 0x00000000,
                .va = 0x00000000,
                .size = 0x40000000,
                .main = TRUE
	    },
            {
                .va = 0x38800000,
                .size = 0x00100000,
                .blacklisted = TRUE
            },
	    {
		.interrupt_num = 1,
		.interrupts = (uint64_t[]) { 27 }
	    },
	    {
		.interrupt_num = 128,
		.interrupts = (uint64_t[]) {
		    32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,
		    52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,
		    72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,
		    92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,
		    109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,
		    124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,
		    139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,
		    154,155,156,157,158,159,
		},
	    }
        },

        .arch = {
	    .gic = {
		.gicd_addr = 0x38800000,
		.gicr_addr = 0x38880000,
	    },
        }

This way Bao could punch the holes in the MMIO region itself, and it is easier to give a guest access to all devices.

[josecm]

I've started working on this by implementing support for interrupt ranges in the config at wip/default-vm.

Regarding the MMIO regions: I'm still ruminating on which of the ideas would be the best, mainly from a user's (both configuration writer and reader) perspective. Having the excluded field directly on the "owner" OS device configuration (as @wjhtinger suggested) would be easier for the writer since they would only need to write this information in one place. But it wouldn't be straightforward for a reader to understand which devices a given VM has access to only by looking at that VM's configuration. On the other hand, having excluded as an extra "blacklist" field (as @DavidMCerdeira suggested) in any of the MMIO device configurations would make it easier for a user to discern which devices are and are not part of that VM, but would make the configuration writer manage the excluded/"included" pair in two VM configurations (which might be error more error prone). The latter would be probably simpler to implement given that, at the point the hypervisor assigns the devices, it only has direct knowledge of the VM that is currently being configured and that will run in that specific core.

A third idea would be to add information about all the MMIO regions to the platforms' descriptions and have a default_vm field in the overall configuration. The hypervisor would then need look at all other VM configurations, and assign all MMIO/interrupts not assigned to any other VM, to the default VM.