Anyone want to create a schema for tracking tables in ROMs?

[cboles]

I'm very close to releasing a reflashing tool that will allow you to replace the code / data in WRX ecu's. The next big step after that will be to start creating tools to edit the various tables / data in ROM. One of the difficulties with this is that each version of the ECU code potentially stores this data in different places. We need to create some kind of definition file / database schema to populate over time which holds information about what all of the data is, and where it is for each ROM version. Something open source that we can use across platforms. Maybe XML or SQL. e.g.

Code: Select all

<rom>
   <versionrom>A4SGD10C_US</versionrom>
   <versioncase>AF423</versioncase>
   <versioncar>WRX USDM 2002 MT</versioncar>
   <romfilesize>163840</romfilesize>
   <romarea name="FB1" start=0x00000 end=0x0FFFF></romarea>
   <romarea name="FB2" start=0x10000 end=0x1FFFF></romarea>
   <romarea name="FB3" start=0x28000 end=0x2FFFF></romarea>
   <tables>
      <table type="3D" name="Main Ignition Map" units="degrees" scalefactor=12.2 offset=0 storagetype="unsigned byte" storageaddress=0x283BA>
         <axis type="X" name="Engine Load in Air per Cylinder Charge" units="grams" scalefactor=1.1 offset=0 storagetype="unsigned byte" storageaddress=0x283A0 length=16></axis>
         <axis type="Y" name="Engine RPM" units="RPM" scalefactor=100 offset=0 storagetype="unsigned byte" storageaddress=0x283B0 length=16></axis>
         <description>"This is a map of base ignition values"</description>
      </table>
      <table>
      .
      .
      </table>
   </tables>
</rom>


...you get the idea. In some ways a relational DB could be better than this XML, particularly when a table is essentially the same in many ROMs except for the address it is located at. We could do this in XML too - we just need to use the table info by reference. Once we have something like this, we can start making table editors and the like relatively easily. Is anyone up for tackling this side of the project? It would be nice to use a data storage format and code to read/write it that are cross-platform compatible...

[NeverLies]

We may use a MySQL table that may be accessible by all members. Than this database may be copied on each target machine and used as a part of the editor software.
There may be some problems as some map may have different size regarding ECU type, need to review this while designing database.

Can you store MySQL database on your site ? Or do we have to look for antoher way to store it while in development ?

[cboles]

It's no problem to host a DB on the site. The issues I see are

* who has rights to edit / manage the DB?
* we don't want to require a net connection to use the software.

It might be nice to moderate submissions to a mysql DB and have that DB available online and as a downloadable file. Or is there some kind of DB synchronization support so that users can use an offline copy?

[NeverLies]

I think best way would be to have an online DB. This DB may be exported to a SQL file that may be used locally by the editor software. So no internet access should be needed while using editor sw.

[LGT-3-6]

This is why XML was created after all, no? It could also be tracked using CVS.

[calvinc]

is there no definition within the ECU ROM image itself? or is it just the raw map table data?

calvin.

[cboles]

we're talking assembly-level code here. table references are just hard coded in. going forward, i would like to create "live tuning" features for the ROMs which would have them look to a pointer in RAM instead for the table address. that way each table could be mapped to its ROM version or one changed on the fly in RAM.

is there no definition within the ECU ROM image itself? or is it just the raw map table data?

calvin.

[ahains]

One other benefit of using XML is we can define an XML Schema to validate the form/values of the document.
It is nice to remove this kind of checking logic from the apps that need to auther/edit/read the info, and let them just validate the doc against the schema. Regardless of what platform folks are coding in, there should be XML and XML Schema classes available to them to make that very simple.

I work with XML and XSD quite a bit, so I could whip up the XSD once we have the allowable shape and value space of the XML data defined.

-Adrian

[ahains]

I whipped up a general XML Schema to define this kind of document. I have attached it here.
I have made only slight changes to what is above. I think camel casing makes it more readible, so I have used this. I have also broken out the model info into seperate fields so it can be queried against/utilized more easily. I have defined many of the enumeration type fields as such, but I don't know if we care about this level of constraint. If so, we'll obviously have to compile a list of allowed values and fill out the type definitions. We could just type everything as "string", and let there be wide variations between the definitions that folks submit. Personally I prefer some stronger typing - it makes it easier to deal with them manually, and makes it much easier to for a map editing program to work with them.
Plus, it makes it extremely simple to write a definition if you have an xml editor that is schema aware (such as Visual Studio Whidbey )... the intellisense stuff is handy.

Let me know what you think, or any changes you would like to see.

Here is an example definition that validates against the schema I wrote:

<rom xmlns="urn:RomMetadata">
<versionInfo>
<romId>A4SGD10C_US</romId>
<caseId>AF423</caseId>
<sourceCar>
<market>USDM</market>
<model>Impreza WRX</model>
<modelYear>2002</modelYear>
<transmissionType>5MT</transmissionType>
</sourceCar>
</versionInfo>
<fileInfo>
<fileSize>163480</fileSize>
<areaInfo>
<romArea name="FB1" start="0x00000" end="0x0FFFF"/>
<romArea name="FB2" start="0x10000" end="0x1FFFF"/>
<romArea name="FB3" start="0x28000" end="0x2FFFF"/>
</areaInfo>
</fileInfo>
<tables>
<table tableType="3D" name="Main Ignition Map">
<axis type="X" name="Engine Load in Air per Cylinder Charge" units="grams" scaleFactor="1.1" offset="0" storageType="unsignedByte" storageAddress="0x283A0" fieldLength="16"/>
<axis type="Y" name="Engine RPM" units="RPM" scaleFactor="100" offset="0" storageType="unsignedByte" storageAddress="0x283B0" fieldLength="16"></axis>
<datum type="Some particular related field, outside of th table" name="My Datum" scaleFactor="1" storageAddress="0xBEEEF" storageType="unsignedByte" units="grams"/>
</table>
</tables>
</rom>

Thanks!
Adrian

[Spiider]

Shouldn't we add the hex ecu identifier somwhere near the top of the def?

That way when the software connects to the ecu it can do a auto lookup of the appropriate tables.

You seem to know what you are doing with xml, my vote is to let you run with this part of the program. Do what you think would work the best.

[ahains]

Shouldn't we add the hex ecu identifier somwhere near the top of the def?

That way when the software connects to the ecu it can do a auto lookup of the appropriate tables.

You seem to know what you are doing with xml, my vote is to let you run with this part of the program. Do what you think would work the best.

Sounds like a good idea.
Do you have an example? Is it always the same number of bytes?
Does it map 1 to 1 with the string identifier, like A4SGD10C_US?

Note that I am not aware of the specifics of what the ECU image contains, nor the stuff you get back in the protocol. I'm just going off of the example XML that colby posted above. So if there are any other pieces of data that should be included in a ROM definition, or other semantics that should be captured in the XML, please let me know

Thanks!
Adrian

[cboles]

I vote that Adrian runs with this too! I think Spiider has some sample data you can use. I was thinking of parsing the data using Xerces, which is open-source C++:

http://xml.apache.org/#xerces

[RussianE39]

Why just not to use TunerPro and to make a .ecu files for it? Thrust me, it is very useful tool for tuning regular EEPROM ECUs, so why not to use it for editing Subie's firmwares?

[ahains]

Why just not to use TunerPro and to make a .ecu files for it? Thrust me, it is very useful tool for tuning regular EEPROM ECUs, so why not to use it for editing Subie's firmwares?

I had never heard of TunerPro before, but looking at the website it looks like it may be a really good match. Looks like it is stable, and free

The ECU description format is outlined here:
http://www.supportfitness.com/mark/tune ... ngECUs.htm

We may however wish to still have xml definitions of each ECU type so that open source folks can write their own editors. TunerPro is great for those of us that run Windows, but it is NOT open source so it is not versatile enough to run on linux/pda/etc.

I'm swamped at work currently (shipping a product in a few weeks), so I probably won't have time to check it out until after things settle down.

Thanks for the pointer!
-Adrian

[cboles]

FYI, I'm in the process of doing this XML work now for the map editor in the new EcuFlash. Hopefully will have a standard format for creating map definitions very soon!