Many systems have requirements to store configuration parameters. In these systems, a number of choices can be made for how to store that data; sometimes this diversity is painful, however. Choices for storing configuration data are often:<\/p>\n
This diversity of configuration formats sometimes sees cross pollination, however. Sometimes, an application only reads in one format but another application only outputs another format. Sometimes, one has a toolset which works with only one and many an application grown organically can find itself using many formats itself.<\/p>\n
Annoyingly, not all formats support the same set of features either. For example, SQLite3 and XML can be multidimensional; SQLite3 supports multiple N-row by M-columns sized tables in a SQLite3 file, while XML support a hierarchical tree structure of tags with with multiple leaves using attributes on tags. JSON is comparable to XML, offering rich structure for organizing one’s data. The initialization file implementation in Python is only a two-level hierarchy; Java Properties files are flat but often use Java dot-notation to make namespaces which can represent an arbitrarily deep hierarchy. Domain specific languages can be as rich or simple as desired, but there is no commonality or properties inherent in such a configuration format.<\/p>\n
This asymmetry can make conversion across formats difficult in general but one should always be able to go from a less rich to a more rich structure. And when possible, it is nice to have some tools to go between them.<\/p>\n
One can find a recipe<\/a> to read and write Java Properties files from Python. This re-implementation of the java.util.Properties<\/tt> class provides a convenient interface for working with properties files:<\/p>\n One can write an XML version of a Java properties file within Java by simply calling the storeToXML()<\/tt><\/a> method on a Properties()<\/tt> object.<\/p>\n I use a lot of Hadoop programs which store their outputs in various XML forms, but one which always drives me nuts is Apache Oozie<\/a>. Oozie will dump out a workflow job configuration in XML; but not a standard Java XML properties file. Oozie takes in the workflow properties as a non-XML Java properties file provided but it will not accept the XML it produces. However, via the joys of XML Style Sheet Transforms<\/a>, we can write a simple script which can convert between the two!<\/p>\n An example (Oozie) Properties file in XML:<\/strong><\/p>\n General XSLT transformation from XML to Java properties file<\/strong><\/p>\n Resulting Java properties file<\/strong><\/p>\n For those who are not very programming language literate, on Linux, one can nicely use the simple libxml<\/a> tool xsltproc(1)<\/tt><\/a> to run this conversion. For example, to take in my_config<\/tt> in Java properties XML format and product the same file in Java properties format one would run: xsltproc to_property.xslt my_config.xml > my_config.properties<\/tt><\/p>\n JSON provides a rich language for expression similar to XML. JSON is often used for data interchange, now often used in AJAX web-requests, etc. However, JSON,<\/p>\n Python has a very feature-rich JSON module which takes the JSON objects and arrays and all their pairs and members representing them akin to native Python list()<\/tt> and dict()<\/tt> objects. Further, the JSON module can provide very rich encoding and decoding functionality, as evidenced in the module’s PyDoc<\/a> and particular when using<\/a> hooks for encoding and decoding.<\/p>\n","protected":false},"excerpt":{"rendered":" Many systems have requirements to store configuration parameters. In these systems, a number of choices can be made for how to store that data; sometimes this diversity is painful, however. Choices for storing configuration data are often: Firefox uses and Apple often chooses to use sqlite3 databases1 Python programs often use ConfigParser to process\u00c2\u00a0initialization (.ini) […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14,17],"tags":[],"_links":{"self":[{"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/posts\/64"}],"collection":[{"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/comments?post=64"}],"version-history":[{"count":0,"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/posts\/64\/revisions"}],"wp:attachment":[{"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/media?parent=64"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/categories?post=64"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/clayb.net\/blog\/wp-json\/wp\/v2\/tags?post=64"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}>>> import properties\r\n>>> p=properties.Properties()\r\n>>> with file(\"my.properties\") as f:\r\n... p.read(f)\r\n>>> p.getPropertyDict()['some_property_I_want']\r\n'this_is_not_the_property_value_you_want!'\r\n>>> p.setProperty('some_property_I_want', 'with_the_value_I_want!')\r\n>>> with file(\"my.properties\") as f:\r\n... p.store(f)<\/pre>\nProperties in XML<\/h3>\n
Oozie’s XML outputs<\/h3>\n
<configuration>\r\n <property>\r\n <name>date<\/name>\r\n <value>2011-12-01T00:00Z<\/value>\r\n <\/property>\r\n <property>\r\n <name>endTime<\/name>\r\n <value>2011-12-01T23:59Z<\/value>\r\n <\/property>\r\n <property>\r\n <name>frequency<\/name>\r\n <value>1440<\/value>\r\n <\/property>\r\n <property>\r\n <name>group.name<\/name>\r\n <value>users<\/value>\r\n <\/property>\r\n <property>\r\n <name>jobTracker<\/name>\r\n <value>jobtracker.example.com:9001<\/value>\r\n <\/property>\r\n <property>\r\n <name>nameNode<\/name>\r\n <value>hdfs:\/\/namenode.example.com:9000<\/value>\r\n <\/property>\r\n <property>\r\n <name>oozie.coord.application.path<\/name>\r\n <value>\/export\/my_workflow\/coordinator.xml<\/value>\r\n <\/property>\r\n <property>\r\n <name>oozie.wf.application.path<\/name>\r\n <value>hdfs:\/\/namenode.example.com:9000\/user\/john_doe\/my_workflow\/workflow.xml<\/value>\r\n <\/property>\r\n <property>\r\n <name>queueName<\/name>\r\n <value>default<\/value>\r\n <\/property>\r\n <property>\r\n <name>startTime<\/name>\r\n <value>2011-12-01T00:00Z<\/value>\r\n <\/property>\r\n <property>\r\n <name>user.name<\/name>\r\n <value>john_doe<\/value>\r\n <\/property>\r\n<\/configuration><\/pre>\n
<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\r\n<xsl:stylesheet version=\"1.0\" xmlns:xsl=\"http:\/\/www.w3.org\/1999\/XSL\/Transform\">\r\n<xsl:output method=\"text\" version=\"1.0\" omit-xml-declaration=\"yes\"\/>\r\n <xsl:template match=\"\/*\">\r\n <xsl:for-each select=\"property\">\r\n <xsl:value-of select=\"name\"\/><xsl:text>=<\/xsl:text><xsl:value-of select=\"value\"\/><xsl:text>
<\/xsl:text>\r\n <\/xsl:for-each>\r\n <\/xsl:template>\r\n<\/xsl:stylesheet><\/pre>\n
date=2011-12-01T00:00Z\r\nendTime=2011-12-01T23:59Z\r\nfrequency=1440\r\ngroup.name=users\r\njobTracker=jobtracker.example.com:9001\r\nnameNode=hdfs:\/\/namenode.example.com:9000\r\noozie.coord.application.path=\/export\/my_workflow\/coordinator.xml\r\noozie.wf.application.path=hdfs:\/\/namenode.example.com:9000\/user\/john_doe\/my_workflow\/workflow.xml\r\nqueueName=default\r\nstartTime=2011-12-01T00:00Z\r\nuser.name=john_doe<\/pre>\n
JSON<\/h2>\n
Using with Python<\/h3>\n