/ code snippets

Dictionary from key-map value pair

While working on a project, I came across a situation where I would want to create a nested dictionary structure where the series of keys could be defined by some string. Essentially to do the following:

>>> dict_entry = ("my.dict.keys", "myvalue")
>>> print somefunc(dict_entry)
{
    "my": {
        "dict": {
            "keys": "myvalue
        }
    }
}

Additionally, I wanted to be able to combine many of these resulting dictionaries into one.

To list my requirements plainly:

  • Create nested dictionary based on a key map and value
  • Merge multiple dictionaries together without overwriting keys

key map to dict

I initially approached this problem using for loops and two functions, however I quickly realized that this is more easily solved using functional programming methods. In short, I came up with the following:

def keymap_to_dict(keymap, value):
    key_list = keymap.split(".")[::-1]
    return reduce(lambda v, k: {k: v}, key_list, value)

merging dicts

This seemed like a problem that should have already been solved before, and sure enough the solution was laying in wait on Stack Overflow. Many thanks to SO user andrew cooke.

def merge(a, b, path=None):
    "merges b into a"
    if path is None: path = []
    for key in b:
        if key in a:
            if isinstance(a[key], dict) and isinstance(b[key], dict):
                merge(a[key], b[key], path + [str(key)])
            elif a[key] == b[key]:
                pass # same leaf value
            else:
                raise Exception('Conflict at %s' % '.'.join(path + [str(key)]))
        else:
            a[key] = b[key]
    return a

solution draft 1

With these two functions sorted out, I wanted to connect everything together, which actually turned out to be more interesting than I anticipated. I came up with the following pretty quickly:

def build_dict_structure(keymap_list):
    dict_list = []
    for i in keymap_list:
        dict_list.append(keymap_to_dict(*i))
    return reduce(merge, dict_list)

my_data = [("test", "helloworld"),
           ("my.data.example", 5),
           ("my.data.foo", 9000),
           ("my.bar", "sillywalks")]
d = build_dict_structure(my_data)

Though, the for loop that is collecting function output in a list is a perfect use case for map. The problem is that I'm expanding the tuple in the call to the keymap_to_dict function, since I can't tell map to expand each item it pulls from the list.

Enter ittertools.starmap, which does exactly as it sounds, and exactly as I need. The official documentation explains it better than I can, so I'll just show the resulting code:

def build_dict_structure(keymap_list):
    dicts = starmap(keymap_to_dict, keymap_list)
    return reduce(merge, dicts)

my_data = [("test", "helloworld"),
           ("my.data.example", 5),
           ("my.data.foo", 9000),
           ("my.bar", "sillywalks")]
d = build_dict_structure(my_data)

Conclusion

That's pretty much it. It was a short and simple bit of code, but an enjoyable puzzle to polish with my newfound functional programming knowledge. For the sake of completeness, the full code is listed below:

from functools import reduce
from itertools import starmap
import json

def keymap_to_dict(keymap, value):
    """ Create nested dict with value at the bottom
    """
    # Splits the attribute map into a list, and then reverse the list
    key_list = keymap.split(".")[::-1]
    # Reduces the key list and value into a next dict structure
    # Example:
    #  key_list = ["attribute", "key", "my"]
    #  value = 5
    #  {'my': {'key': {'attribute': 5}}}
    return reduce(lambda v, k: {k: v}, key_list, value)

def merge(a, b, path=None):
    """ Merges b into a
    Beautiful dictionary merge function from Stack Overflow user `andrew cooke`
    https://stackoverflow.com/a/7205107
	"""
    if path is None: path = []
    for key in b:
        if key in a:
            if isinstance(a[key], dict) and isinstance(b[key], dict):
                merge(a[key], b[key], path + [str(key)])
            elif a[key] == b[key]:
                pass # same leaf value
            else:
                raise Exception('Conflict at %s' % '.'.join(path + [str(key)]))
        else:
            a[key] = b[key]
    return a

def build_dict_structure(settings_list):
    return reduce(merge, starmap(keymap_to_dict, settings_list))

my_data = [("test", "helloworld"),
           ("my.data.example", 5),
           ("my.data.foo", 9000),
           ("my.bar", "sillywalks")]

d = build_dict_structure(my_data)
# Dump to json just to get pretty output
print(json.dumps(d, sort_keys=True, indent=4))

# Resultant output is:
# {
#     "my": {
#         "bar": "sillywalks",
#         "data": {
#             "example": 5,
#             "foo": 9000
#         }
#     },
#     "test": "helloworld"
# }