pathtree: Make dataitems() use the cache dict under restricted args

This achieves a 100x improvement in iteration time when the subtree
rooted at the root of the full tree and there is no depth limi

pathtree: Make dataitems() use the cache dict under restricted args

This achieves a 100x improvement in iteration time when the subtree
rooted at the root of the full tree and there is no depth limit (i.e.
the caller has requested all "populated" items). Whilst it sounds like a
restricted case, it is a very common query by way of phosphor-objmgr's
obmc/mapper/server.py:ObjectMapper.process_old_owner()

Note this is a quasi API break, as some keys that were previously
not present in the results may now appear: Explicitly storing 'None'
into the data structure will have the key with 'None' appear in the
full-tree dataitems() case. However, given this was filtered previously,
no existing callers should be storing 'None' into the tree as they would
not have been able to retrieve it via dataitems().

Before:
$ python -m obmc.utils.testpathtree
...
Iteration tests:
depth=1, width=1, n=1000: 0.135853052139
depth=1, width=2, n=1000: 0.203811883926
depth=1, width=3, n=1000: 0.26814198494
depth=1, width=4, n=1000: 0.333888053894
depth=2, width=1, n=1000: 0.193987131119
depth=2, width=2, n=1000: 0.264018058777
depth=2, width=3, n=1000: 0.327262878418
depth=2, width=4, n=1000: 0.38805603981
depth=3, width=1, n=1000: 0.253651857376
depth=3, width=2, n=1000: 0.317117929459
depth=3, width=3, n=1000: 0.385557889938
depth=3, width=4, n=1000: 0.452265024185
depth=4, width=1, n=1000: 0.327889919281
depth=4, width=2, n=1000: 0.390358924866
depth=4, width=3, n=1000: 0.459683895111
depth=4, width=4, n=1000: 0.530153989792

After:
$ python -m obmc.utils.testpathtree
...
Iteration tests:
depth=1, width=1, n=1000: 0.0012412071228
depth=1, width=2, n=1000: 0.00455403327942
depth=1, width=3, n=1000: 0.00307989120483
depth=1, width=4, n=1000: 0.00356507301331
depth=2, width=1, n=1000: 0.00118088722229
depth=2, width=2, n=1000: 0.00169396400452
depth=2, width=3, n=1000: 0.00234699249268
depth=2, width=4, n=1000: 0.00300288200378
depth=3, width=1, n=1000: 0.00100708007812
depth=3, width=2, n=1000: 0.00161695480347
depth=3, width=3, n=1000: 0.00234794616699
depth=3, width=4, n=1000: 0.00315403938293
depth=4, width=1, n=1000: 0.00101804733276
depth=4, width=2, n=1000: 0.00204801559448
depth=4, width=3, n=1000: 0.00281095504761
depth=4, width=4, n=1000: 0.0070219039917

Change-Id: Ice3afd12e2b112227735f0f1dedb6a8ea594740c
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

pathtree: Allow get() on subtrees

When running recent PathTree modifications on a BMC we can see failures
in the log such as "Failed to invoke ObjectMapper method" from the
phosphor-fru-fault-monito

pathtree: Allow get() on subtrees

When running recent PathTree modifications on a BMC we can see failures
in the log such as "Failed to invoke ObjectMapper method" from the
phosphor-fru-fault-monitor. This can be triggered via the
openbmc-test-automation suite, for example with the following command

$ robot -v OPENBMC_HOST:w5.aus.stglabs.ibm.com --include \
Create_Test_Error_Callout_And_Verify extended/test_association.robot

Capturing the DBus traffic across this test case
(`busctl capture > dbus.pcap`) and analysing it with Wireshark showed the
following method invocation:

```
0000 6c 01 00 01 53 00 00 00 16 00 00 00 c6 00 00 00 l...S...........
0010 01 01 6f 00 22 00 00 00 2f 78 79 7a 2f 6f 70 65 ..o.".../xyz/ope
0020 6e 62 6d 63 5f 70 72 6f 6a 65 63 74 2f 6f 62 6a nbmc_project/obj
0030 65 63 74 5f 6d 61 70 70 65 72 00 00 00 00 00 00 ect_mapper......
0040 03 01 73 00 09 00 00 00 47 65 74 4f 62 6a 65 63 ..s.....GetObjec
0050 74 00 00 00 00 00 00 00 02 01 73 00 20 00 00 00 t.........s. ...
0060 78 79 7a 2e 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a xyz.openbmc_proj
0070 65 63 74 2e 4f 62 6a 65 63 74 4d 61 70 70 65 72 ect.ObjectMapper
0080 00 00 00 00 00 00 00 00 06 01 73 00 20 00 00 00 ..........s. ...
0090 78 79 7a 2e 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a xyz.openbmc_proj
00a0 65 63 74 2e 4f 62 6a 65 63 74 4d 61 70 70 65 72 ect.ObjectMapper
00b0 00 00 00 00 00 00 00 00 08 01 67 00 03 73 61 73 ..........g..sas
00c0 00 00 00 00 00 00 00 00 07 01 73 00 05 00 00 00 ..........s.....
00d0 3a 31 2e 35 37 00 00 00 20 00 00 00 2f 78 79 7a :1.57... .../xyz
00e0 2f 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a 65 63 74 /openbmc_project
00f0 2f 6c 65 64 2f 67 72 6f 75 70 73 2f 00 00 00 00 /led/groups/....
0100 27 00 00 00 22 00 00 00 6f 72 67 2e 66 72 65 65 '..."...org.free
0110 64 65 73 6b 74 6f 70 2e 44 42 75 73 2e 4f 62 6a desktop.DBus.Obj
0120 65 63 74 4d 61 6e 61 67 65 72 00 ectManager.
```

and the response

```
0000 6c 03 01 01 3f 00 00 00 9d 08 00 00 5e 00 00 00 l...?.......^...
0010 06 01 73 00 05 00 00 00 3a 31 2e 35 37 00 00 00 ..s.....:1.57...
0020 04 01 73 00 27 00 00 00 6f 72 67 2e 66 72 65 65 ..s.'...org.free
0030 64 65 73 6b 74 6f 70 2e 44 42 75 73 2e 45 72 72 desktop.DBus.Err
0040 6f 72 2e 46 69 6c 65 4e 6f 74 46 6f 75 6e 64 00 or.FileNotFound.
0050 05 01 75 00 16 00 00 00 08 01 67 00 01 73 00 00 ..u.......g..s..
0060 07 01 73 00 05 00 00 00 3a 31 2e 31 35 00 00 00 ..s.....:1.15...
0070 3a 00 00 00 70 61 74 68 20 6f 72 20 6f 62 6a 65 :...path or obje
0080 63 74 20 6e 6f 74 20 66 6f 75 6e 64 3a 20 2f 78 ct not found: /x
0090 79 7a 2f 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a 65 yz/openbmc_proje
00a0 63 74 2f 6c 65 64 2f 67 72 6f 75 70 73 2f 00 ct/led/groups/.
```

These packets translate to the following `busctl` invocation and response:

```
root@witherspoon:~# busctl call \
xyz.openbmc_project.ObjectMapper \
/xyz/openbmc_project/object_mapper \
xyz.openbmc_project.ObjectMapper \
GetObject sas \
/xyz/openbmc_project/led/groups/ \
1 \
org.freedesktop.Dbus.ObjectManager
path or object not found: /xyz/openbmc_project/led/groups/
root@witherspoon:~#
```

From `busctl tree` we can see that the objects do actually appear on the
bus:

root@witherspoon:~# busctl tree
...
Service xyz.openbmc_project.LED.GroupManager:
└─/xyz
└─/xyz/openbmc_project
└─/xyz/openbmc_project/led
└─/xyz/openbmc_project/led/groups
├─/xyz/openbmc_project/led/groups/bmc_booted
├─/xyz/openbmc_project/led/groups/bmc_fault
├─/xyz/openbmc_project/led/groups/boxelder_fault
├─/xyz/openbmc_project/led/groups/core0_fault
├─/xyz/openbmc_project/led/groups/core10_fault
...
```

However, the /xyz/openbmc_project/led/groups/ path represents a subtree
root, not a leaf element. I didn't realise this was a supported
use-case, and over-aggressively removed support for it in the recent
patch adding the flat dict cache.

Add in a fallback to iterating the PathTree datastructure when we take a
cache miss.

Change-Id: I658c3fdd6b87be9d814c59a654e63ec5bb83e2bd
Fixes: 52aeb314c0eb ("pathtree: Cache keys in a flat dict for fast lookup")
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

pathtree: Cache keys in a flat dict for fast lookup

This knocks about a second off of the depth stress tests:

$ python -m obmc.utils.testpathtree
Depth tests:
1: 5.18097305298
2: 6.

pathtree: Cache keys in a flat dict for fast lookup

This knocks about a second off of the depth stress tests:

$ python -m obmc.utils.testpathtree
Depth tests:
1: 5.18097305298
2: 6.18765211105
3: 7.3427259922
4: 8.33903598785
5: 9.55236506462
6: 10.7165560722
7: 12.1563079357
8: 13.3570129871
9: 14.7827298641
10: 16.0967490673

Width tests:
1: 1.20649981499
2: 1.20889282227
4: 1.20902085304
8: 1.22063994408
16: 1.22446990013
32: 1.21649098396
64: 1.21024799347
128: 1.21360588074
256: 1.22187900543
512: 1.23233604431
1024: 1.21607708931
2048: 1.21069407463
4096: 1.22389698029
8192: 1.20828580856
16384: 1.21290493011
32768: 1.21552395821
65536: 1.23201704025
131072: 1.21459794044
262144: 1.24190402031
524288: 1.21342992783
1048576: 1.21248602867

However, we get just over a 300% improvement in the host reboot
scenario, which is captured by some separate tools used to analyse the
PathTree's performance issues:

$ # Before
$ ./replay.py -n 1000 ptinit.log ptreboot.log
deep copy: 1000 iterations took 3.0744523939938517s
ops script: 1000 iterations took 10.03650910500437s
net: 1000 iterations took 6.9620567110105185s
$ # After
$ ./replay.py -n 1000 ptinit.log ptreboot.log
deep copy: 1000 iterations took 6.784729664999759s
ops script: 1000 iterations took 9.0453162470003s
net: 1000 iterations took 2.260586582000542s

Change-Id: I6826b99950fa1561d292908c4ffff1137ecaa1bc
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

show more ...

utils: Add characterisation tests for PathTree

Change-Id: I1ad4832779e85a54a75b3551a8beeba429679e1c
Signed-off-by: Andrew Jeffery <andrew@aj.id.au>

Port to python 3

Port code to python 3 yet remain backward compatible to python 2.
This port is in preparation for yocto 2.4 upgrade.

Resolves openbmc/openbmc#2920

Change-Id: Ie06d206a3de3de2ea636

Port to python 3

Port code to python 3 yet remain backward compatible to python 2.
This port is in preparation for yocto 2.4 upgrade.

Resolves openbmc/openbmc#2920

Change-Id: Ie06d206a3de3de2ea636b043c099a0185b32ce53
Signed-off-by: CamVan Nguyen <ctnguyen@us.ibm.com>

show more ...

Revert "Port obmc to python3.5"

Parse error with this change:
TypeError: iter() returned non-iterator of type 'PathTreeItemIterator'

This reverts commit f99783be6592130bbad2d0f69ada2b7ac61618f3.

C

Revert "Port obmc to python3.5"

Parse error with this change:
TypeError: iter() returned non-iterator of type 'PathTreeItemIterator'

This reverts commit f99783be6592130bbad2d0f69ada2b7ac61618f3.

Change-Id: Id2b7f4af5284d25d59e19905c7b873726b02f700
Signed-off-by: Brad Bishop <bradleyb@fuzziesquirrel.com>

show more ...

Port obmc to python3.5

Change-Id: I1ed6c6bd914635485a697369048a8be9e5919fc6
Signed-off-by: Balaji B Rao <balajibapu@gmail.com>

Merge pull request #1 from bradbishop/pull

Pull

Add dictionary export to PathTree

PathTree is a dictionary extension with paths as keys and lets
you do things like iterate on subpaths, etc. Internally it is
implemented as a nested dictionary but

Add dictionary export to PathTree

PathTree is a dictionary extension with paths as keys and lets
you do things like iterate on subpaths, etc. Internally it is
implemented as a nested dictionary but on the outside it acts
like a normal dictionary.

This change enables exporting a PathTree structure as a nested
dictionary, which is useful for python_2_xyz encoders that understand
nested data structures.

show more ...

Introducing pyobmc

We have openbmc python utilities scattered all over the place. To
facilitate reuse, bring them together in a single python package.

None of this is new code, it was all simply p

Introducing pyobmc

We have openbmc python utilities scattered all over the place. To
facilitate reuse, bring them together in a single python package.

None of this is new code, it was all simply ported and re-arranged
from other projects.

Ran everything through pep8.

show more ...